To fix the situation I started wondering if there was a JUnit version of the TODO blocks in Perl's Test::More. The problem is that JUnit can only report tests as successful or failing (or erroneous if they throw an exception), there is no way to easily mark test failures as TODOs. Googling around and asking the Jackrabbit mailing list produced some workarounds:
- Use a system property to determine whether to perform or skip the known issue test cases.
- Put the known issues tests in separate test case classes and exclude them from the test suite.
- Use a JUnit addon to ignore marked test cases as explained in an article that discusses this same issue.
- Use an alternative test framework like TestNG, that has this functionality built-in.
I didn't want to start changing the entire test suite or even tweaking the build environment, so the last two options were out. I also wanted to make the setup easily configurable so a developer can selectively enable testing for a known issue, thus the first alternative of using a system property looks like the best solution. It seems that the Apache OJB project has reached the same solution.
I really should do that when I'm doing things in Commons. Instead I tend to attach them to the JIRA issue until I figure out a fix. Previously I'd copied them to the side and not committed, but I lost too many that way.
ReplyDeleteReading your solutions; I'd be tempted to make JUnit test classes that end with Todo.java and not Test.java and have them ignored by the build. Then you'd need something custom to actually run them.
Henri Yandell:
ReplyDelete> Reading your solutions; I’d be tempted to make JUnit test classes that end with Todo.java
> and not Test.java and have them ignored by the build. Then you’d need something custom
> to actually run them.
Yes, that's another solution. My requirement was to be able to selectively enable individual known issue tests directly from the Maven command line, so the system property approach suits my needs best. See my recent followup post on how I did it.