Jester - the JUnit test tester.

Copyright (2000-2005) Ivan Moore.

Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed.

Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.

Jester is different than code coverage tools, because it can find code that is executed by the running of tests but not actually tested. However, Jester is not meant as a replacement for code coverage tools, merely as a complementary approach.

Jester is available only under the terms of the license agreement.

Jester has been tried on various operating systems, using Sun's JDK 1.4.1 and JUnit 3.8.1. There is absolutely no warranty whatsoever (see license agreement ); it might work on those or other platforms - see the Jester web page for latest known bugs, platforms that it has been tried on, FAQs, etc). Older versions of Jester might work with older versions of Java/JUnit (e.g. Jester < 1.30 for Java < 1.4).

Installation:

unzip the jester.zip file (e.g. jester105.zip) in an empty directory
java (>=1.4) must already be installed, with java and javac on the path
JUnit 3.8.1 must already be installed, and on the classpath (Jester might also be compatable with other versions of JUnit)
add jester to the classpath; 'set classpath=%classpath%;INSTALL_DIR\jester.jar' (Note - for windows you can run the included batch file 'setcp' to set the classpath to include Jester and JUnit for running the examples (test.bat and acceptancetest.bat) and Jester's tests (runAllTests.bat) in the installation directory.)
for a simple example: run the batch file 'test.bat';NOTE, the directory containing 'test.bat' must also be on the classpath.
after less than 2 minutes, there should be the following text:

For File jester\functionaltests\NotTested.java: 1 mutations survived out of 1 changes. Score = 0
jester\functionaltests\NotTested.java - changed source on line 9 (char index=146) from 1 to 2
Anything() {
//1
return >>>1;
}
}
For File jester\functionaltests\TestAll.java: 0 mutations survived out of 0 changes. Score = -1
For File jester\functionaltests\VeryTested.java: 0 mutations survived out of 1 changes. Score = 100
For File jester\functionaltests\VeryTestedTest.java: 0 mutations survived out of 1 changes. Score = 100
1 mutations survived out of 3 changes. Score = 67
took 0 minutes

if the message above does not appear, see the FAQ referenced at the end of this document.
for a longer example, run 'acceptancetest.bat' (this needs JUnit >= 3.7 to run); this should result in '... 19 mutations survived out of 39 changes. Score = 52 ...' This example also needs Python >= 2.0 installed and generates a file "jester.html" which shows the changes Jester made that did not cause the tests to fail.

Jester has a progress window - the progress bar shows the proportion of source files that have been mutated; the text area shows the most recent mutation that has been made; the progress bar is green if the last mutation caused the tests to fail (i.e. good) and red if the tests still passed despite the mutation (i.e. bad). Note that the progress bar does not show progress while making different mutations to the same file - that is shown by text in the text area changing - because that was simpler to implement and the progress bar as it is gives a reasonably accurate indication of the amount of progress that has been made.

Usage:

Very IMPORTANT - only run Jester on a copy of your source files - as Jester changes the source code that it works on. If you use source control (which everyone should!) then you can leave your source code where it is (making sure everything you want to be in source control is checked in) and overwrite all your source from source control after running Jester. If you don't use source control, only run Jester on a copy of your source in a different directory, otherwise you'll regret it!
To run Jester, execute 'java jester.TestTester TEST_CLASS SOURCE_FILE_OR_DIRECTORY' where TEST_CLASS is the name of the JUnit test class (a subclass of TestCase that can be used by the JUnit TestRunners) and SOURCE_FILE_OR_DIRECTORY (or SOURCE_FILE_OR_DIRECTORY1 SOURCE_FILE_OR_DIRECTORY2 ... etc) are the (one or more) directories (or source files) that contain the source code that Jester changes to try to find if the tests are not covering those changes (make sure that you have another copy of those source files as Jester will change these files and might not restore them to their original if things go wrong).
The .class files needed to run your TEST_CLASS must be in the SOURCE_FILE_OR_DIRECTORY (SOURCE_FILE_OR_DIRECTORY must be on the classpath). NOTE that there are several variants of specifying the classpath:

You can have everything you need to run your tests on the static classpath
You can alter the command that Jester uses to run the tests as described later
You can specify the classpath Jester should use with the -cp option of Jester. Then you execute 'java jester.TestTester -cp=CLASSPATH TEST_CLASS SOURCE_DIRECTORY' where CLASSPATH is the classpath needed by the invoked java compiler and by the java vm to execute the tests. This way is often the most suitable when you run Jester directly from your IDE or ANT.

TEST_CLASS should be the test class that is expected to show up any changes to code in the SOURCE_DIRECTORY. Typically, TEST_CLASS would be the TestAll class of a package, and SOURCE_DIRECTORY would be the subdirectory that contains the code being tested by thatTEST_CLASS.
For any change that Jester was able to make without the tests failing, it prints the name of the file changed, the position in the file of the change (the character index), and some of the original source file from roughly 30 characters before to 30 characters after so that the change can be identified within the source file. Note that Jester can take a long time to run - try it on a small part of your system before trying it on everything.
Jester also produces a file called "jesterReport.xml" that contains all the changes that Jester made that did not cause the tests to fail.
Execute "python makeAllChangesFiles.py jesterReport.xml" to generate copies of the original source files with all the changes Jester made that did not cause the tests to fail. These files are in the same directories, and have the same names, exept have a ".jester" extension (configurable). These files are useful for comparing against the original source code using whatever you usually use to compare files, to easily find the changes Jester made that did not cause the tests to fail.
Execute "python makeWebView.py" to generate copies of the original source files as ".html" pages, with all the changes Jester made that did not cause the tests to fail shown in red, and the code it changed shown with a line through it. A file "jester.html" is generated which links to all these files. These files are in the same directories, and have the same names, as the original source files, except have a ".html" extension. This script has options:
- -z to ignore files that Jester didn't make any changes to
- -p to ignore files that Jester didn't make any changes to where the tests still passed
- -s to sort the files by file name in the "jester.html" file.
For other options, try "python makeWebView.py -h".
Jester prints out warnings when it can't find it's config files. In such circumstances, Jester uses default values which are OK in most cases.
(New small feature) You can specify that Jester should not show the progress dialog by using the "-q" command line argument.
Jester writes out a file "jesterTimeout.txt" (in the directory where it is executed) whose contents is the number of milliseconds the tests took to run the first time they were run (i.e. before Jester made any changes). This is used by Jester to stop running the tests if they are taking too long (e.g. in case a change made by Jester causes the code to enter an infinite loop).
If you kill Jester during a run, by CTRL-C, there is a risk that your source code will not be reverted back to its original state and hence the tests might fail and hence Jester will not be able to run against your code unless you revert the effected file. This might be fixed in a future version, but might not - Jester cannot be held responsible for messing up your source code, it's up to you to look after it.

Configuration options

For the following features, edit the file called "jester.cfg" from the installation directory and make sure it is on the classpath. For example, compilationCommand=javac refers to a line of text in the jester.cfg file.

Jester can use a different compiler, by setting compilationCommand to whatever you want Jester to use. By default, this is set to compilationCommand=javac.
If your compiler uses a different source file extension than .java then set sourceFileExtension as appropriate.
Jester can also be made to print the results as it goes rather than waiting until it has finished to print an amalgamated report of all the changes it has made per file, by setting shouldReportEagerly=true. This can be useful if Jester is failing to complete it's run.

Configurable mutations

The mutations that Jester applies to the source code are configurable, by editing the file "mutations.cfg" from the installation directory, which must be on the classpath. Each line of this file must have a format like:
%if(%if(true || where % can be any single character as a delimiter, and the text if( will be replaced by if(true ||. Note that this simple scheme of replacing text without parsing has proven adequate so far, but is likely to change in a future version of Jester.

Configurable Ignore List (e.g. for ignoring comments)

Jester can be made to ignore parts of every source file, by using the ignorelist.cfg file. This file contains lines starting with a delimiter character that specify regions of the source code to ignore; i.e. regions where it will not try to make a mutation. The first string is where to start ignoring the source and the second is where to stop ignoring the source. For example, to ignore comments in Java code, the ignorelist.cfg file would contain:

%/*%*/
%//%\n

(this is not entirely correct but is close enough; e.g. a string "http://jester.sf.net" up to the end of the line would not be considered for mutation by Jester using this ignorelist.cfg file, but that's not so bad.) (Note - these are included in the ignorelist.cfg file in the standard build).
This feature can be used by Pester too, e.g. to ignore python comments the ignorelist.cfg file would contain:

%#%\n

Note - the standard build also includes the line %//stopJesting%//resumeJesting so that you can easily tell Jester to ignore parts if a source file if it is causing problems for Jester. i.e. put a comment //stopJesting on the line before the code you want Jester to ignore and //resumeJesting on the line after the code you want Jester to ignore.

Using ANT

Here is a code snippet for using Jester from ANT. It copies the source file to a location where Jester can work - the tests are excluded for performance reasons. The classpath isn't specified statically but with the -cp parameter to Jester.

  <target name="jester" depends="">
    <delete dir="${modifiedsrc}"/>
    <copy todir="${modifiedsrc}">
      <fileset dir="${src}">
        <exclude name="**/*Test.java"/>
        <exclude name="**/AllTests.java"/>
      </fileset>
    </copy>
    <java classpath="${classpath}" classname="jester.TestTester">
      <arg value="-cp=${classpath}"/>
      <arg value="myproject.AllTests"/>
      <arg value="${modifiedsrc}/myproject"/>
    </java>
  </target>

If you get the source tree from CVS, you can try ant "run functionaltests" for an example of using ANT to run Jester. Have a look at Jester's build.xml to see how that works.

Known Problems:

If the code will not compile (either because of classes missing from the class path before Jester tried to compile any classes, or as a result of a change to a source file by Jester) then Jester might hang indefinitely.

This version of Jester will not run in VisualAge for Java (see Jester web page in case a new version exists that does run in VAJ).

NOTE that this version of Jester will not work with version 1.0 of Pester; a new version of Pester will be released that will work with this version of Jester - check the Jester web page if you want to see if there is a new version of Pester.

Jester can't seem to find it's config files when run from ANT - the default values usually mean that this isn't much of a problem though.

For Jester Developers:

If you want to look at Jester's java source code it is included in your "jester.zip" as a file called "source.jar". The source for the python code are the .py files. To run Jester's tests

execute 'runAllTests.bat', and the PyUnit and JUnit tests for Jester should run with no errors. Note that these tests need Python 2.2, JUnit 3.7/3.8.1 and the MockObjects framework - Jester itself will run with other versions of JUnit >= 3.2 and doesn't need the MockObjects framework to run (Some test code was generated automatically using MockMaker). (BTW - the tests are not complete - see Jester web page for details). The MockMaker modified MockObjects runtime is included as lib/mmmockobjects.jar under the relevant licenses (MockObjects license and MockMaker license), hence: This product includes software developed by the Apache Software Foundation (http://www.apache.org/). and the JUnit runtime is included as lib/junit.jar under it's license.

Acknowledgements

Very many thanks to:

Elliotte Rusty Harold for reporting and fixing bug 1035010 and for many feature requests and bug reports.
Simon Lever for his feature requests and the forthcoming Java port of "makeAllChangesFiles.py".
Diego Vallespir for reporting bugs 1031730 and 1031735.
Bernd Schiffer for sending fixes for using source directories with spaces in their names.
Stefan Roock, who has joined the Jester project as a developer.
Brett Neumeier for the suggested improvements to the makeWebView.py script.
Aho Augasmagi for getting me to implement the Ignore Lists.
Kent Beck and Robert Martin for their great quotes about Jester on the Jester home page.
Duncan Pierce for the idea of using the jesterReport.xml file to generate the ".html" files to show the changes Jester made that did not cause the tests to fail in an easily browsable way.
Manfred Kaul for sending a fix to make Jester report problems when it tries to 'exec' things that don't work (in particular, if things are missing from the classpath).

Support:

Please check the Jester web pages (hosted by sourceforge) for updates and the FAQs, and please enter comments or bug reports on the appropriate pages linked from there.

If it doesn't work

The file "jester.log" (in the directory where you ran jester) shows the commands that Jester has tried to run. In many cases, you can work out what is wrong by trying to run the last command that Jester tried to run. E.g. if the last line of the "jester.log" says "Sat Dec 01 21:46:36 PST 2001 Trying to run command "java jester.TestRunnerImpl jester.functionaltests.TestAll"", then try running "java jester.TestRunnerImpl jester.functionaltests.TestAll"; this may then reveal the problem. In the case of tests not passing (e.g. the message "Couldn't run test tester because tests didn't pass before any changes made"), try running the tests both directly (e.g. "java jester.functionaltests.TestAll") and using the command that Jester uses (e.g. "java jester.TestRunnerImpl jester.functionaltests.TestAll"). Please delete the file "jester.log" whenever it gets too large, or before running Jester if preferred.

version 1.37
jester@tadmad.co.uk