Add test framework docs

1 files changed, 85 insertions, 0 deletions

diff --git a/plugins/org.eclipse.emf.cdo.releng.version.tests/README b/plugins/org.eclipse.emf.cdo.releng.version.tests/README
new file mode 100644
index 0000000000..52c0da4e6e
--- /dev/null
+++ b/plugins/org.eclipse.emf.cdo.releng.version.tests/README
@@ -0,0 +1,85 @@
+Declaring Version Builder Tests
+===============================
+
+This plug-in contains a declarative test framework that supports efficient specification
+of test cases and automatic execution of the test suite. The dynamic test suite can be
+executed by starting the supplied "VersionBuilderTests" JUnit Plug-in launch configuration.
+
+The test suite inspects the "tests" folder in this plug-in. Each child folder
+is a test case. The test cases are executed in alphabetical order.
+
+Each test case is made of one or several phases that are declared by child
+folders within the test case folder. The phases are executed in alphabetical order.
+
+Each phase modifies the projects, folders and files in the runtime workspace. The
+modifications are specified by the folders and files in the phase folder. Top-level
+folders (those directly contained by the phase folder) represent the projects in the
+runtime workspace. To delete folders or files from the runtime workspace the suffix
+"-DELETE" must be added to the respective folder or file in the phase folder. All other
+folders and files in the phase folder lead to an addition or modification (overwrite).
+
+When the runtime workspace has been updated as outlined above the entire runtime workspace
+is built. Only the first build of a test case is a clean build. Possible subsequent builds
+(see below) are incremental builds. If a build results in problem markers they are serialized
+to a string and this string is compared to the contents of the file "build.markers" in the
+phase folder. If they are not identical the test case fails. If the "build.markers" file does
+not exist it is created and initialized with the serialized string.
+
+										The "*.markers" files must be verified manually!
+
+The content of this file could look like this:
+
+	Marker
+	  <resource>  = com.foo.project1/META-INF/MANIFEST.MF
+	  <charStart> = (4,18)
+	  <charEnd>   = (4,23)
+	  <severity>  = ERROR
+	  <message>   = Version must be increased to 1.0.100 because the project's contents have changed
+	  problemType = component.version
+	  quickFixPattern = Bundle-Version: *(\d+(\.\d+(\.\d+)?)?)
+	  quickFixReplacement = 1.0.100
+	  FIX = Change the version (Change the version to 1.0.100)
+
+If the build resulted in one or more problem markers that have quick fixes associated then
+a second file named "build.resolutions" is written to the phase folder. You can now select
+one or more of the quick fixes by adding an asterisk (*) at the beginning of the respective
+"FIX..." line. Example:
+
+	Marker
+	  <resource>  = com.foo.project1/META-INF/MANIFEST.MF
+	  <charStart> = (4,18)
+	  <charEnd>   = (4,23)
+	  <severity>  = ERROR
+	  <message>   = Version must be increased to 1.0.100 because the project's contents have changed
+	  problemType = component.version
+	  quickFixPattern = Bundle-Version: *(\d+(\.\d+(\.\d+)?)?)
+	  quickFixReplacement = 1.0.100
+	  *FIX = Change the version (Change the version to 1.0.100)
+
+If one or more quick fixes are marked in the way outlined above a new build cycle is triggered.
+This cycle is basically the same as for the initial workspace updates of the phase. It checks or
+creates new files in the phase folder. These files are named "fix1.markers" and "fix1.resolutions".
+The number is increased by 1 per fix/build/check cycle until no markers are left or the remaining
+markers are accepted (i.e. no quick fixes are selected).
+
+---
+
+Specifying the test cases is an iterative process. It comprises
+
+1) the specification of workspace contents,
+2) the execution of the test case(s),
+3) the verification of the resulting "*.markers" files,
+4) the selection of quick fixes to apply and
+5) the repetition from 2).
+
+Unfortunately the reflective nature of JUnit prevents the re-execution of selected dynamic test cases.
+In the process of specification a test case it may be handy to execute that specific specific test case
+without the entire suite. This can be achieved by just moving the test case folder from the "tests"
+folder into the "test" folder. Later it should be moved back.
+
+---
+
+Open issues:
+
+- The framework does currently not capture the actual files or changes that result from applying the
+  quick fixes. Nor is there currently a means to verify these files or changes.