WTP Builds
Last Updated: October 20, 2004

This document describes the build policy and how to build WTP components from the command line using Ant scripts.

Table of Contents

Build Policy
Tools
Setup
Run the Build
Ant properties defined build.properties
Build Infrastructure
CVS Structure
Map File
Generating source features and plug-ins at build time
Release Team

Build Policy

The process described here is modeled after other eclipse build processes, and borrows general principles from other sources, some of which are described in detail by the document http://www.martinfowler.com/articles/continuousIntegration.html.

General principles:
  1. Automatic and continuos: A build must be a fully automated and reproducible, must include quality assurance (i.e. testing), and must be able to run as often as possible. Automatic build processes are more reliable and higher quality than manually and ensure that most obvious errors during a build. It is less risky, it is independent of a human builder, it is repeatable and communicable. It is typically faster than a manual build. Finally it allows for built-in QA process by integration tests and post-processing such as potentially running metrics on the code. A build must be a fully automated and reproducible, must include quality assurance (i.e. testing), and must be able to run as often as possible.that runs many times a day. This imposes some constraints: Storage - CVS Provides a common place where all code lives and anyone can obtain the current sources from. Scripting: Use a single command to build the system from the sources. An automated build process and scripting suitable for eclipse projects is described in http://dev.eclipse.org/viewcvs/index.cgi/~checkout~/org.eclipse.releng.basebuilder/readme.html?rev=1.21&content-type=text/plain. We should follow similar principles as they have already build the tools and the scripts to automate eclipse builds. Testing: Automate the testing so that you can run a good suite of tests on the system at any time with a single command. A frequency of minimum one build a day seems often enough for geographically and logistically challenging projects such as WTP. It is also important to describe what will happen when a build fails. Typically, it is assumed that sending an email will suffice as an action. This is not so. A build must succeed!
  2. A build must succeed, it should not be an attempt. A build is a success if:
    • All the sources are checked out of the configuration management system as described by the .map (not necessarily the latest) files. Every file is compiled from scratch The resulting object files (Java classes in our case) and linked and deployed for execution (put into jars).
    • The system is started and suite of tests (JUnit and PDE JUnits Tests) is run against the system. If all of these steps execute without error or human intervention and every test passes, then we have a successful build
    • It is expected that all "smooth" builds can be managed this way. The problem is when things go wrong. WTP release engineering team intervenes with the process for managing build errors!
  3. Build Management: An automated build provides the most return when it can produce a master build with as little human intervention as possible. Although there are arguments that there should be a human in the loop for a build, it should not be the act of producing a build. Build automation makes a lot of sense for individual developers, they can test a build before it is "checked in" to CVS. But the most critical activity of human intervention is to facilitate a procedure to bring the team together in resolving integration and build problems. WTP is a geographically and logistically challenging so (put the team together to fix problems). Build manager is responsible for the builds/infrastructure (scripts and environment) so a build can run. S/he facilitates to bring the teams/people together to fix build errors (sending automated emails from scripts is not sufficient). Make sure that when there are problems, there is a process describing every stage to fix it, work out what went wrong when the process fails. Make sure the mind of the process captured in automated script code, rather than people's head so that it can be dealt with minimal interaction the next time. Manage major builds and release process (For major integration builds (milestones), release candidates and releases.
  4. Keeping in sync: With an automated build we should integrate regularly (once a week perhaps - with an integration build). For an integration build to be meaningful, there must be a critical mass of code checked-in. Being able to run the master build locally will also give developers a chance to minimize build errors and they will have to do less at integration time.
Tools used to manage the build process

1. CVS version 1.10 or higher.
2. 1.4 level JDK.
3. Info-Zip zip and unzip
4. org.eclipse.releng.basebuilder and org.eclipse.releng.eclipsebuilder from dev.eclipse.org.
5. Ant optional tasks (ftp) This task depends on external libraries not included in the Ant distribution. See ant library dependencies for ftp, rexec and telnet tasks
for more information.

For more information on eclipse release engineering tool, refer to the plugin itself.

For examples, refer to the org.eclips.releng.eclipsebuilder itself. Readme

Setup

  1. Add the zip, unzip, and cvs executables to your PATH environment variable.
  2. Add the jre/bin directory of the installed JDK to your PATH environment variable.
  3. Add the ant/bin directory of the installed Ant to your PATH environment variable.
  4. Create a directory for the build, e.g.
    • mkdir /tmp/wtpbuild
  5. Install the eclipse builder (this installation will be used to run the build tools, it is not itself a WebTools installation): Check out org.eclipse.releng.basebuilder and org.eclipse.releng.basebuilder from the CVS repository /home/eclipse (note: not /home/webtools) into the build directory, e.g.
    • cd /tmp/wtpbuild
      cvs -d :pserver:anonymous@dev.eclipse.org:/home/eclipse co org.eclipse.releng.basebuilder
      cvs -d :pserver:anonymous@dev.eclipse.org:/home/eclipse co org.eclipse.releng.eclipsebuilder
          
  6. Checkout org.eclipse.wtp.releng into the build directory.
    • cvs -d :pserver:anonymous@dev.eclipse.org:/home/webtools co org.eclipse.wtp.releng
  7. You have also to adjust some build properties in the org.eclipse.wtp.releng/buildAll.properties (see below).
Run the Build

Run the Build

cd to org.eclipse.wtp.releng directory and execute the following command. Alternatively, you can checkout the module org.eclipse.wtp.releng in an eclipse workspace and use the ant run tools.

"ant -f cruise.xml"

There are three ant targets in cruise.xml (an ant build file): build, publish

 

Ant Target Description
all Default. Builds WTP from scratch (all plugins and features), runs tests, and publishes to a ftp site. For ftp push two propserties ftpUser and ftpPassowrd must be defined.
build Builds WTP from scratch (all plugins and features), does not run tests, and does not publish to a ftp site.
publish Must have run all, or build with no "clean" option prior to running this task. It creates drop files and uploads them to an ftp site. For ftp push two propserties ftpUser and ftpPassowrd must be defined.

[Ant properties to override in buildAll.properties] ". To override a property you must use a syntax like "ant -f cruise.xml -Dclean=true"


Ant Properties Settings in buildAll.properties

The following properties are pre-defined in the buildAll.properties file for the build. They can be overridden by setting them at the command line at build time or by changing them directly in the component's build.properties file.

Name Description
baseLocation

A directory separate from buildDirectory which contains pre-built plugins and features against which to compile. The basedirectory must not contain any features, plugins or fragments which are already or will be located in the buildDirectory (see below).

Build script unzips previosly downloaded eclipse, plugisn and features to use for the build. This configuration is maintained in build.cfg.

baseLocationRoot A directory that hosts the current built. baseLocation be created under this folder.
localDownloads A directory that contains pre-downloaded zip files defined in the build configuration.
bootclasspath Sets the value for the attribute "bootclasspath" in calls to the Ant <javac> task in a plugins' build.xml. Default set to ${java.home}/lib/rt.jar.
buildDirectory The absolute path to a working directory where the source for the build will be exported, where scripts will be generated and where the end products of the build will be located. On Windows systems, the path length should not exceed thirty characters due to path length limitations when compiling some classes in eclipse.
buildId The build name. Default set to "build".
buildLabel Refers to the name of the directory which will contain the end result of the build. Default set to ${buildType}.${buildId}, ie."I.build". This directory will be created inside the location specified by the ${buildDirectory} property.
buildType

Letters I, N, S, R or M are used in Eclipse builds to identify builds as being one of the following:

I - Integration
N - Nightly
S - Stable
R - Release
M - Maintenance

If set to N, all source will be checked out from the HEAD stream. In all other cases, tags as specifed in map files will be used when exporting plugins to the buildDirectory.

archivePrefix The name of the top level directory which should exist in the produced zip file. Typically set to "eclipse".
collectingFolder The name of the top level directory where the built features and plugins will be gathered. This is typically set to "eclipse".
configs

An ampersand separated list of configurations to build where a configuration is specified as <os>,<ws>,<arch>.
ie.configs="win32,win32,x86 & linux, motif, x86 & linux, gtk, x86". See list of build configurations below.

javacDebugInfo Sets the value for the attribute "debug" in calls to the Ant <javac> task in a plugins' build.xml. Default set to on.
javacFailOnError Sets the value for the attribute "failonerror" in calls to the Ant <javac> task in a plugins' build.xml. Build will continue even if there are compilation errors when this is set to false.
javacSource Sets the value for the attribute "source" in calls to the Ant <javac> task in a plugins' build.xml. Default set to 1.3.
javacTarget Sets the value for the attribute "target" in calls to the Ant <javac> task in a plugins' build.xml. Default set to 1.1.
javacVerbose Sets the value for the attribute "verbose" in calls to the Ant <javac> task in a plugins' build.xml. Default set to true.
mapVersionTag

Sets the tag attribute in a call to the Ant <cvs> task to check out org.eclipse.releng, the map file project on dev.eclipse.org used in the build.

Can be set to a specific tag to download the map files used for a previous integration build. Typically, these tags are in the form "v<date/timestamp>" for example "v200307110800" will checkout the map files used to run the integration at the specified date/time. These scripts work with 3.0 stream builds > 20030701.

timestamp A timestamp used to fill in value for buildid in about.mappings files. The timestamp is also used to tag the org.eclipse.releng project on dev.eclipse.org only when an appropriate value for mapCvsRoot is provided and when the tagMaps Ant property is set.
eclipseBuilderDirectory A directory separate from buildDirectory and baseLocation, which contains eclipse releng base builder, plugins and features that is used to run the PDE build tools and locate the template scripts.
cvsRoot CVS root (i.e. /home/webtools)
cvsServer Address of the CVS server (i.e. dev.eclipse.org)
cvsProtocol CVS protocol (i.e. pserver, local)
cvsUser User id used for CVS authentication. You will need to run cvs -d <cvs-root> login prior to build so that authentication information is cached.
ftpUser ftp user id
ftpPassword ftp password
ftpServer Address of the ftp server (i.e. download.eclipse.org)
ftpRemoteDirectory Directory that will be used to upload the build folder.
publishDirectory A local directory that is used to keep a copy of the final build artifacts .
clean Existence of this property causes the build script to delete all working folders after completion. It's value is not used.
zipargs Arguments to send to the zip executable. ie. zipargs=-y on Linux preserves symbolic links.

The following properties are pre-defined in the build.cfg file for the build.
Name Description
baseos,basews,basearch The os, ws and arch values of a pre-built eclipse component being compiled against. See list of possible values in the table of build configurations.
eclipseURL
Eclipse download URL. If you haven't got any eclipse-*.SDK.zip in your download build-home/downloads dir. Download it from this url.
eclipseFile Eclipse zip file which will be used for building project
emfURL EMF download URL. If you haven't got any emf*.zip in your download build-home/downloads dir. Download it from this url.
emfFile EMF zip file which will be used for building project
Build Infrastructure
The automated build process currently runs on a Redhat Linux based build machine hosted at eteration a.s. Build process is run and monitored by cruise control
CVS Structure
The CVS structure uses a granular approach to to break up the WTP project into its subprojects and each subproject into its its main components to achieve the following benefits:

1.Organize the large number of contributions and teams into manageable pieces.
2.Organize the code into smaller collections so they are easier to understand.
3. Make it more extensible for future projects and components.

Each component group in a subproject implies ownership of a chunk of functionality by a group of committers (i.e. server, sse).

/home/webtools CVS ROOT
  /wst J2EE Standard Tools Subproject
    /components Each component group in a subproject implies ownership of a chunk of functionality by a group of committers
      /sse Structured Source Editor framework
        /plugins    
          org.eclipse.wst.sse.core The sse plugin(s)
        /features    
          org.eclipse.wst.sse-feature The sse feature(s)
        /tests    
          org.eclipse.wst.sse-tests sse test plugin(s)
  /jst  
How to Create Map File
A .map file is a java property file which contains mappings of elements to their CVS locations and access methods. Map files are used by PDE Build early in the build process to generate Ant scripts which use the Ant <cvs> task to export source to a directory. This is described further below.
Map file entries use the following format:
feature|fragment|plugin@element.Id=<cvs tag>,<access method>:<cvs user>@<cvs repository>,<cvs password>[,<repository path> (no starting slash) ]

The <repository path> is only required when the module (or directory) containing the source for the element does not match the element.Id or if the directory is not at the root of the repository.

A map file entry must exist for each feature being built, it's <plugin> elements and it's <includes> elements (ie. nested features and their plug-ins). Adding a plug-in or fragment to a feature therefore requires updating the map files with the new element.

Map File Entry Examples
 
  • The org.eclipse.platform plug-in source is located at the root of the dev.eclipse.org:/home/eclipse repository. It is contained in a directory with the name "org.eclipse.platform". The resulting map file entry for this plug-in is:

    plugin@org.eclipse.platform=v20031121,:pserver:anonymous@dev.eclipse.org:/home/eclipse,

  • The org.eclipse.platform feature source is located at the root of the dev.eclipse.org:/home/eclipse repository in a directory with the name "org.eclipse.platform-feature". The resulting map file entry for this feature is:

    feature@org.eclipse.platform=v20031128,:pserver:anonymous@dev.eclipse.org:/home/eclipse,,org.eclipse.platform-feature

  • The org.eclipse.gef.sdk feature source is located in directory in the dev.eclipse.org:/home/tools repository in a subdirectory called "org.eclipse.gef.sdk" of "org.eclipse-gef feature". The resulting map file entry for this feature is:

    feature@org.eclipse.gef.sdk=I_20031117, :pserver:anonymous@dev.eclipse.org:/home/tools,,org.eclipse.gef-feature/org.eclipse.gef.sdk

One or more map files can be used to list the elements. The map files can be kept under version control. Some examples of map file projects include org.eclipse.releng, org.eclipse.gef.releng, org.eclipse.ve.releng.

Generating source features and plug-ins at build time

Source features and plug-ins can be generated at build time by PDE Build. Source features and associated source plug-ins are typically generated for a development kit distributions (i.e. SDK). It is also possible to generate a source plug-in only. This is typically the case for example features or JUnit testing features.

To generate a source feature and associated source plug-in at build time, you will need to do the following:

  1. Add an entry to the build.properties file in the feature project for which you wish to include the source feature and plug-in. The generated source feature should also be listed in the feature.xml as an <includes> element.

    The build.properties entry should use the following format:

         generate.feature@<source.feature.id to generate>=<feature.id from which to which to collect source>, plugin@<plugin.id>

    Example taken from org.eclipse.sdk-feature/build.properties:
         generate.feature@org.eclipse.jdt.source=org.eclipse.jdt, plugin@org.eclipse.jdt.doc.isv

    In this example, a source feature and a plug-in, both with id "org.eclipse.jdt.source" will be generated and will contain source from plug-ins listed in the org.eclipse.jdt feature and will also include the plug-in org.eclipse.jdt.doc.isv. The generated org.eclipse.jdt.source plug-in will be automatically listed in the org.eclipse.jdt.source feature.xml.

  2. In the feature project from which the source feature will be generated, a directory called "sourceTemplateFeature" and a directory called "sourceTemplatePlugin" will be required. These directories should contain the files that are included in the root of the generated source feature and plug-in. The feature.xml and plugin.xml files are not required since these are generated. A build.properties is required in the sourceTemplatePlugin directory. This should contain a "bin.includes" setting as well as the entry "sourcePlugin = true". The plugin.xml file and src/ directory should be listed in the bin.includes.

    See org.eclipse.jdt-feature and org.eclipse.platform-feature for examples.

To generate a source plug-in only at build time, you will need to do the following:

  1. Add an entry to the build.properties file in the feature project for which you wish to include the source plug-in. The generated source plug-in should also be listed in the feature.xml as a <plugin> element..

    The build.properties entry should use the following format:

    generate.plugin@<source.plugin.id to generate>=<feature.id from which to which to collect source>, plugin@<plugin.id>

    Example taken from org.eclipse.ve.tests-feature/build.properties:
         generate.plugin@org.eclipse.ve.tests.source=org.eclipse.ve.tests

  2. In the runtime feature project from which the source plug-in will be generated, create a directory called "sourceTemplatePlugin" which must contain a build.properties with a "bin.includes" setting and "sourcePlugin=true". The plugin.xml file and src/ directory should be listed in the bin.includes.

    See org.eclipse.ve.examples-feature/sourceTemplatePlugin for an example.

WTP Release Engineering Team
Ozgur Tumer (ozgur.tumer_at_eteration.com)
Dominique Devito (dominique.devito_at_objectweb.org)