Skip to main content
aboutsummaryrefslogtreecommitdiffstats
blob: 5d915cae198b3d7151c0575b8ee1f56b0e802743 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
The Papyrus project is built using [http://www.eclipse.org/buckminster/ Buckminster], with this releng project:
http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.papyrus/trunk/releng/org.eclipse.mdt.papyrus.releng.buckminster
(only for the trunk; the branches are built with the corresponding project from the branch).

This project contains sub-folders containing the Buckminster configuration for each Hudson job. 

== Hudson Jobs ==
The Papyrus builds are started from Hudson. The following jobs exist currently:
* papyrus-trunk-nightly : builds the main plug-ins on the trunk of Papyrus
* papyrus-trunk-nightly-tests : installs the latest Papyrus nightly build and runs the unit tests (on Eclipse 4.x)
* papyrus-trunk-nightly-3.8-tests : installs the latest Papyrus nightly build and runs the unit tests (on Eclipse 3.x)
* papyrus-trunk-extra-nightly : builds the "extra" plug-ins of Papyrus (on the trunk) 
* papyrus-trunk-extra-nightly-tests (not implemented yet): installs the latest Papyrus nightly build with extras and runs the unit tests for the extra plug-ins
* papyrus-0.8-maintenance-nightly : builds the main plug-ins on the branch 0.8 of Papyrus
* papyrus-0.8-maintenance-nightly-tests : installs the latest Papyrus nightly build for branch 0.8 and runs the unit tests
* papyrus-0.8-maintenance-extra-nightly : builds the "extra" plug-ins of Papyrus (on branch 0.8) 
* papyrus-0.8-maintenance-extra-nightly-tests (not implemented yet): installs the latest Papyrus nightly build with extras (branch 0.8) and runs the unit tests for the extra plug-ins

See all the currently defined Papyrus jobs on this page : https://hudson.eclipse.org/hudson/user/vlorenzo/my-views/view/Papyrus

== What is built? ==

Taking the example of "papyrus-trunk-nightly", one feature is built:
* org.eclipse.papyrus.build.feature

It is referenced as a root dependency in <code>buckminster.cspec</code>.

This feature must include all other features that must be built, either directly or indirectly.

The releng project contains a Buckminster rmap, which specifies how to get the plug-ins and features that are to be built. For each plug-in or feature, the map defines its location on a version control system (CVS, SVN, Git, etc.).

The rmap also specifies the update sites from which to retrieve the binary dependencies from the dependent projects.

=== Updating the rmap ===
The rmap reflects the dependencies on the other projects. So, it must be updated at least before each milestone, to make sure that the code that is built works with the latest versions of all the dependencies.

The easiest way to update the dependencies is to copy them from the aggregation build model, which all projects must fill in with the correct reference to their update site(s) and feature(s).

This model is in a project with the release name on the Eclipse CVS:
* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.juno.build/?root=Callisto "org.eclipse.<releaseName>.build"] on <code>dev.eclipse.org:/cvsroot/callisto</code>, where releaseName is helios or indigo or juno etc. depending on the version being built.

There is a tool that copies most of the information automatically in this plug-in:
* http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/releng/org.eclipse.emf.facet.releng.tools (see the [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/releng/org.eclipse.emf.facet.releng.tools/README.TXT README.TXT])

'''Warning''': don't forget to update the aggregation model project before copying information from it

A few dependencies are not in the aggregation build model though, so they must be updated manually:
* '''Orbit''' : you can find the latest version listed on : http://download.eclipse.org/tools/orbit/downloads/

==== Finding dependencies ====
If you don't know where a particular plug-in or feature is located, the easiest way to find it is to search the Eclipse download area using "find" on build.eclipse.org.
For example, to look for "org.eclipse.emf.compare" :
 find /home/data/httpd/download.eclipse.org/modeling/emf/ -name 'org.eclipse.emf.compare_*'

== How to start a build? ==

=== Automatically ===

* Nightly builds are run automatically every night at 00:43 every day.

=== Manually ===

Only committers in the Papyrus project can launch Papyrus build jobs from Hudson.

Go to one of these pages, depending on the build you want to launch:

* Trunk nightly build : https://hudson.eclipse.org/hudson/job/papyrus-trunk-nightly/
* Trunk extras nightly build : https://hudson.eclipse.org/hudson/job/papyrus-trunk-extra-nightly/
* Trunk tests nightly build on 4.x : https://hudson.eclipse.org/hudson/job/papyrus-trunk-nightly-tests/
* Trunk tests nightly build on 3.x : https://hudson.eclipse.org/hudson/job/papyrus-trunk-nightly-3.8-tests/
* Trunk extra tests nightly build : https://hudson.eclipse.org/hudson/job/papyrus-trunk-extra-nightly-tests/
* Branch 0.8 nightly build : https://hudson.eclipse.org/hudson/job/papyrus-0.8-maintenance-nightly
* Branch 0.8 extra nightly build : https://hudson.eclipse.org/hudson/job/papyrus-0.8-maintenance-extra-nightly/
* Branch 0.8 tests nightly build : https://hudson.eclipse.org/hudson/job/papyrus-0.8-maintenance-nightly-tests/
* Branch 0.8 extra tests nightly build : https://hudson.eclipse.org/hudson/job/papyrus-0.8-maintenance-extra-nightly-tests/

Then:
* In Hudson, click on '''Build Now''', change the build parameters as needed (see [[#Build parameters]]), and click on '''Build'''.
* You can then click on the job name in the ''Build History'' section in the left column, and then on '''Console Output''', to follow build progress in real time.

== How to publish a build? ==
=== Automatically ===
Successful '''N'''ightly builds are automatically published to download.eclipse.org by a [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.papyrus/trunk/releng/org.eclipse.mdt.papyrus.releng.buckminster/serverConfig/cronPromote.sh cron job] that runs on build.eclipse.org. For example, a papyrus-trunk-nightly build for version 0.9.0, created on April 21 2011 at 04:13 would be published to:
* the download drops: http://download.eclipse.org/modeling/mdt/papyrus/downloads/drops/0.9.0/N201104210413
* the nightly update site: http://download.eclipse.org/modeling/mdt/papyrus/updates/nightly/juno/main

These builds can then be seen and downloaded from http://www.eclipse.org/modeling/mdt/papyrus/downloads/, where additional information is available (test results, build log), or installed from the update sites:
* http://download.eclipse.org/modeling/mdt/papyrus/updates/nightly/juno (composite update site)
* http://download.eclipse.org/modeling/mdt/papyrus/updates/nightly/juno/main
* http://download.eclipse.org/modeling/mdt/papyrus/updates/nightly/juno/extras

=== With the script ===

To publish a milestone or release from a trunk build, you can use the manualPromote.sh script on: '''/opt/public/modeling/mdt/papyrus/manualPromote.sh'''
Start this script, and fill in the parameters in the interactive script. For example:

 $ cd /opt/public/modeling/mdt/papyrus/
 $ ./manualPromote.sh
 -------------------- user parameters --------------------
 mainBuildNumber (the number of the "papyrus-trunk-nightly" Hudson build from which to publish the main Papyrus plug-ins):
 ? 519
 extrasBuildNumber (the number of the "papyrus-trunk-extra-nightly" Hudson build from which to publish the extra Papyrus plug-ins, or 0 to not publish):
 ? 43
 testsBuildNumber (the number of the "papyrus-trunk-nightly-tests" Hudson build from which to publish the test results, or 0 to not publish):
 ? 93
 version (e.g. "0.9.0"):
 ? 0.9.0
 updateSite (e.g. "nightly/juno", "milestones/0.9/M5", "releases/indigo/0.8.1") :
 ? milestones/0.9/M5
 Are you sure you want to publish with these parameters (yes|no)?
 ? yes

In the previous example, the script will:
* publish the main Papyrus plug-ins from papyrus-trunk-nightly build #519 to the Papyrus download in "drops/0.9.0" and the update site in "milestones/0.9/M5/main"
* publish the extra plug-ins from papyrus-trunk-extra-nightly build #43 to the Papyrus download in "drops/0.9.0" in the same build folder as the main build, and the update site in "milestones/0.9/M5/extra"
* publish the test results from papyrus-trunk-nightly-tests build #93 to the Papyrus download in "drops/0.9.0" in the same build folder as the main build
* make "milestones/0.9/M5" a composite update site, which contains "milestones/0.9/M5/main" and "milestones/0.9/M5/extra"
* enable p2 download statistics both on "milestones/0.9/M5/main" and "milestones/0.9/M5/extra"
* set the correct linux access rights on the new folders

Once the script is done, all you have left to do is to:
* [[#Tagging|Tag the build]]
* [[#Simultaneous_Release|update the aggregation model]] if this build must be part of the simultaneous release

=== Manually ===
'''S'''table, '''M'''aintenance and '''R'''elease builds are not automatically published. They should be first tested internally before publishing. For example, to publish the 0.1.0M6 milestone build:
* First, fetch the build archive to test (from Hudson, or using <code>wget</code> or <code>scp</code> for example):
 wget https://hudson.eclipse.org/hudson/job/papyrus-trunk-nightly/lastSuccessfulBuild/artifact/S201103151256.zip
or, if https access doesn't work, a NFS access can be used (from build.eclipse.org). For example:
 cp /shared/jobs/papyrus-trunk-nightly/lastSuccessful/archive/S201103151256.zip .
* Then, test the build locally
* Then, publish it:
 ssh <commiterid>@build.eclipse.org
 wget https://hudson.eclipse.org/hudson/job/papyrus-trunk-nightly/lastSuccessfulBuild/artifact/S201103151256.zip
 unzip S201103151256.zip -d /home/data/httpd/download.eclipse.org/modeling/mdt/papyrus/downloads/drops/0.1.0/
Add a new update site with the new build to the composite:
* unzip the content (the update site zip that was in the first zip) in a new folder with the release number:
 unzip Papyrus-Update-0.1.0M6.zip -d /home/data/httpd/download.eclipse.org/modeling/mdt/papyrus/updates/milestones/0.1/M6
* update both the <code>compositeContent.xml</code> and <code>compositeArtifacts.xml</code> files of the update site (that are located in the parent of the folder to which you extracted the update site) to add a reference to your newly added update site
** set the value of p2.timestamp to the result of "<code>date +%s000</code>"
** increase the "size" attribute of the children element
** add a "child" element inside the "children" element with a "location" set to the name of the folder (e.g "M6")
* You can enable download stats on the repository by running [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.papyrus/trunk/releng/org.eclipse.mdt.papyrus.releng.buckminster/serverConfig/addDownloadStats.sh /opt/public/modeling/mdt/papyrus/addDownloadStats.sh] on the update site
* Check that the new build appears on [http://www.eclipse.org/modeling/mdt/papyrus/downloads/ http://www.eclipse.org/modeling/mdt/papyrus/downloads/].
* Builds can be hidden from this page before a release by modifying <code>downloads-scripts.php</code> in <code>www/modeling/mdt/papyrus/downloads/</code> on <code>:pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse</code>
* [[#Tagging|Tag the build]]

=== Simultaneous Release ===
If the build must be part of the simultaneous release, you must also:
* Use the [http://wiki.eclipse.org/Eclipse_b3/aggregator/manual B3 Aggregator] (or a text editor if the modification is trivial) to [http://wiki.eclipse.org/Juno/Contributing_to_Juno_Build update the build model for the aggregator] ([http://wiki.eclipse.org/Git after checking it out from git] the path is : ssh://username@git.eclipse.org/gitroot/simrel/org.eclipse.simrel.build.git)
Usually, the only thing to change is the location (for example "milestones/0.9/M5/main"):
 <repositories location="http://download.eclipse.org/modeling/mdt/papyrus/updates/milestones/0.9/M5/main" description="Papyrus">

== Build parameters ==
Hudson builds expect these parameters:
* BUILDTYPE : the kind of build, represented by a code letter (see [http://www.eclipse.org/modeling/downloads/build-types.php this page]):
** '''N''': Nightly
** '''I''': Integration
** '''M''': Maintenance ('''NOT''' milestone)
** '''S''': Stable (for Milestones and Release Candidate builds)
** '''R''': Release
* BUILD_TARGET :
**test : build Papyrus, create the update site, and launch the JUnit tests on the result
**site.p2 : Launch the build of Papyrus only - no tests
**provision : Gets only the material required to launch the build
* BUILD_ALIAS : the name of the release (0.8.0, 0.9.0M1, etc.); leave it blank for nightly and integration builds
* SIGN_UPDATE_SITE : whether to sign the update site (can take up to an hour on the Eclipse build server); mandatory for all builds except nightlies
* VERSION : the version being built. Dictates in which folder the build will be placed under the download "drops" folder.

== Tagging ==
Releases should have a tag like '''0.9.0''', and milestones should have a tag like '''0.9.0_M7'''.

== Checking ==

You can execute the following Bash snippets to check the compliance of different aspects.

=== Check bundles ===
Check that each bundle contains an about.html file:
 for f in $( ls *.jar ); do unzip -t $f | grep -q about.html || echo $f; done
or for plug-ins in your workspace:
 find $workspaceRoot -name 'build.properties' | while read i; do grep -q about.html "$i" || echo "$i"; done

=== Check licenses ===
md5 is useful to see quickly which files differ:
 find $workspaceRoot -type f -name 'license.html' | while read i; do md5 $i; done

=== Check Manifest.MF ===
 find $workspaceRoot -name 'MANIFEST.MF' | while read i; do grep -q Bundle-Vendor "$i" || echo "$i"; done
 find $workspaceRoot -name 'feature.xml' | while read i; do grep -q provider-name "$i" || echo "$i"; done

=== Check simrel reports ===
 for url in \
    "http://build.eclipse.org/juno/simrel/reports/layoutCheck.txt" \
    "http://build.eclipse.org/juno/simrel/reports/verifydiroutput/unsigned.txt" \
    "http://build.eclipse.org/juno/simrel/reports/versionPatternCheck.txt" \
    "http://build.eclipse.org/juno/simrel/reports/breedata.txt" \
    "http://build.eclipse.org/juno/simrel/reports/pack200data.txt" \
    "http://build.eclipse.org/juno/simrel/reports/nonUniqueVersions.txt"
 do
   echo "checking $url"
   curl -s -S "$url" | egrep 'papyrus'
 done

 url="http://build.eclipse.org/juno/simrel/reports/licenseConsistency.html"
 echo "checking $url"
 curl -s -S $url > licenseConsistency
 n=$(cat licenseConsistency | grep -n "Features with matching" | sed 's/:.*//')
 cat licenseConsistency | head -n $n | egrep 'papyrus'

 url="http://build.eclipse.org/juno/simrel/reports/featureNames.html"
 echo "checking $url"
 curl -s -S $url > featureNames
 n=$(cat featureNames | grep -n "Probably correct names" | sed 's/:.*//')
 cat featureNames | head -n $n | egrep 'papyrus'

 url="http://build.eclipse.org/juno/simrel/reports/bundleNames.html"
 echo "checking $url"
 curl -s -S $url > bundleNames
 n=$(cat bundleNames | grep -n "Probably correct bundle name" | sed 's/:.*//')
 cat bundleNames | head -n $n | egrep 'papyrus'

 url="http://build.eclipse.org/juno/simrel/reports/providerNames.html"
 echo "checking $url"
 curl -s -S $url > providerNames
 n=$(cat providerNames | grep -n "Probably using correct provider name" | sed 's/:.*//')
 cat providerNames | head -n $n | egrep 'papyrus'

 url="http://build.eclipse.org/juno/simrel/reports/copyrights.html"
 echo "checking $url"
 curl -s -S $url > copyrights
 n=$(cat copyrights | grep -n "Features with copyrights that are probably ok" | sed 's/:.*//')
 cat copyrights | head -n $n | egrep 'papyrus'

 url="http://build.eclipse.org/juno/simrel/reports/esdata.txt"
 echo "checking $url"
 curl -s -S $url > esdata
 n=$(cat esdata | grep -n "Bundles without an Eclipse-SourceReference" | sed 's/:.*//')
 cat esdata | tail -n +$n | egrep 'papyrus'

Back to the top