Bug 423715 - Support multiple algorithms for artifact checksumsI20180301-2000

Developer of p2-based software should be able to check integrity of
artifact checksums using any MessageDigest implementation available,
at his own discretion, without ever touching p2 internals.

For the purpose of this bug, limit enabled checksum algorithms to:
- MD5 (legacy one, deprecated now, for backward compatibility only)
- SHA-256 (more collision-resistant than MD5)

See artifactChecksums.exsd for more details.

First, encode ID of the checksum algorithm into the name of the
property itself so that artifact descriptor now may look like:

    <artifact classifier='binary' id='testKeyId' version='1.2.3'>
      <properties size='6'>
        <property name='download.md5' value='b3788632488d48b850255acf68669651'/>
        <property name='artifact.md5' value='b3788632488d48b850255acf68669651'/>
        <property name='download.checksum.sha-256' value='d594816b995d1689c2dfc97dc244859abe6bdb9ebeb4b396e401afd85a97ee16'/>
        <property name='download.checksum.whirlpool' value='e4f3ece3d3f289cc2686a68f0b1b5a2d03da3a8ccdc3cd6d03209e4c789af724c8fa915bb890079e1abe78df44875cec132885dd6ae1176eed7938dfb3c7b551'/>
        <property name='artifact.checksum.sha-256' value='d594816b995d1689c2dfc97dc244859abe6bdb9ebeb4b396e401afd85a97ee16'/>
        <property name='artifact.checksum.tiger' value='462a72a1a593b9e2de8721b8d79335fd03e974f2e2e1f35a'/>
      </properties>
    </artifact>

Prefix helps to avoid conflicts with other, not checksum-related,
properties. To find checksums of specific type, we need all properties
with prefix 'artifact.checksum.', Extracting checksum id from the name
of the property is also trivial.

Store these prefixes in IArtifactDescriptor's DOWNLOAD_CHECKSUM and
ARTIFACT_CHECKSUM.

ChecksumUtilities provides internal API to deal with properties.

Second, map ID of checksum algorithm to the name of MessageDigest
implementation with a contribution to a new extension point
org.eclipse.equinox.p2.artifact.repository.artifactChecksums:

    <extension point="org.eclipse.equinox.p2.artifact.repository.artifactChecksums">
      <artifactChecksum algorithm="SHA-256"
                        id="sha-256"
			...
                        />
    </extension>

Number of actual checksums in artifact descriptor depends on the
configuration of the application that created such descriptor because
p2 will:
- handle MD5 checksums as usual, preserving backward compatibility
- calculate other checksums using extensions that contribute to
  artifactChecksums extension point

Last, we use id of specific checksum to get MessageDigest instance
with getInstance(String).

There is a number of standalone applications (like MirrorApplication)
and Ant tasks (like ValidateTask) that allow user to chose specific
comparator by id and ArtifactComparatorFactory is responsible for
instantiating it. Legacy MD5 comparator requires no special
configuration while the new ArtifactChecksumComparator is more generic
and needs some parameters to instantiate. ArtifactComparatorFactory
handles this by accepting fake comparator id - concatenated
ArtifactChecksumComparator's id and checksum algorithm id.

In other words, to compare artifacts using SHA-256 algorithm use
'org.eclipse.equinox.artifact.comparator.checksum.sha-256' as
comparator id.

We generate, consume and compare artifact checksums using all enabled
algorithms. Ideally, configuration options should provide more control
for both repository publisher and p2 client like:
- a priority (compare with SHA-512 first, then SHA-256 and, finally,
  MD5)
- skip specific checksum (do not use MD5 even if provided)
- mandatory checksum (require Whirlpool and it's an error if its not
  available)

While I have no good solution for this right now, there are some
configuration options provided. Also, number of new enabled checksum
algorithms is limited to SHA-256 only (more checksums = more reasons
to fail = more reasons to disable).

To turn checksum verification off completely:
- for p2 client, use property 'eclipse.p2.checksums.disable' (see
  SimpleArtifactRepository.CHECKSUMS_ENABLED constant).
- for p2 publisher, the old PublisherInfo's setArtifactOptions(int)
  and IPublisherInfo.A_NO_MD5 still could be used.

To disable MD5 checksums only, existing properties
'eclipse.p2.MD5Check' and 'eclipse.p2.MD5ArtifactCheck' still can be
used.

Change-Id: Iacd267e13d5d096694001d34cafbaea5451e7157
Signed-off-by: Mykola Nikishov <mn@mn.com.ua>

6 files changed, 228 insertions, 3 deletions

diff --git a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/AllTests.java b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/AllTests.java
index aeaca5a72..92dac347c 100644
--- a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/AllTests.java
+++ b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/AllTests.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2007, 2008 compeople AG and others.
+ * Copyright (c) 2007, 2018 compeople AG and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
@@ -21,6 +21,7 @@ public class AllTests extends TestCase {
 		TestSuite suite = new TestSuite(AllTests.class.getName());
 		suite.addTestSuite(Pack200ProcessorTest.class);
 		suite.addTestSuite(ZipVerifierProcessorTest.class);
+		suite.addTest(new JUnit4TestAdapter(ChecksumVerifierTest.class));
 		return suite;
 	}
 
diff --git a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/ChecksumVerifierTest.java b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/ChecksumVerifierTest.java
new file mode 100644
index 000000000..c71515105
--- /dev/null
+++ b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/processors/ChecksumVerifierTest.java
@@ -0,0 +1,114 @@
+/*******************************************************************************
+ * Copyright (c) 2015, 2018 Mykola Nikishov.
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v1.0
+ * which accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ *
+ * Contributors:
+ * 	Mykola Nikishov - initial API and implementation
+ *******************************************************************************/
+package org.eclipse.equinox.p2.tests.artifact.processors;
+
+import static org.easymock.EasyMock.*;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import org.eclipse.core.runtime.Status;
+import org.eclipse.equinox.internal.p2.artifact.processors.checksum.ChecksumVerifier;
+import org.eclipse.equinox.p2.repository.artifact.IArtifactDescriptor;
+import org.eclipse.equinox.p2.repository.artifact.IProcessingStepDescriptor;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+@RunWith(Parameterized.class)
+public class ChecksumVerifierTest {
+	@Parameters
+	public static Collection<Object[]> generateData() {
+		return Arrays.asList(new Object[][] {
+			{"MD5", "md5", IArtifactDescriptor.DOWNLOAD_CHECKSUM.concat(".md5"), IArtifactDescriptor.ARTIFACT_CHECKSUM.concat(".md5"), "123456789_123456789_123456789_12"},
+			{"SHA-256", "sha-256", IArtifactDescriptor.DOWNLOAD_CHECKSUM.concat(".sha-256"), IArtifactDescriptor.ARTIFACT_CHECKSUM.concat(".sha-256"), "123456789_123456789_123456789_123456789_123456789_123456789_1234"}});
+	}
+
+	@Parameter(0)
+	public String digestAlgorithm;
+	@Parameter(1)
+	public String algorithmId;
+	@Parameter(2)
+	public String downloadProperty;
+	@Parameter(3)
+	public String artifactProperty;
+	@Parameter(4)
+	public String checksum;
+
+	@Test
+	public void testInitialize() throws IOException, IllegalArgumentException, SecurityException {
+		IProcessingStepDescriptor processingStepDescriptor = createMock(IProcessingStepDescriptor.class);
+		expect(processingStepDescriptor.getData()).andReturn(checksum);
+		expect(processingStepDescriptor.isRequired()).andReturn(true);
+		replay(processingStepDescriptor);
+
+		ChecksumVerifier verifier = new ChecksumVerifier(digestAlgorithm, algorithmId, checksum);
+
+		verifier.initialize(null, processingStepDescriptor, null);
+
+		Assert.assertEquals(Status.OK_STATUS, verifier.getStatus());
+
+		verifier.close();
+		verify(processingStepDescriptor);
+	}
+
+	@Test
+	public void testInitialize_DownloadChecksum() throws IOException, IllegalArgumentException, SecurityException {
+		IProcessingStepDescriptor processingStepDescriptor = createMock(IProcessingStepDescriptor.class);
+		expect(processingStepDescriptor.getData()).andReturn(downloadProperty);
+		expect(processingStepDescriptor.isRequired()).andReturn(true);
+		IArtifactDescriptor artifactDescriptor = createMock(IArtifactDescriptor.class);
+		replay(processingStepDescriptor);
+		expect(artifactDescriptor.getProperty(eq(downloadProperty))).andReturn(checksum);
+		expect(artifactDescriptor.getProperty(not(eq(downloadProperty)))).andReturn(null).times(1, 2);
+		HashMap<String, String> properties = new HashMap<>();
+		properties.put(downloadProperty, checksum);
+		expect(artifactDescriptor.getProperties()).andReturn(properties);
+		replay(artifactDescriptor);
+
+		ChecksumVerifier verifier = new ChecksumVerifier(digestAlgorithm, algorithmId, checksum);
+
+		verifier.initialize(null, processingStepDescriptor, artifactDescriptor);
+
+		Assert.assertEquals(Status.OK_STATUS, verifier.getStatus());
+
+		verifier.close();
+		verify(processingStepDescriptor);
+	}
+
+	@Test
+	public void testInitialize_ArtifactChecksum() throws IOException, IllegalArgumentException, SecurityException {
+		IProcessingStepDescriptor processingStepDescriptor = createMock(IProcessingStepDescriptor.class);
+		expect(processingStepDescriptor.getData()).andReturn(artifactProperty);
+		expect(processingStepDescriptor.isRequired()).andReturn(true);
+		IArtifactDescriptor artifactDescriptor = createMock(IArtifactDescriptor.class);
+		replay(processingStepDescriptor);
+		expect(artifactDescriptor.getProperty(eq(artifactProperty))).andReturn(checksum);
+		HashMap<String, String> properties = new HashMap<>();
+		properties.put(artifactProperty, checksum);
+		expect(artifactDescriptor.getProperties()).andReturn(properties);
+		expect(artifactDescriptor.getProperty(not(eq(artifactProperty)))).andReturn(null).times(1, 2);
+		replay(artifactDescriptor);
+
+		ChecksumVerifier verifier = new ChecksumVerifier(digestAlgorithm, algorithmId, checksum);
+
+		verifier.initialize(null, processingStepDescriptor, artifactDescriptor);
+
+		Assert.assertEquals(Status.OK_STATUS, verifier.getStatus());
+
+		verifier.close();
+		verify(processingStepDescriptor);
+	}
+}
diff --git a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/AllTests.java b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/AllTests.java
index 348b1b170..d2697791c 100644
--- a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/AllTests.java
+++ b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/AllTests.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2008, 2011 Code 9 and others. All rights reserved. This
+ * Copyright (c) 2008, 2018 Code 9 and others. All rights reserved. This
  * program and the accompanying materials are made available under the terms of
  * the Eclipse Public License v1.0 which accompanies this distribution, and is
  * available at http://www.eclipse.org/legal/epl-v10.html
@@ -34,6 +34,7 @@ public class AllTests extends TestCase {
 		suite.addTestSuite(LocalizationTests.class);
 		suite.addTestSuite(LocalUpdateSiteActionTest.class);
 		suite.addTestSuite(MD5GenerationTest.class);
+		suite.addTest(new JUnit4TestAdapter(ChecksumGenerationTest.class));
 		suite.addTestSuite(ProductActionTest.class);
 		suite.addTestSuite(ProductActionCapturingTest.class);
 		suite.addTestSuite(ProductActionTestMac.class);
diff --git a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/ChecksumGenerationTest.java b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/ChecksumGenerationTest.java
new file mode 100644
index 000000000..e6d3cca2e
--- /dev/null
+++ b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/publisher/ChecksumGenerationTest.java
@@ -0,0 +1,57 @@
+/*******************************************************************************
+ *  Copyright (c) 2015, 2018 Mykola Nikishov.
+ *  All rights reserved. This program and the accompanying materials
+ *  are made available under the terms of the Eclipse Public License v1.0
+ *  which accompanies this distribution, and is available at
+ *  http://www.eclipse.org/legal/epl-v10.html
+ *
+ *  Contributors:
+ *     Mykola Nikishov - multiple artifact checksums
+ *******************************************************************************/
+package org.eclipse.equinox.p2.tests.publisher;
+
+import java.util.Arrays;
+import java.util.Collection;
+import org.eclipse.equinox.internal.p2.metadata.ArtifactKey;
+import org.eclipse.equinox.p2.metadata.Version;
+import org.eclipse.equinox.p2.repository.artifact.IArtifactDescriptor;
+import org.eclipse.equinox.p2.tests.AbstractProvisioningTest;
+import org.eclipse.equinox.spi.p2.publisher.PublisherHelper;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+@RunWith(Parameterized.class)
+public class ChecksumGenerationTest extends AbstractProvisioningTest {
+	@Parameter(0)
+	public String checksumProperty;
+	@Parameter(1)
+	public String checksumValue;
+
+	@Parameters
+	public static Collection<Object[]> generateChecksums() {
+		return Arrays.asList(new Object[][] {{IArtifactDescriptor.DOWNLOAD_MD5, "50d4ea58b02706ab373a908338877e02"}, {IArtifactDescriptor.DOWNLOAD_CHECKSUM.concat(".sha-256"), "11da2dd636ab76f460513cbcbfe8c56a6e5ad47aa9b38b36c6d04f8ee7722252"}});
+	}
+
+	@Test
+	public void testGenerationFile() {
+		IArtifactDescriptor ad = PublisherHelper.createArtifactDescriptor(new ArtifactKey("classifierTest", "idTest", Version.createOSGi(1, 0, 0)), getTestData("Artifact to generate from", "testData/artifactRepo/simpleWithMD5/plugins/aaPlugin_1.0.0.jar"));
+		assertEquals(String.format("%s checksum property", checksumProperty), checksumValue, ad.getProperty(checksumProperty));
+	}
+
+	@Test
+	public void testGenerationFolder() {
+		IArtifactDescriptor ad = PublisherHelper.createArtifactDescriptor(new ArtifactKey("classifierTest", "idTest", Version.createOSGi(1, 0, 0)), getTestData("Artifact to generate from", "testData/artifactRepo/simpleWithMD5/plugins/"));
+		assertEquals(String.format("%s checksum property", checksumProperty), null, ad.getProperty(checksumProperty));
+	}
+
+	@Test
+	public void testGenerationNoFolder() {
+		IArtifactDescriptor ad = PublisherHelper.createArtifactDescriptor(new ArtifactKey("classifierTest", "idTest", Version.createOSGi(1, 0, 0)), null);
+		Assert.assertThat(ad.getProperty(checksumProperty), CoreMatchers.not(CoreMatchers.containsString(checksumValue)));
+	}
+}
diff --git a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/AllTests.java b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/AllTests.java
index f01588cda..852cb758f 100644
--- a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/AllTests.java
+++ b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/AllTests.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2009, 2012 Cloudsmith Inc and others.
+ * Copyright (c) 2009, 2018 Cloudsmith Inc and others.
  * All rights reserved. This program and the accompanying materials 
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
@@ -23,6 +23,7 @@ public class AllTests extends TestCase {
 		suite.addTestSuite(RepositoryHelperTest.class);
 		suite.addTestSuite(RepositoryExtensionPointTest.class);
 		suite.addTestSuite(FileReaderTest2.class);
+		suite.addTest(new JUnit4TestAdapter(ChecksumHelperTest.class));
 		return suite;
 	}
 }
diff --git a/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/ChecksumHelperTest.java b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/ChecksumHelperTest.java
new file mode 100644
index 000000000..7eeddfb6e
--- /dev/null
+++ b/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/repository/ChecksumHelperTest.java
@@ -0,0 +1,51 @@
+/*******************************************************************************
+ *  Copyright (c) 2015, 2018 Mykola Nikishov.
+ *  All rights reserved. This program and the accompanying materials
+ *  are made available under the terms of the Eclipse Public License v1.0
+ *  which accompanies this distribution, and is available at
+ *  http://www.eclipse.org/legal/epl-v10.html
+ *
+ *  Contributors:
+ *     Mykola Nikishov - initial API and implementation
+ ******************************************************************************/
+package org.eclipse.equinox.p2.tests.repository;
+
+import java.util.*;
+import org.eclipse.equinox.internal.p2.metadata.ArtifactKey;
+import org.eclipse.equinox.internal.p2.metadata.OSGiVersion;
+import org.eclipse.equinox.internal.p2.repository.helpers.ChecksumHelper;
+import org.eclipse.equinox.p2.repository.artifact.IArtifactDescriptor;
+import org.eclipse.equinox.p2.repository.artifact.spi.ArtifactDescriptor;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+@RunWith(Parameterized.class)
+public class ChecksumHelperTest {
+	@Parameters
+	public static Collection<Object[]> generateData() {
+		return Arrays.asList(new Object[][] {{IArtifactDescriptor.ARTIFACT_CHECKSUM}, {IArtifactDescriptor.DOWNLOAD_CHECKSUM}});
+	}
+
+	@Parameter(0)
+	public String property;
+
+	@Test
+	public void testGetChecksums() {
+		String checksumId = "checksumAlgo";
+		String checksumValue = "value";
+		ArtifactDescriptor descriptor = new ArtifactDescriptor(new ArtifactKey("", "", new OSGiVersion(1, 1, 1, "")));
+		descriptor.setProperty(property.concat(".").concat(checksumId), checksumValue);
+		descriptor.setProperty("download.size", "1234");
+		Map<String, String> expectedChecksums = new HashMap<>();
+		expectedChecksums.put(checksumId, checksumValue);
+
+		Map<String, String> checksums = ChecksumHelper.getChecksums(descriptor, property);
+
+		Assert.assertEquals(expectedChecksums, checksums);
+	}
+
+}