Bug 423715 - Support multiple algorithms for artifact checksumsI20180301-2000

Developer of p2-based software should be able to check integrity of
artifact checksums using any MessageDigest implementation available,
at his own discretion, without ever touching p2 internals.

For the purpose of this bug, limit enabled checksum algorithms to:
- MD5 (legacy one, deprecated now, for backward compatibility only)
- SHA-256 (more collision-resistant than MD5)

See artifactChecksums.exsd for more details.

First, encode ID of the checksum algorithm into the name of the
property itself so that artifact descriptor now may look like:

    <artifact classifier='binary' id='testKeyId' version='1.2.3'>
      <properties size='6'>
        <property name='download.md5' value='b3788632488d48b850255acf68669651'/>
        <property name='artifact.md5' value='b3788632488d48b850255acf68669651'/>
        <property name='download.checksum.sha-256' value='d594816b995d1689c2dfc97dc244859abe6bdb9ebeb4b396e401afd85a97ee16'/>
        <property name='download.checksum.whirlpool' value='e4f3ece3d3f289cc2686a68f0b1b5a2d03da3a8ccdc3cd6d03209e4c789af724c8fa915bb890079e1abe78df44875cec132885dd6ae1176eed7938dfb3c7b551'/>
        <property name='artifact.checksum.sha-256' value='d594816b995d1689c2dfc97dc244859abe6bdb9ebeb4b396e401afd85a97ee16'/>
        <property name='artifact.checksum.tiger' value='462a72a1a593b9e2de8721b8d79335fd03e974f2e2e1f35a'/>
      </properties>
    </artifact>

Prefix helps to avoid conflicts with other, not checksum-related,
properties. To find checksums of specific type, we need all properties
with prefix 'artifact.checksum.', Extracting checksum id from the name
of the property is also trivial.

Store these prefixes in IArtifactDescriptor's DOWNLOAD_CHECKSUM and
ARTIFACT_CHECKSUM.

ChecksumUtilities provides internal API to deal with properties.

Second, map ID of checksum algorithm to the name of MessageDigest
implementation with a contribution to a new extension point
org.eclipse.equinox.p2.artifact.repository.artifactChecksums:

    <extension point="org.eclipse.equinox.p2.artifact.repository.artifactChecksums">
      <artifactChecksum algorithm="SHA-256"
                        id="sha-256"
			...
                        />
    </extension>

Number of actual checksums in artifact descriptor depends on the
configuration of the application that created such descriptor because
p2 will:
- handle MD5 checksums as usual, preserving backward compatibility
- calculate other checksums using extensions that contribute to
  artifactChecksums extension point

Last, we use id of specific checksum to get MessageDigest instance
with getInstance(String).

There is a number of standalone applications (like MirrorApplication)
and Ant tasks (like ValidateTask) that allow user to chose specific
comparator by id and ArtifactComparatorFactory is responsible for
instantiating it. Legacy MD5 comparator requires no special
configuration while the new ArtifactChecksumComparator is more generic
and needs some parameters to instantiate. ArtifactComparatorFactory
handles this by accepting fake comparator id - concatenated
ArtifactChecksumComparator's id and checksum algorithm id.

In other words, to compare artifacts using SHA-256 algorithm use
'org.eclipse.equinox.artifact.comparator.checksum.sha-256' as
comparator id.

We generate, consume and compare artifact checksums using all enabled
algorithms. Ideally, configuration options should provide more control
for both repository publisher and p2 client like:
- a priority (compare with SHA-512 first, then SHA-256 and, finally,
  MD5)
- skip specific checksum (do not use MD5 even if provided)
- mandatory checksum (require Whirlpool and it's an error if its not
  available)

While I have no good solution for this right now, there are some
configuration options provided. Also, number of new enabled checksum
algorithms is limited to SHA-256 only (more checksums = more reasons
to fail = more reasons to disable).

To turn checksum verification off completely:
- for p2 client, use property 'eclipse.p2.checksums.disable' (see
  SimpleArtifactRepository.CHECKSUMS_ENABLED constant).
- for p2 publisher, the old PublisherInfo's setArtifactOptions(int)
  and IPublisherInfo.A_NO_MD5 still could be used.

To disable MD5 checksums only, existing properties
'eclipse.p2.MD5Check' and 'eclipse.p2.MD5ArtifactCheck' still can be
used.

Change-Id: Iacd267e13d5d096694001d34cafbaea5451e7157
Signed-off-by: Mykola Nikishov <mn@mn.com.ua>

6 files changed, 152 insertions, 24 deletions

diff --git a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/ArtifactChecksumComparator.java b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/ArtifactChecksumComparator.java
new file mode 100644
index 000000000..0c156e0cb
--- /dev/null
+++ b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/ArtifactChecksumComparator.java
@@ -0,0 +1,62 @@
+/*******************************************************************************
+ *  Copyright (c) 2015, 2018 Mykola Nikishov.
+ *  All rights reserved. This program and the accompanying materials
+ *  are made available under the terms of the Eclipse Public License v1.0
+ *  which accompanies this distribution, and is available at
+ *  http://www.eclipse.org/legal/epl-v10.html
+ *
+ *  Contributors:
+ *      Mykola Nikishov - initial API and implementation
+ *******************************************************************************/
+package org.eclipse.equinox.p2.internal.repository.comparator;
+
+import org.eclipse.core.runtime.*;
+import org.eclipse.equinox.internal.p2.artifact.repository.Activator;
+import org.eclipse.equinox.internal.p2.repository.helpers.ChecksumHelper;
+import org.eclipse.equinox.p2.internal.repository.tools.Messages;
+import org.eclipse.equinox.p2.repository.artifact.IArtifactDescriptor;
+import org.eclipse.equinox.p2.repository.artifact.IArtifactRepository;
+import org.eclipse.equinox.p2.repository.tools.comparator.IArtifactComparator;
+import org.eclipse.osgi.util.NLS;
+
+/**
+ * A comparator that compares two artifacts by checking the checksum
+ * recorded in the artifact descriptor. This comparator doesn't actually compute
+ * checksums directly.
+ */
+final public class ArtifactChecksumComparator implements IArtifactComparator {
+	final public static String COMPARATOR_ID = "org.eclipse.equinox.artifact.comparator.checksum"; //$NON-NLS-1$
+
+	final private String name;
+	final private String id;
+
+	/**
+	 * @param checksumId
+	 * @param checksumName human-readable name of the checksum algorithm
+	 */
+	public ArtifactChecksumComparator(String checksumId, String checksumName) {
+		this.name = checksumName;
+		this.id = checksumId;
+	}
+
+	@Override
+	final public IStatus compare(IArtifactRepository source, IArtifactDescriptor sourceDescriptor, IArtifactRepository destination, IArtifactDescriptor destDescriptor) {
+		String sourceChecksum = ChecksumHelper.getChecksums(sourceDescriptor, IArtifactDescriptor.DOWNLOAD_CHECKSUM).get(id);
+		String destChecksum = ChecksumHelper.getChecksums(destDescriptor, IArtifactDescriptor.DOWNLOAD_CHECKSUM).get(id);
+
+		if (sourceChecksum == null && destChecksum == null)
+			return new Status(IStatus.INFO, Activator.ID, NLS.bind(Messages.info_noChecksumInfomation, name, sourceDescriptor));
+
+		if (sourceChecksum == null)
+			return new Status(IStatus.INFO, Activator.ID, NLS.bind(Messages.info_noChecksumInRepository, new Object[] {source, name, sourceDescriptor}));
+
+		if (destChecksum == null)
+			return new Status(IStatus.INFO, Activator.ID, NLS.bind(Messages.info_noChecksumInRepository, new Object[] {destination, name, destDescriptor}));
+
+		if (sourceChecksum.equals(destChecksum))
+			return Status.OK_STATUS;
+
+		return new Status(IStatus.WARNING, Activator.ID, NLS.bind(Messages.warning_different_checksum, new Object[] {URIUtil.toUnencodedString(sourceDescriptor.getRepository().getLocation()), URIUtil.toUnencodedString(destDescriptor.getRepository().getLocation()), name, sourceDescriptor}));
+	}
+
+}
diff --git a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/MD5ArtifactComparator.java b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/MD5ArtifactComparator.java
index a83e931f3..dc0ee1c73 100644
--- a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/MD5ArtifactComparator.java
+++ b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/comparator/MD5ArtifactComparator.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- *  Copyright (c) 2008, 2017 IBM Corporation and others.
+ *  Copyright (c) 2008, 2018 IBM Corporation and others.
  *  All rights reserved. This program and the accompanying materials
  *  are made available under the terms of the Eclipse Public License v1.0
  *  which accompanies this distribution, and is available at
@@ -23,7 +23,14 @@ import org.eclipse.osgi.util.NLS;
  * A comparator that compares two artifacts by checking the MD5 checksum
  * recorded in the artifact descriptor. This comparator doesn't actually compute MD5
  * checksums directly.
+ *
+ * @deprecated
+ * @noreference
+ * @noextend
+ * @noinstantiate
+ * @see ArtifactChecksumComparator
  */
+@Deprecated
 public class MD5ArtifactComparator implements IArtifactComparator {
 
 	public static String MD5_COMPARATOR_ID = "org.eclipse.equinox.artifact.md5.comparator"; //$NON-NLS-1$
diff --git a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/Messages.java b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/Messages.java
index 47cc723c1..648cd795f 100644
--- a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/Messages.java
+++ b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/Messages.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2009, 2010 IBM Corporation and others.
+ * Copyright (c) 2009, 2018 IBM Corporation and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
@@ -67,6 +67,10 @@ public class Messages extends NLS {
 	public static String info_noMD5InRepository;
 	public static String warning_differentMD5;
 
+	public static String info_noChecksumInfomation;
+	public static String info_noChecksumInRepository;
+	public static String warning_different_checksum;
+
 	static {
 		// initialize resource bundles
 		NLS.initializeMessages(BUNDLE_NAME, Messages.class);
diff --git a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/RecreateRepositoryApplication.java b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/RecreateRepositoryApplication.java
index 0c56617b9..a56fdb73d 100644
--- a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/RecreateRepositoryApplication.java
+++ b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/RecreateRepositoryApplication.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2009, 2017 IBM Corporation and others.
+ * Copyright (c) 2009, 2018 IBM Corporation and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
@@ -8,18 +8,18 @@
  * Contributors:
  *     IBM Corporation - initial API and implementation
  *     Sonatype Inc - ongoing development
+ *     Mykola Nikishov - multiple artifact checksums
  *******************************************************************************/
 
 package org.eclipse.equinox.p2.internal.repository.tools;
 
 import java.io.File;
-import java.io.IOException;
 import java.net.URI;
 import java.util.*;
 import org.eclipse.core.runtime.*;
+import org.eclipse.equinox.internal.p2.artifact.processors.checksum.ChecksumUtilities;
 import org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository;
 import org.eclipse.equinox.internal.p2.core.helpers.LogHelper;
-import org.eclipse.equinox.internal.p2.repository.helpers.ChecksumProducer;
 import org.eclipse.equinox.p2.core.ProvisionException;
 import org.eclipse.equinox.p2.metadata.IArtifactKey;
 import org.eclipse.equinox.p2.query.IQueryResult;
@@ -31,6 +31,7 @@ import org.eclipse.equinox.p2.repository.artifact.spi.ProcessingStepDescriptor;
 import org.eclipse.osgi.util.NLS;
 
 public class RecreateRepositoryApplication extends AbstractApplication {
+	private static final String MD5_CHECKSUM_ID = "md5"; //$NON-NLS-1$
 	static final private String PUBLISH_PACK_FILES_AS_SIBLINGS = "publishPackFilesAsSiblings"; //$NON-NLS-1$
 	private URI repoLocation;
 	private String repoName = null;
@@ -121,15 +122,18 @@ public class RecreateRepositoryApplication extends AbstractApplication {
 				newDescriptor.setProperty(IArtifactDescriptor.ARTIFACT_SIZE, size);
 				newDescriptor.setProperty(IArtifactDescriptor.DOWNLOAD_SIZE, size);
 
-				try {
-					String md5 = ChecksumProducer.computeMD5(artifactFile);
-					if (md5 != null)
-						newDescriptor.setProperty(IArtifactDescriptor.DOWNLOAD_MD5, md5);
-				} catch (IOException e) {
-					// don't care if failed to compute checksum
-					// TODO provide message?
-					LogHelper.log(new Status(IStatus.WARNING, Activator.ID, "", e));
-				}
+				Map<String, String> checksums = new HashMap<>();
+				IStatus status = ChecksumUtilities.calculateChecksums(artifactFile, checksums, Collections.emptyList());
+				if (!status.isOK())
+					// TODO handle errors in some way
+					LogHelper.log(status);
+
+				String md5 = checksums.get(MD5_CHECKSUM_ID);
+				if (md5 != null)
+					// preserve legacy MD5 checksum location
+					newDescriptor.setProperty(IArtifactDescriptor.DOWNLOAD_MD5, md5);
+
+				newDescriptor.addProperties(ChecksumUtilities.checksumsToProperties(IArtifactDescriptor.DOWNLOAD_CHECKSUM, checksums));
 
 				File temp = new File(artifactFile.getParentFile(), artifactFile.getName() + ".pack.gz"); //$NON-NLS-1$
 				if (temp.exists()) {
diff --git a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/messages.properties b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/messages.properties
index aef9767a3..097b86b86 100644
--- a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/messages.properties
+++ b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/internal/repository/tools/messages.properties
@@ -1,5 +1,5 @@
 ###############################################################################
-# Copyright (c) 2009, 2010 IBM Corporation and others.
+# Copyright (c) 2009, 2018 IBM Corporation and others.
 # All rights reserved. This program and the accompanying materials
 # are made available under the terms of the Eclipse Public License v1.0
 # which accompanies this distribution, and is available at
@@ -15,9 +15,9 @@ CompositeRepository_composite_repository_exists=Composite repository already exi
 CompositeRepository_default_artifactRepo_name=Composite Artifact Repository
 CompositeRepository_default_metadataRepo_name=Composite Artifact Repository
 
-info_noMD5Infomation=No MD5 information available for the artifact [{0}].
-info_noMD5InRepository=The repository {0} does not contain MD5 information for artifact [{1}].
-warning_differentMD5=The repositories {0} and {1} have different MD5 sums for the artifact [{2}].
+info_noChecksumInfomation=No {0} information available for the artifact [{1}].
+info_noChecksumInRepository=The repository {0} does not contain {1} information for artifact [{2}].
+warning_different_checksum=The repositories {0} and {1} have different MessageDigest={2} sums for the artifact [{3}].
 
 no_artifactRepo_manager=Unable to acquire artifact repository manager service.
 no_metadataRepo_manager=Unable to acquire metadata repository manager service.
diff --git a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/repository/tools/comparator/ArtifactComparatorFactory.java b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/repository/tools/comparator/ArtifactComparatorFactory.java
index 68a249bbc..47074aca1 100644
--- a/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/repository/tools/comparator/ArtifactComparatorFactory.java
+++ b/bundles/org.eclipse.equinox.p2.repository.tools/src/org/eclipse/equinox/p2/repository/tools/comparator/ArtifactComparatorFactory.java
@@ -8,14 +8,17 @@
  *  Contributors:
  *     IBM Corporation - initial API and implementation
  *     Compeople AG (Stefan Liebig) - various ongoing maintenance
- *     Mykola Nikishov - maintenance and documentation
+ *     Mykola Nikishov - multiple artifact checksums
  *******************************************************************************/
 package org.eclipse.equinox.p2.repository.tools.comparator;
 
-import java.util.*;
+import java.util.List;
+import java.util.Optional;
 import java.util.stream.Collectors;
+import java.util.stream.Stream;
 import org.eclipse.core.runtime.IConfigurationElement;
 import org.eclipse.core.runtime.RegistryFactory;
+import org.eclipse.equinox.internal.p2.artifact.processors.checksum.ChecksumUtilities;
 import org.eclipse.equinox.internal.p2.artifact.repository.Messages;
 import org.eclipse.equinox.p2.internal.repository.comparator.*;
 import org.eclipse.osgi.util.NLS;
@@ -35,20 +38,43 @@ public class ArtifactComparatorFactory {
 	 * <li><code>org.eclipse.equinox.artifact.md5.comparator</code> for {@link MD5ArtifactComparator}</li>
 	 * </ul>
 	 *
+	 * For {@link ArtifactChecksumComparator}, there is no static id that could be used directly. Instead, its extension point id <code>org.eclipse.equinox.artifact.comparator.checksum</code> should be concatenated with id of desired checksum algorithm:
+	 * <ul>
+	 * <li><code>org.eclipse.equinox.artifact.comparator.checksum.sha-256</code> to use artifact checksum comparator with SHA-256 message digest</li>
+	 * <li><code>org.eclipse.equinox.artifact.comparator.checksum.tiger</code> to use Tiger message digest</li>
+	 * </ul>
+	 *
 	 * @param comparatorID id of the extension contributed to <code>org.eclipse.equinox.p2.artifact.repository.artifactComparators</code> extension point
 	 * @return if found, instance of artifact comparator
 	 * @throws {@link IllegalArgumentException} otherwise
 	 */
 	public static IArtifactComparator getArtifactComparator(String comparatorID) {
-		List<IConfigurationElement> extensions = Arrays.asList(RegistryFactory.getRegistry().getConfigurationElementsFor(COMPARATOR_POINT));
+		List<IConfigurationElement> extensions = Stream.of(RegistryFactory.getRegistry().getConfigurationElementsFor(COMPARATOR_POINT))
+			.collect(Collectors.toList());
+		// exclude artifact checksum comparator, needs special care
+		extensions.removeIf(extension -> extension.getAttribute(ATTR_ID).equals(ArtifactChecksumComparator.COMPARATOR_ID));
 
-		Optional<IArtifactComparator> artifactComparator = findComparatorConfiguration(comparatorID, extensions).map(ArtifactComparatorFactory::createArtifactComparator);
+		// handle generic artifact comparators
+		Optional<IArtifactComparator> artifactComparator = findComparatorConfiguration(comparatorID, extensions)
+			.map(ArtifactComparatorFactory::createArtifactComparator);
 		if (artifactComparator.isPresent())
 			return artifactComparator.get();
 
+		// handle artifact checksum comparator
+		if (comparatorID != null && comparatorID.startsWith(ArtifactChecksumComparator.COMPARATOR_ID)) {
+			Optional<ArtifactChecksumComparator> checksumComparator = getChecksumComparator(comparatorID);
+			if (checksumComparator.isPresent())
+				return checksumComparator.get();
+		}
+
 		if (comparatorID != null) {
-			String comparators = extensions.stream().map(extension -> extension.getAttribute(ATTR_ID)).collect(Collectors.joining(", ")); //$NON-NLS-1$
-			throw new IllegalArgumentException(NLS.bind(Messages.exception_comparatorNotFound, comparatorID, comparators));
+			Stream<String> checksumComparators = Stream.of(ChecksumUtilities.getChecksumComparatorConfigurations())
+				.map(element -> element.getAttribute(CHECKSUM_ID))
+				.map(checksumId -> ArtifactChecksumComparator.COMPARATOR_ID.concat(".").concat(checksumId)); //$NON-NLS-1$
+			Stream<String> comparators = extensions.stream()
+				.map(extension -> extension.getAttribute(ATTR_ID));
+			String availableComparators = Stream.concat(comparators, checksumComparators).collect(Collectors.joining(", ")); //$NON-NLS-1$
+			throw new IllegalArgumentException(NLS.bind(Messages.exception_comparatorNotFound, comparatorID, availableComparators));
 		}
 		throw new IllegalArgumentException(Messages.exception_noComparators);
 	}
@@ -71,4 +97,29 @@ public class ArtifactComparatorFactory {
 		}
 		return null;
 	}
+
+	private static final String CHECKSUM_ID = "id"; //$NON-NLS-1$
+	private static final String CHECKSUM_ALGORITHM = "algorithm"; //$NON-NLS-1$
+
+	private static Optional<ArtifactChecksumComparator> getChecksumComparator(String comparatorId) {
+		boolean hasNoChecksumId = !comparatorId.startsWith(ArtifactChecksumComparator.COMPARATOR_ID.concat(".")); //$NON-NLS-1$
+		if (hasNoChecksumId)
+			return Optional.empty();
+
+		String comparatorChecksum = Optional.ofNullable(comparatorId.substring(ArtifactChecksumComparator.COMPARATOR_ID.length() + 1))
+			.orElse(""); //$NON-NLS-1$
+		if (comparatorChecksum.isEmpty())
+			return Optional.empty();
+
+		IConfigurationElement[] comparatorConfigurations = ChecksumUtilities.getChecksumComparatorConfigurations();
+		Optional<IConfigurationElement> comparator = Stream.of(comparatorConfigurations)
+			.filter(config -> config.getAttribute(CHECKSUM_ID).equals(comparatorChecksum))
+			.findAny();
+
+		return comparator.map(c -> {
+			String checksumName = c.getAttribute(CHECKSUM_ALGORITHM);
+			String checksumId = c.getAttribute(CHECKSUM_ID);
+			return Optional.of(new ArtifactChecksumComparator(checksumId, checksumName));
+		}).orElse(Optional.empty());
+	}
 }