Bug 423715 - Support multiple algorithms for artifact checksumsI20180301-2000

Developer of p2-based software should be able to check integrity of
artifact checksums using any MessageDigest implementation available,
at his own discretion, without ever touching p2 internals.

For the purpose of this bug, limit enabled checksum algorithms to:
- MD5 (legacy one, deprecated now, for backward compatibility only)
- SHA-256 (more collision-resistant than MD5)

See artifactChecksums.exsd for more details.

First, encode ID of the checksum algorithm into the name of the
property itself so that artifact descriptor now may look like:

    <artifact classifier='binary' id='testKeyId' version='1.2.3'>
      <properties size='6'>
        <property name='download.md5' value='b3788632488d48b850255acf68669651'/>
        <property name='artifact.md5' value='b3788632488d48b850255acf68669651'/>
        <property name='download.checksum.sha-256' value='d594816b995d1689c2dfc97dc244859abe6bdb9ebeb4b396e401afd85a97ee16'/>
        <property name='download.checksum.whirlpool' value='e4f3ece3d3f289cc2686a68f0b1b5a2d03da3a8ccdc3cd6d03209e4c789af724c8fa915bb890079e1abe78df44875cec132885dd6ae1176eed7938dfb3c7b551'/>
        <property name='artifact.checksum.sha-256' value='d594816b995d1689c2dfc97dc244859abe6bdb9ebeb4b396e401afd85a97ee16'/>
        <property name='artifact.checksum.tiger' value='462a72a1a593b9e2de8721b8d79335fd03e974f2e2e1f35a'/>
      </properties>
    </artifact>

Prefix helps to avoid conflicts with other, not checksum-related,
properties. To find checksums of specific type, we need all properties
with prefix 'artifact.checksum.', Extracting checksum id from the name
of the property is also trivial.

Store these prefixes in IArtifactDescriptor's DOWNLOAD_CHECKSUM and
ARTIFACT_CHECKSUM.

ChecksumUtilities provides internal API to deal with properties.

Second, map ID of checksum algorithm to the name of MessageDigest
implementation with a contribution to a new extension point
org.eclipse.equinox.p2.artifact.repository.artifactChecksums:

    <extension point="org.eclipse.equinox.p2.artifact.repository.artifactChecksums">
      <artifactChecksum algorithm="SHA-256"
                        id="sha-256"
			...
                        />
    </extension>

Number of actual checksums in artifact descriptor depends on the
configuration of the application that created such descriptor because
p2 will:
- handle MD5 checksums as usual, preserving backward compatibility
- calculate other checksums using extensions that contribute to
  artifactChecksums extension point

Last, we use id of specific checksum to get MessageDigest instance
with getInstance(String).

There is a number of standalone applications (like MirrorApplication)
and Ant tasks (like ValidateTask) that allow user to chose specific
comparator by id and ArtifactComparatorFactory is responsible for
instantiating it. Legacy MD5 comparator requires no special
configuration while the new ArtifactChecksumComparator is more generic
and needs some parameters to instantiate. ArtifactComparatorFactory
handles this by accepting fake comparator id - concatenated
ArtifactChecksumComparator's id and checksum algorithm id.

In other words, to compare artifacts using SHA-256 algorithm use
'org.eclipse.equinox.artifact.comparator.checksum.sha-256' as
comparator id.

We generate, consume and compare artifact checksums using all enabled
algorithms. Ideally, configuration options should provide more control
for both repository publisher and p2 client like:
- a priority (compare with SHA-512 first, then SHA-256 and, finally,
  MD5)
- skip specific checksum (do not use MD5 even if provided)
- mandatory checksum (require Whirlpool and it's an error if its not
  available)

While I have no good solution for this right now, there are some
configuration options provided. Also, number of new enabled checksum
algorithms is limited to SHA-256 only (more checksums = more reasons
to fail = more reasons to disable).

To turn checksum verification off completely:
- for p2 client, use property 'eclipse.p2.checksums.disable' (see
  SimpleArtifactRepository.CHECKSUMS_ENABLED constant).
- for p2 publisher, the old PublisherInfo's setArtifactOptions(int)
  and IPublisherInfo.A_NO_MD5 still could be used.

To disable MD5 checksums only, existing properties
'eclipse.p2.MD5Check' and 'eclipse.p2.MD5ArtifactCheck' still can be
used.

Change-Id: Iacd267e13d5d096694001d34cafbaea5451e7157
Signed-off-by: Mykola Nikishov <mn@mn.com.ua>

3 files changed, 133 insertions, 29 deletions

diff --git a/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumHelper.java b/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumHelper.java
new file mode 100644
index 000000000..f87747535
--- /dev/null
+++ b/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumHelper.java
@@ -0,0 +1,79 @@
+/*******************************************************************************
+ *  Copyright (c) 2015, 2018 Mykola Nikishov
+ *  All rights reserved. This program and the accompanying materials
+ *  are made available under the terms of the Eclipse Public License v1.0
+ *  which accompanies this distribution, and is available at
+ *  http://www.eclipse.org/legal/epl-v10.html
+ *
+ *  Contributors:
+ *     Mykola Nikishov - initial API and implementation
+ *******************************************************************************/
+package org.eclipse.equinox.internal.p2.repository.helpers;
+
+import java.util.*;
+import java.util.Map.Entry;
+import org.eclipse.equinox.p2.repository.artifact.IArtifactDescriptor;
+
+public class ChecksumHelper {
+
+	private static final String MD5 = "md5"; //$NON-NLS-1$
+
+	/**
+	 * @param property either {@link IArtifactDescriptor#ARTIFACT_CHECKSUM} or {@link IArtifactDescriptor#DOWNLOAD_CHECKSUM}
+	 * @return (mutable) map of <algorithm,checksum>
+	 * @throws IllegalArgumentException if checksum property neither {@link IArtifactDescriptor#ARTIFACT_CHECKSUM} nor {@link IArtifactDescriptor#DOWNLOAD_CHECKSUM}
+	 */
+	static public Map<String, String> getChecksums(IArtifactDescriptor descriptor, String property) throws IllegalArgumentException {
+		if (!IArtifactDescriptor.ARTIFACT_CHECKSUM.equals(property) && !IArtifactDescriptor.DOWNLOAD_CHECKSUM.equals(property))
+			// TODO provide more details
+			throw new IllegalArgumentException();
+
+		Map<String, String> checksumsByAlgo = new HashMap<>();
+
+		String md5Checksum = getLegacyMd5Checksum(descriptor, property);
+		if (md5Checksum != null) {
+			checksumsByAlgo.put(MD5, md5Checksum);
+		}
+
+		// get checksum properties
+		for (Entry<String, String> p : descriptor.getProperties().entrySet()) {
+			String key = p.getKey();
+			if (key.startsWith(property)) {
+				String checksumAlgorithmId = key // "artifact.checksum.sha3"
+						.substring(property.length()) // ".sha3"
+						.substring(1); // "sha3"
+				String checksumValue = Objects.requireNonNull(p.getValue());
+				String duplicatedChecksum = checksumsByAlgo.put(checksumAlgorithmId, checksumValue);
+				if (duplicatedChecksum != null)
+					// should never happen - duplicated checksum
+					;
+			}
+		}
+
+		return checksumsByAlgo;
+	}
+
+	/**
+	 * @return MD5 checksum from legacy property, either {@link IArtifactDescriptor#DOWNLOAD_MD5} or {@link IArtifactDescriptor#ARTIFACT_MD5}
+	 */
+	private static String getLegacyMd5Checksum(IArtifactDescriptor descriptor, String property) {
+		switch (property) {
+			case IArtifactDescriptor.ARTIFACT_CHECKSUM :
+				return descriptor.getProperty(IArtifactDescriptor.ARTIFACT_MD5);
+			case IArtifactDescriptor.DOWNLOAD_CHECKSUM :
+				return descriptor.getProperty(IArtifactDescriptor.DOWNLOAD_MD5);
+			default :
+				return null;
+		}
+	}
+
+	public static String toHexString(byte[] digest) {
+		StringBuilder buf = new StringBuilder();
+		for (byte element : digest) {
+			if ((element & 0xFF) < 0x10)
+				buf.append('0');
+			buf.append(Integer.toHexString(element & 0xFF));
+		}
+		return buf.toString();
+	}
+}
diff --git a/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumProducer.java b/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumProducer.java
index fc8cabec6..c6616eb6c 100644
--- a/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumProducer.java
+++ b/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/internal/p2/repository/helpers/ChecksumProducer.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2009, 2017 IBM Corporation and others.
+ * Copyright (c) 2009, 2018 IBM Corporation and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
@@ -7,51 +7,53 @@
  *
  * Contributors:
  *     IBM Corporation - initial API and implementation
- *     Mykola Nikishov - extract MD5 checksum calculation
+ *     Mykola Nikishov - multiple artifact checksums
  *******************************************************************************/
 
 package org.eclipse.equinox.internal.p2.repository.helpers;
 
 import java.io.*;
-import java.security.MessageDigest;
-import java.security.NoSuchAlgorithmException;
+import java.security.*;
 
+/**
+ * Calculates a checksum using {@link java.security.MessageDigest}
+ */
 public class ChecksumProducer {
 
+	private static final int BUFFER_SIZE = 4 * 1024;
+
 	/**
 	 * @param file should not be <code>null</code>
 	 * @return MD5 checksum of the file or <code>null</code> in case of NoSuchAlgorithmException
 	 * @throws IOException
 	 */
+	@Deprecated
+	// bug #509401 - still here to not break x-friends like in bug #507193
 	public static String computeMD5(File file) throws IOException {
-		InputStream fis = null;
 		try {
-			MessageDigest md5Checker = MessageDigest.getInstance("MD5"); //$NON-NLS-1$
-			fis = new BufferedInputStream(new FileInputStream(file));
-			int read = -1;
-			final int bufferSize = 4 * 1024;
-			byte[] buffer = new byte[bufferSize];
-			while ((read = fis.read(buffer, 0, bufferSize)) != -1) {
-				md5Checker.update(buffer, 0, read);
-			}
-			byte[] digest = md5Checker.digest();
-			StringBuffer buf = new StringBuffer();
-			for (int i = 0; i < digest.length; i++) {
-				if ((digest[i] & 0xFF) < 0x10)
-					buf.append('0');
-				buf.append(Integer.toHexString(digest[i] & 0xFF));
-			}
-			return buf.toString();
+			return produce(file, "MD5"); //$NON-NLS-1$
 		} catch (NoSuchAlgorithmException e) {
 			return null;
-		} finally {
-			if (fis != null)
-				try {
-					fis.close();
-				} catch (IOException e) {
-					// ignore
-				}
 		}
 	}
 
+	/**
+	 * @param file should not be <code>null</code>
+	 * @param algorithm {@link java.security.MessageDigest#getInstance(String)}
+	 * @return checksum of the file
+	 * @throws IOException
+	 * @throws NoSuchAlgorithmException
+	 */
+	public static String produce(File file, String algorithm) throws IOException, NoSuchAlgorithmException {
+		MessageDigest messageDigest = MessageDigest.getInstance(algorithm);
+		try (InputStream fis = new DigestInputStream(new BufferedInputStream(new FileInputStream(file)), messageDigest)) {
+			byte[] buffer = new byte[BUFFER_SIZE];
+			while (fis.read(buffer) != -1) {
+				// consume stream to update digest
+			}
+		}
+		byte[] digest = messageDigest.digest();
+		return ChecksumHelper.toHexString(digest);
+	}
+
 }
diff --git a/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/p2/repository/artifact/IArtifactDescriptor.java b/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/p2/repository/artifact/IArtifactDescriptor.java
index 81557cf03..826e17a59 100644
--- a/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/p2/repository/artifact/IArtifactDescriptor.java
+++ b/bundles/org.eclipse.equinox.p2.repository/src/org/eclipse/equinox/p2/repository/artifact/IArtifactDescriptor.java
@@ -1,5 +1,5 @@
 /*******************************************************************************
- * Copyright (c) 2007, 2010 IBM Corporation and others.
+ * Copyright (c) 2007, 2018 IBM Corporation and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
@@ -7,6 +7,7 @@
  *
  * Contributors:
  *     IBM Corporation - initial API and implementation
+ *     Mykola Nikishov - multiple artifact checksums
  *******************************************************************************/
 package org.eclipse.equinox.p2.repository.artifact;
 
@@ -36,11 +37,22 @@ public interface IArtifactDescriptor {
 	 * bytes of the artifact in its native format (after processing steps have been applied).
 	 */
 	public static final String ARTIFACT_SIZE = "artifact.size"; //$NON-NLS-1$
+
+	/**
+	 * A prefix of an artifact descriptor properties (value "download.checksums") storing checksum
+	 * of the artifact bytes that are transferred.
+	 * @since 2.4.0
+	 */
+	public static final String DOWNLOAD_CHECKSUM = "download.checksum"; //$NON-NLS-1$
+
 	/**
 	 * An artifact descriptor property (value "download.md5") indicating the MD5
 	 * checksum of the artifact bytes that are transferred.
+	 * @see #DOWNLOAD_CHECKSUM
 	 */
+	@Deprecated
 	public static final String DOWNLOAD_MD5 = "download.md5"; //$NON-NLS-1$
+
 	/**
 	 * An artifact descriptor property (value "download.contentType") indicating the 
 	 * content type of the artifact bytes that are transferred.
@@ -50,11 +62,22 @@ public interface IArtifactDescriptor {
 	 * An content type (value "application/zip") indicating the content is a zip file.
 	 */
 	public static final String TYPE_ZIP = "application/zip"; //$NON-NLS-1$
+
+	/**
+	 * An artifact descriptor property (value "artifact.checksums") storing list of
+	 * checksums of the artifact bytes in its native format (after processing steps have
+	 * been applied).
+	 * @since 2.4.0
+	 */
+	public static final String ARTIFACT_CHECKSUM = "artifact.checksum"; //$NON-NLS-1$
+
 	/**
 	 * An artifact descriptor property (value "artifact.md5") indicating the MD5
 	 * checksum of the artifact bytes in its native format (after processing steps have
 	 * been applied).
+	 * @see #ARTIFACT_CHECKSUM
 	 */
+	@Deprecated
 	public static final String ARTIFACT_MD5 = "artifact.md5"; //$NON-NLS-1$
 
 	/**