Annotation processors may implement either the Java 5 com.sun.mirror.apt interfaces or the Java 6 javax.annotation.processing interfaces. The two interfaces are functionally similar but the details are different. Note that although the Java 5 apt tool has been deprecated by Sun, it is still included in the Java 6 JDK and developers continue to write processors against it. However, the com.sun.mirror.apt interface is considered proprietary and there are no Java compliance tests which validate it.
The JDT system performs three kinds of build: full build, incremental build, and reconcile. When a single file is edited and saved (if autobuild is enabled) or manually built, an incremental build is performed: only the affected files will be rebuilt. This will include the edited file as well as any other files with relevant dependencies on the edited file. After a clean, or when incremental build detects an overwhelming number of files needing to be rebuilt, a full build will be performed; in this case every file in the project will be rebuilt. During an editing session, prior to saving, a file will frequently be reconciled, in order to identify errors while typing. Reconcile operates on only a single file at a time.
Java 5 and Java 6 processors are handled by different code and interact with the Eclipse Java compiler in different ways and at different points in the build cycle. This is partly for historical reasons and partly in order to support Java 6 processors in the context of the command-line Eclipse compiler (ecj.jar). The Java 5 annotation processing code is built against the JDT's public API; the Java 6 code uses internal interfaces within the compiler, which was necessary because the public APIs use platform classes not available in the command-line package. A pleasant consequence is that the Java 6 annotation processing code is somewhat easier to understand.
The main challenge of introducing APT into Eclipse is that annotation processors can contribute additional Java types, which must in turn be compiled; this compilation can affect previously compiled types, because a formerly unresolved reference can be resolved. Because of this, it is necessary for annotation processing to be implemented within the compile process, rather than as a later build step.
TODO: overview of build process, i.e., when Java 5 processors are called relative to Java 6 processors
The invocation path for Java 5 processors is quite complex, in part because when Java 5 processing was implemented it was not possible to integrate it closely with the compiler, and in part for silly historical reasons that serve no current purpose. It is ripe for refactoring.
A typical call stack when a Java 5 processor is invoked looks like this:
SomeProcessor.process() line: xxx APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628 APTDispatchRunnable.runAPTInFileBasedMode(BuildEnv) line: 317 APTDispatchRunnable.build(BuildEnv) line: 655 APTDispatchRunnable.access$1(APTDispatchRunnable, BuildEnv) line: 647 APTDispatchRunnable$1.run(AbstractCompilationEnv) line: 265 BuildEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 611 CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766 CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478 ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737 BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856 BuildEnv.createASTs(BuildContext[]) line: 356 AbstractCompilationEnv.newBuildEnv(BuildContext[], BuildContext[], IJavaProject, AbstractCompilationEnv$EnvCallback) line: 111 APTDispatchRunnable.build() line: 271 APTDispatchRunnable.run(IProgressMonitor) line: 217 Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 1800 APTDispatchRunnable.runAPTDuringBuild(BuildContext[], BuildContext[], Map<IFile,CategorizedProblem[]>, AptProject, Map<AnnotationProcessorFactory,Attributes>, Set<AnnotationProcessorFactory>, boolean) line: 142 AptCompilationParticipant.processAnnotations(BuildContext[]) line: 193 IncrementalImageBuilder(AbstractImageBuilder).processAnnotations(CompilationParticipantResult[]) line: 627 IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 338 IncrementalImageBuilder.build(SimpleLookupTable) line: 134 JavaBuilder.buildDeltas(SimpleLookupTable) line: 265 JavaBuilder.build(int, Map, IProgressMonitor) line: 193 [higher calls omitted for brevity]
Java 5 processors are invoked via the CompilationParticipant interface. The Java compiler calls AptCompilationParticipant.processAnnotations(), passing in a list of all the files being built. Note that the compiler may break large projects into multiple groups of files, resulting in multiple calls to processAnnotations(). Files are marked with whether or not they contain annotations.
Within processAnnotations(), annotation processor factories are discovered from the processor factory path (and cached for subsequent invocations); factories and files are then passed in to APTDispatchRunnable.runAPTDuringBuild(). This in turn stores the information in a new instance of APTDispatchRunnable, and invokes its run() method within a workspace.run() operation.
The run() method invokes BuildEnv.newBuildEnv(), passing in the AptDispatchRunnable as a callback. The newBuildEnv() method creates a BuildEnv, gets compilation units for each of the files being compiled, and then requests ASTs for each of the files. The parser is passed an ASTRequestor callback that is implemented by an inner class of the BuildEnv; in this way, when the ASTs are ready for consumption, APTDispatchRunnable.build(BuildEnv) is ultimately called, within the ASTRequestor callback.
Finally, within build(), processors are selected on the basis of the annotations they support, and each processor's process() method is called on each appropriate file, configuring the BuildEnv before each process() invocation so that it will provide the correct data. The details of this vary depending on whether the processor is normal (file-based) or is marked as requiring batch mode.
Processors can be marked in the Advanced Factory Path Options dialog as requiring batch mode. If any of the processors on the project's factory path require batch mode, AptDispatchRunnable.build(BuildEnv) calls runAPTInMixedMode(); if none of the processors requires batch mode, runAPTInFileBasedMode() is called instead. The chief difference is that batch-mode processors are only run during a full build, and that the batch-mode implementation of AnnotationProcessorEnvironment.getTypeDeclarations() returns all the compilation units in the build, whereas the normal implementation returns only a single compilation unit at a time.
Additionally, a separate classloader is used to load batch-mode processors, and it is discarded after each build; thus the classes are reloaded for each build, and static variables are reinitialized. This is important because some processors written with command-line (non-incremental) compilation in mind store dynamic state in static variables and thus cannot be invoked multiple times.
Most processors are designed (or at least should be designed) to support incremental compilation, meaning that within the AnnotationProcessor.process() invocation, if the processor calls AnnotationProcessorEnvironment.getTypeDeclarations(), it will only retrieve a single type at a time; the process() method will be called repeatedly during the course of a build so that all files are eventually processed.
The invocation process for Java 6 processors is more straightforward. A typical call stack looks like this:
ModelTesterProc.process(Set<TypeElement>, RoundEnvironment) line: 107 RoundDispatcher.handleProcessor(ProcessorInfo) line: 139 RoundDispatcher.round() line: 121 IdeAnnotationProcessorManager(BaseAnnotationProcessorManager).processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 159 IdeAnnotationProcessorManager.processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 134 Compiler.processAnnotations() line: 810 Compiler.compile(ICompilationUnit[]) line: 428 IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[], SourceFile[], boolean) line: 364 IncrementalImageBuilder.compile(SourceFile[], SourceFile[], boolean) line: 321 IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 301 IncrementalImageBuilder.build(SimpleLookupTable) line: 134 JavaBuilder.buildDeltas(SimpleLookupTable) line: 265 JavaBuilder.build(int, Map, IProgressMonitor) line: 193 [higher calls omitted for brevity]
Note that the processing is performed by an IdeAnnotationProcessorManager; this is distinguished from the BatchAnnotationProcessorManager, which would be used if processing was invoked from a command line (ecj.jar) compilation. Command-line compilation with ecj.jar uses the same typesystem implementation as IDE compilation, but somewhat different Messager and Filer implementations.
Reconcile-time processing is only supported for Java 5 processors. This is chiefly because, after developing the Java 5 processing implementation, it became clear that writing highly-performant annotation processors requires a level of compiler experience that most programmers do not have, and if less-performant processors are inserted into the reconcile step, reconcile can become painfully and confusingly slow. Thus when the Java 6 processing implementation was added it was decided to not support reconcile-time processing. Eclipse's effectiveness as an IDE depends on reconcile being an extremely fast operation. In hindsight, edit-time processing should have been limited to reporting errors, and should have been implemented as a background validation task rather than within the reconcile step.
TODO: describe practical differences between reconcile and build.
The stack trace of a typical reconcile-time invocation looks like this:
SomeProcessor.process() line: xxx APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628 APTDispatchRunnable.access$0(APTDispatchRunnable, AbstractCompilationEnv, boolean, boolean) line: 598 APTDispatchRunnable$ReconcileEnvCallback.run(AbstractCompilationEnv) line: 77 ReconcileEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 135 CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766 CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478 ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737 BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856 ReconcileEnv.openPipeline() line: 108 AbstractCompilationEnv.newReconcileEnv(ReconcileContext, AbstractCompilationEnv$EnvCallback) line: 97 APTDispatchRunnable.reconcile(ReconcileContext, IJavaProject) line: 211 APTDispatchRunnable.runAPTDuringReconcile(ReconcileContext, AptProject, Map<AnnotationProcessorFactory,Attributes>) line: 159 AptCompilationParticipant.reconcile(ReconcileContext) line: 223 ReconcileWorkingCopyOperation$1.run() line: 257 SafeRunner.run(ISafeRunnable) line: 42 ReconcileWorkingCopyOperation.notifyParticipants(CompilationUnit) line: 244 ReconcileWorkingCopyOperation.executeOperation() line: 94 ReconcileWorkingCopyOperation(JavaModelOperation).run(IProgressMonitor) line: 728 ReconcileWorkingCopyOperation(JavaModelOperation).runOperation(IProgressMonitor) line: 788 CompilationUnit.reconcile(int, int, WorkingCopyOwner, IProgressMonitor) line: 1242 JavaReconcilingStrategy.reconcile(ICompilationUnit, boolean) line: 126 [higher calls omitted for brevity]
When annotation processors are invoked, they can in turn call back into the compiler through various interfaces in order to generate new files and report errors. The interfaces are naturally not the same as those used within the JDT compiler, so wrappers and proxies are employed to glue the different systems together. The most significant difference in systems is that, in both the Java 5 and Java 6 processor interfaces, the typesystem is separated into Type and Element hierarchies, in which a Type object represents a particular reification of a type while an Element represents a declaration in code. The distinction is easiest to understand in the context of generics: for example, the following code sample contains declarations of classes Lister and Test, which contain declarations of method get() and fields 'strings' and 'numbers' respectively. The type of the return of get() is List<T>, the type of strings is Lister<String>, and the type of numbers is Lister<Number>. Note that the single declaration of Lister<T> was able to generate multiple Types.
import java.util.List;
class Lister<T> {
List<T> get() { ... }
}
class Test {
Lister<String> strings;
Lister<Number> numbers;
}
TODO
TODO
TODO
TODO