Support Gradle Incremental Processing

Gradle 4.7+ has a new feature to let annotation processors run incrementally. In order to make that work, Immutables has to tell Gradle how its annotation processor behave during execution.

For details see:

https://docs.gradle.org/nightly/userguide/java_plugin.html#sec:incremental_annotation_processing

gradle/gradle#5277

Supporting this feature, would greatly reduce build times using Gradle.

@elucash

@elucash Hi, could you please just give some pointers how to move this forward? If it is in line with your plans, of course.

I have put together some resources to understand what the issue is about:

Thank you for the feature request! I think it would be really great to support this. I would definitely like such a feature if this would not require a lot of work.

Looking at examples I barely understand what exactly has to be done here (with respect to annotation processing code). Is there a way to externally tell / configure Annotation processor to be run in one of these sandbox/isolation modes and see if processor works before actually augmenting annotation processor's metadata?

Immutables is already incremental capable, like when working inside of Eclipse on ECJ with auto-build on, so in theory we should be ok if incrementality enabled
Some features will not work or work slightly differently if put in Gradle isolation sandbox, depending how properly Gradle wrappers are coded (as both Javac and ECJ dev screwed up implementing those elements properly, little hope that Gradle will get it right early on)
PRs are welcome. The main problem here is not implementing any change, but ensuring things work properly and I definitely need help here. One of the testing strategies might be copying value-fixture content under Gradle build and see how much of edge cases compile.

As a side note. I think it is unfair for Gradle to promote such a feature, so they try to artificially lock-in into their tool. The problem with incremental compilation is that no one trust it, so people are doing clean builds with Maven, Ant, Gradle, thus defeating the purpose of incremental compilation. Tools like Bazel (or Buck etc) have a proven approach to reproducible and reliable builds by using isolated modules (as opposing to separate custom compilation-unit/file change tracking inside module as Gradle apparently tries to do). Also, Gradle, by creating yet another wrapping around Java Lang Model elements make annotation processor authors life more complicated as we already try to "fix" or workaround compiler bugs by inspecting those elements, so now, with Gradle wrappers added to the mix, the things might get complicated i.e. certain features which benefit from fixes(workarounds) to compiler bugs will now fail again, because of Gradle's "efforts to improve".

@oehme

Thank you for the detailed response. I'm not really qualified to answer your question as I'm only a passerby who happens to use Immutables (love it!) and Gradle (it works for us, so 👍 too)

I think that your question can be answered by @oehme (https://github.com/oehme). Hopefully, he'll notice the mention. :-)

Looking at examples I barely understand what exactly has to be done here

It's all explained here. There is nothing more to it. You need to know which category your processor falls into and declare that category in a single-line file. Since you say your processor works with JDT, it must be in the "isolating" category, since that is the only kind of processor that JDT handles correctly.

as both Javac and ECJ dev screwed up implementing those elements properly, little hope that Gradle will get it right early on

You are assuming we are doing something invasive, which we don't. The only thing we decorate are the ProcessingEnvironment and the Filer. Everything will still work like in plain javac, unless you try to cast these two interfaces to a concrete implementation (which would be a breakage of the processing contract).

Now to respond to the unconstructive negativity at the end. I don't know where that was coming from, given your otherwise positive comment:

As a side note. I think it is unfair for Gradle to promote such a feature, so they try to artificially lock-in into their tool.

What is unfair? We're providing a great new feature that requires a single line of code to opt in. How is this "lock in"? JetBrains will soon adopt it for the Kotlin compiler too and we'd be more than happy for others to pick up this metadata file too.

The problem with incremental compilation is that no one trust it

Incremental compilation is the default in Gradle. Our users have full trust in it and for good reason. We have a huge set of end to end tests ensuring its correctness in all kinds of interesting corner cases.

so people are doing clean builds with Maven, Ant, Gradle

Maven and Ant yes. Gradle users don't use clean.

Tools like Bazel (or Buck etc) have a proven approach to reproducible and reliable builds by using isolated modules (as opposing to separate custom compilation-unit/file change tracking inside module as Gradle apparently tries to do).

This is just FUD. There is no difference between how Gradle detects file changes between modules or inside a single module. Bazel simply decided not to expose the exact change information to the individual tasks, so they always rerun fully if anything has changed. That's fine, but it means bigger modules will always be slow to build on incremental changes. Gradle goes the extra mile and gives you fast and reliable incremental builds, no matter where you are on the modularization spectrum.

Also, Gradle, by creating yet another wrapping around Java Lang Model elements make annotation processor authors life more complicated as we already try to "fix" or workaround compiler bugs by inspecting those elements, so now, with Gradle wrappers added to the mix, the things might get complicated

Again, this is based on wrong assumptions. We don't wrap the model elements. Also, blaming Eclipse (or Gradle or any other tool) is a bit weird. It is your decision to access compiler internals. By doing that you are breaking encapsulation. You only have yourself to blame for any resulting failure. I know you're doing it for good reasons, but don't blame others when it breaks.

because of Gradle's "efforts to improve".

Where is this negativity coming from? Did you have a bad day?

@oehme

@oehme I sympathize with your desire to defend the product and clear some FUD, and I stand corrected about Gradle not wrapping model object (just env and filer). Also apologies about apparent negativity (and yes, I'm not positive about Gradle as a product due do a lot of reasons, sorry, and this is not a place to argue about those). Please, don't go personal with "bad day" or assuming I blaming something, it is just an exposure to another information bubble where Gradle is not considered "cool".

It is your decision to access compiler internals.

Yes, and we workaround bugs that either stay unfixed for years or only available in newest compiler version. Immutables processor is competent to not break on those, just some fixes might disappear and. I and other project contributors spent countless hours tuning for many compilers environments including Eclipse (batch and incremental), Javac, ErrorProne's Javac, Kapt, running with some other rogue processors. Supporting yet another specialized compiler runtime is not something we can take too lightly, the position that "if you only use standard APIs everything will be ok" is incomplete at best and probably inaccurate.

You should have very little trouble supporting the Gradle case since it's just straight Javac with a wrapper around the Filer and ProcessingEnvironment. Using your fixture project as a test case sounds like a good idea. You could even automate that using Gradle's tooling API if you'd like.

@oehme

@oehme thank you for the hints, this should be a way to go (thinking of adding ad-hoc Gradle build to fixture module for a test drive)

Is there a way to externally tell / configure Annotation processor to be run in one of these sandbox/isolation modes and see if processor works before actually augmenting annotation processor's metadata?

If you really want, you could declare the processor as dynamic and use a processor option (-Aimmutables.gradle.incremental=ISOLATING) to activate each mode.

@tbroyer

@tbroyer Thank you for the hint! I've thought about something like that and I guess I'll git it a try.

Fwiw, Dagger is going to use something similar: google/dagger#1120 (comment)

Interesting, actually ordering issue (of calling getSupportedOptions vs init) is something I've considered to be a problem, and indeed it might be, so system props or -A might just work, need to try... still reading the thread

There shouldn't be any ordering issue. init is called before getSupportedOptions.

Apparently, more changes are needed for incremental processing to actually work: with ./gradlew compileJava --info on my project, I got:

Full recompilation is required because the generated type 'xxx.xxx.ImmutableXxx' must have exactly one originating element, but had 0. Analysis took 0.017 secs.

@tbroyer

@tbroyer I wonder how origination is established

In Filer's createSourceFile.

But then what if the change to Main.java is to add @Value.Immutable? 😉

it is also possible to know in theory, or just resort to full recompilation, there's no bottom to complexity here.. unfortunately

Since we're driving javac, we could probably suppress that warning, it doesn't play well with incremental processing as you rightly pointed out. Feel free to open an issue.

The restrictions are quite arbitrary, naive and are not documented.

They're all documented in the incremental annotation processing chapter. I can't comment on "naive", since I don't know what that's supposed to mean. They are not arbitrary, we chose them consciously to keep the scope of this feature in check and because the processors highest on our list didn't need to read/write resource files. This may change in the future, if we encounter enough popular processors that need to read or write resources.

Btw. what resources are you trying to read? Gradle uses an empty "sourcePath" by default, so Filer#getResource would never be successful as far as I can tell, independently of incremental processing.

They're all documented in the incremental annotation processing chapter

this is the issue with evolving documentation, the section contains different information at different moments in time, the current link does not contain anything about reading, and digging into history reveals that it was mentioned (but removed now) but without specifying effect (inhibiting from read resources, issuing error or resorting to full recompilation).

I can't comment on "naive", since I don't know what that's supposed to mean. They are not arbitrary, we chose them consciously to...

Agree about "non-preciseness" of term "naive". The limitation to reading resources (as opposed to creating) is arbitrary and naive in my view because it was not proved to be needed (and later was changed gradle/gradle@8ce7ea5
) and "statistical" approach is "working" but cannot be claimed to be free of "arbitrary and naive".

Or and, ohboi, I[/we] was[/is] naive too: like checking for source information if it's available to overcome compiler bugs for some niche functionality and expect it will not backfire, sigh..

I think that the main problem is that products like Gradle or Immutables (oh, I don't put them on the same level of popularity or maturity, of course not) have one level of documentation, but where it comes to compiler specifications and interoperabiltity related technologies and dealing with bugs and quirks of existing compiler implementations, we're lacking here.

Hi,

how is the state of full incremental compilation of Immutables? I will get following message, when re-compiling:

Gradle: gradle-5.4.1
dependencies { annotationProcessor("org.immutables:value-processor:2.7.4") compileOnly("org.immutables:value-annotations:2.7.4") }

Change any class will print following output:
`

Task :compileJava
Caching disabled for task ':compileJava' because:
Build cache is disabled
Task ':compileJava' is not up-to-date because:
Input property 'source' file /home/sergej/Development/IdeaProjects/playground/src/main/java/Wurst.java has changed.
Created classpath snapshot for incremental compilation in 0.004 secs.
Class dependency analysis for incremental compilation took 0.004 secs.
Full recompilation is required because org.immutables.processor.ProxyProcessor is not incremental. Analysis took 0.01 secs.
Compiling with JDK Java compiler API.
:compileJava (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 0.847 secs.

Full recompilation is required because org.immutables.processor.ProxyProcessor is not incremental. Analysis took 0.01 secs.
`

If I look into the value-processor jar, there is a incremental.annotation.processors file, which contains "org.immutables.processor.ProxyProcessor,dynamic".

@SergejIsbrecht

@SergejIsbrecht You need to opt in:

tasks {
    compileJava {
        options.compilerArgs.add("-Aimmutables.gradle.incremental")
    }
}

@SergejIsbrecht

@SergejIsbrecht You need to opt in:

tasks {
    compileJava {
        options.compilerArgs.add("-Aimmutables.gradle.incremental")
    }
}

Thank you. Looks good to me.

@SergejIsbrecht

Hi there,

Is this working using the above compileJava command, or was this never released?

I see warnings in Intellij after trying the above Warning:java: The following options were not recognized by any processor: '[immutables.gradle.incremental]'

@SergejIsbrecht You need to opt in:

tasks {
    compileJava {
        options.compilerArgs.add("-Aimmutables.gradle.incremental")
    }
}

@ymartin

@ymartin I'm facing a similar issue in my Android project. Were you able to enable it successfully?

On a (Java 11) service that I'm working on:

Bumped Gradle to 7.5 (Immutables version = 2.8.1)
Added the "-Aimmutables.gradle.incremental"
Incremental compilation works 🎉

When using immutables:2.9.1-2.10.1 "-Aimmutables.gradle.incremental" option doesn't work anymore. Gradle build is no more incremental

There were no changes in this area, so if it's stopped to work, this may be something new or unknown. I would ask for help to figure out if we can fix it somehow

@fretfan

@fretfan please provide a reproducible example. I just tried with a minimal immutables project and it works just fine.

I take my words back. I was wrong. All works fine.
I have project where immutables, freebuilder and lombok are used in a mix. Now tried Immutables together with Lombok only and ./gradlew compileJava --info command. Got confirmation in logs, that incremental compilation works fine. Sorry for confusion

Navigation Menu

Repository navigation

Support Gradle Incremental Processing #804

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support Gradle Incremental Processing #804

Description

Activity

MartyIX commented on Aug 13, 2018

elucash commented on Aug 13, 2018

MartyIX commented on Aug 13, 2018

oehme commented on Aug 14, 2018

elucash commented on Aug 14, 2018

oehme commented on Aug 14, 2018

elucash commented on Aug 14, 2018

tbroyer commented on Aug 29, 2018

elucash commented on Aug 29, 2018

tbroyer commented on Aug 29, 2018

elucash commented on Aug 29, 2018

oehme commented on Aug 29, 2018

tbroyer commented on Nov 14, 2018

elucash commented on Nov 14, 2018

tbroyer commented on Nov 14, 2018

18 remaining items

tbroyer commented on Dec 29, 2018

elucash commented on Dec 29, 2018

oehme commented on Dec 31, 2018

elucash commented on Dec 31, 2018

SergejIsbrecht commented on Apr 28, 2019

tbroyer commented on Apr 28, 2019

SergejIsbrecht commented on Apr 28, 2019

ymartin commented on Apr 24, 2020

dijgas commented on Aug 31, 2021

zanella commented on Sep 30, 2022

fretfan commented on Mar 7, 2025

elucash commented on Apr 2, 2025

oehme commented on Apr 3, 2025

fretfan commented on Apr 7, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions