Visitar URL original
GH-1283: Optimize number of build compilations in CI by xborder · Pull Request #1284 · apache/arrow-java · GitHub
Skip to content

GH-1283: Optimize number of build compilations in CI #1284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xborder wants to merge 15 commits into
base: main
Choose a base branch
from
+299 −59
Open

GH-1283: Optimize number of build compilations in CI #1284

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
513a69c
MINOR: Build test artifacts once in CI
xborder Jul 28, 2026
bcc9111
Simplify build-once test workflow
xborder Jul 29, 2026
792fda8
Restore custom Surefire test coverage
xborder Jul 31, 2026
99574ff
Make Vector test passes explicit
xborder Jul 31, 2026
f71a6f6
MINOR: Align build-once workflow with current CI matrix
xborder Sep 2, 2026
e9591df
Fix stale Arrow artifacts in hosted tests
xborder Sep 2, 2026
0fad043
Verify prebuilt test artifacts
xborder Sep 2, 2026
5fc4f18
Simplify prebuilt test validation
xborder Sep 2, 2026
a775dad
Optimize build-once workflow caches
xborder Sep 3, 2026
72e6787
Quote workflow build JDK
xborder Sep 3, 2026
398329f
Fix build JDK workflow context
xborder Sep 3, 2026
04b74d5
Use established Maven cache keys
xborder Sep 3, 2026
9dd2495
Use shallow test checkouts
xborder Sep 3, 2026
6d4c6eb
Sync setup-java version with upstream
xborder Sep 4, 2026
5309167
Fix test workflow review comments
xborder Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Optimize build-once workflow caches
  • Loading branch information
@xborder
xborder committed Oct 1, 2026
commit a775dadfd41b9d0351373606a86b76d6e1475f26
33 changes: 21 additions & 12 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ permissions:
contents: read

env:
BUILD_JDK: 17
DOCKER_VOLUME_PREFIX: ".docker/"
MAVEN: 3.9.16

Expand All @@ -53,12 +54,12 @@ jobs:
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-17-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-17-${{ env.MAVEN }}-
key: java-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Build without tests
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
JDK: 17
JDK: ${{ env.BUILD_JDK }}
run: |
docker compose run \
--rm \
Expand All @@ -77,6 +78,8 @@ jobs:
name: java-build
path: java-build.tgz
retention-days: 1

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With one day of retention, "Re-run failed jobs" stops working a day after the run: build-java succeeded so it is not re-run, and the test jobs then fail on download-artifact. The only way out is re-running everything, which defeats the purpose.

Could you raise this, for example to 7 days? Same for the cdata-build at line 208.

- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

@kou kou Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to remove them? We can't use them to avoid rebuilding Arrow Java, right?

@xborder xborder Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is removed so the cache doesn’t carry old Arrow binaries into future runs. The binaries are already in the build artifacts used by every test job


test-java:
name: ${{ matrix.name || format('AMD64 Ubuntu JDK {0} Maven 3.9.16', matrix.jdk) }}
Expand Down Expand Up @@ -107,7 +110,7 @@ jobs:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
fetch-depth: 1
submodules: recursive
- name: Set up Java
if: ${{ !matrix.compose_service }}
Comment thread
xborder marked this conversation as resolved.
Expand All @@ -121,8 +124,8 @@ jobs:
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-17-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-17-${{ env.MAVEN }}-
key: java-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Download reusable artifacts
uses: actions/download-artifact@v6
Comment thread
xborder marked this conversation as resolved.
Outdated
with:
Expand Down Expand Up @@ -160,14 +163,18 @@ jobs:
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
${{ matrix.compose_service }}
- name: Exclude reactor artifacts from hosted Maven dependency cache
if: ${{ !matrix.compose_service && always() }}
shell: bash
run: rm -rf "${HOME}/.m2/repository/org/apache/arrow"

build-cdata:
name: Build C Data artifacts
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
env:
JDK: 17
JDK: ${{ env.BUILD_JDK }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
Expand All @@ -178,8 +185,8 @@ jobs:
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: cdata-${{ env.JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-${{ env.JDK }}-${{ env.MAVEN }}-
key: cdata-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Build C Data without tests
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
Expand All @@ -202,6 +209,8 @@ jobs:
name: cdata-build
path: cdata-build.tgz
retention-days: 1
- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

test-cdata:
name: AMD64 Conda JNI JDK ${{ matrix.jdk }} Maven 3.9.16
Expand All @@ -219,14 +228,14 @@ jobs:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
fetch-depth: 1
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: cdata-17-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-17-${{ env.MAVEN }}-
key: cdata-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-deps-v2-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Download reusable C Data artifacts
uses: actions/download-artifact@v6
with:
Expand Down