Visitar URL original
GH-1283: Optimize number of build compilations in CI by xborder · Pull Request #1284 · apache/arrow-java · GitHub
Skip to content

GH-1283: Optimize number of build compilations in CI #1284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xborder wants to merge 15 commits into
base: main
Choose a base branch
from
+299 −59
Open

GH-1283: Optimize number of build compilations in CI #1284

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
513a69c
MINOR: Build test artifacts once in CI
xborder Jul 28, 2026
bcc9111
Simplify build-once test workflow
xborder Jul 29, 2026
792fda8
Restore custom Surefire test coverage
xborder Jul 31, 2026
99574ff
Make Vector test passes explicit
xborder Jul 31, 2026
f71a6f6
MINOR: Align build-once workflow with current CI matrix
xborder Sep 2, 2026
e9591df
Fix stale Arrow artifacts in hosted tests
xborder Sep 2, 2026
0fad043
Verify prebuilt test artifacts
xborder Sep 2, 2026
5fc4f18
Simplify prebuilt test validation
xborder Sep 2, 2026
a775dad
Optimize build-once workflow caches
xborder Sep 3, 2026
72e6787
Quote workflow build JDK
xborder Sep 3, 2026
398329f
Fix build JDK workflow context
xborder Sep 3, 2026
04b74d5
Use established Maven cache keys
xborder Sep 3, 2026
9dd2495
Use shallow test checkouts
xborder Sep 3, 2026
6d4c6eb
Sync setup-java version with upstream
xborder Sep 4, 2026
5309167
Fix test workflow review comments
xborder Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Simplify build-once test workflow
  • Loading branch information
@xborder
xborder committed Oct 1, 2026
commit bcc9111c2bde56b4684f797c088ea77d89a2cc2d
73 changes: 21 additions & 52 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ permissions:

env:
DOCKER_VOLUME_PREFIX: ".docker/"
MAVEN: 3.9.9

jobs:
build-java:
Expand All @@ -52,29 +53,23 @@ jobs:
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-17-3.9.9-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-17-3.9.9-
key: java-build-17-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-17-${{ env.MAVEN }}-
- name: Build without tests
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
JDK: 17
MAVEN: 3.9.9
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
ubuntu-artifacts
if [ -d .docker ]; then
sudo chown -R "$(id -u):$(id -g)" .docker
fi
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Pack reusable artifacts
shell: bash
run: |
tar -czf java-build.tgz \
.docker/java-build \
.docker/jni-dist \
.docker/maven-cache/repository/org/apache/arrow
- name: Upload reusable artifacts
uses: actions/upload-artifact@v5
Comment thread
xborder marked this conversation as resolved.
Outdated
Expand All @@ -84,93 +79,77 @@ jobs:
retention-days: 1

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With one day of retention, "Re-run failed jobs" stops working a day after the run: build-java succeeded so it is not re-run, and the test jobs then fail on download-artifact. The only way out is re-running everything, which defeats the purpose.

Could you raise this, for example to 7 days? Same for the cdata-build at line 208.


test-java:
name: ${{ matrix.name || format('AMD64 Ubuntu JDK {0} Maven {1}', matrix.jdk, matrix.maven) }}
name: ${{ matrix.name || format('AMD64 Ubuntu JDK {0} Maven 3.9.9', matrix.jdk) }}
needs: build-java
runs-on: ${{ matrix.os }}
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
target: [ubuntu]
jdk: [17, 21, 23]
maven: [3.9.9]
os: [ubuntu-latest]
include:
- target: ubuntu
os: ubuntu-latest
- os: ubuntu-latest
compose_service: ubuntu-test-prebuilt
- target: macos-intel
name: AMD64 macOS 15-intel Java JDK 17
- name: AMD64 macOS 15-intel Java JDK 17
os: macos-15-intel
jdk: 17
maven: 3.9.9
compose_service: ''
- target: macos-arm
name: AArch64 macOS latest Java JDK 17
- name: AArch64 macOS latest Java JDK 17
os: macos-latest
jdk: 17
maven: 3.9.9
compose_service: ''
- target: windows
name: AMD64 Windows Server 2022 Java JDK 17
- name: AMD64 Windows Server 2022 Java JDK 17
os: windows-latest
jdk: 17
maven: 3.9.9
compose_service: ''
env:
JDK: ${{ matrix.jdk }}
MAVEN: ${{ matrix.maven }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: recursive
- name: Set up Java
if: ${{ matrix.compose_service == '' }}
if: ${{ !matrix.compose_service }}
Comment thread
xborder marked this conversation as resolved.
uses: actions/setup-java@v5
with:
distribution: 'temurin'
java-version: ${{ matrix.jdk }}
cache: 'maven'
- name: Cache Docker Volumes
if: ${{ matrix.compose_service != '' }}
if: ${{ matrix.compose_service }}
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-17-3.9.9-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-17-3.9.9-
key: java-build-17-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-17-${{ env.MAVEN }}-
- name: Download reusable artifacts
uses: actions/download-artifact@v6
Comment thread
xborder marked this conversation as resolved.
Outdated
with:
name: java-build
- name: Restore reusable artifacts
shell: bash
run: |
mkdir -p .docker
rm -rf \
.docker/java-build \
.docker/jni-dist \
.docker/maven-cache/repository/org/apache/arrow
tar -xzf java-build.tgz
- name: Restore reusable artifacts for hosted runner
if: ${{ matrix.compose_service == '' }}
if: ${{ !matrix.compose_service }}
shell: bash
run: |
rm -rf build jni
cp -a .docker/java-build/build build
cp -a .docker/jni-dist jni
mkdir -p "${HOME}/.m2/repository/org/apache"
cp -a .docker/maven-cache/repository/org/apache/arrow "${HOME}/.m2/repository/org/apache/"
- name: Test prebuilt artifacts on macOS/Windows
if: ${{ matrix.compose_service == '' }}
if: ${{ !matrix.compose_service }}
shell: bash
env:
ARROW_JAVA_TEST_PREBUILT: "ON"
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh . build jni

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On main the Windows job runs ci/scripts/build.sh before test.sh. Here it only runs Surefire against classes compiled on Linux, and as far as I can see that leaves no Maven build on Windows in any workflow (jni-windows in rc.yml only runs jni_windows_build.sh, which is CMake).

A change that breaks the build on Windows (codegen paths, a plugin without a Windows binary, line-ending-sensitive checks) would no longer be caught. I would keep the full build for the Windows job. If we decide that is acceptable to drop, please say so in the PR description, which currently states that the existing platform coverage is retained.

I'm more in favor to keep Windows job.

- name: Test prebuilt artifacts on Ubuntu Docker
if: ${{ matrix.compose_service != '' }}
if: ${{ matrix.compose_service }}
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
Expand All @@ -188,7 +167,6 @@ jobs:
timeout-minutes: 30
env:
JDK: 17
MAVEN: 3.9.9
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
Expand All @@ -202,7 +180,6 @@ jobs:
key: cdata-${{ env.JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-${{ env.JDK }}-${{ env.MAVEN }}-
- name: Build C Data without tests
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: |
Expand All @@ -211,11 +188,8 @@ jobs:
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
cdata-artifacts
if [ -d .docker ]; then
sudo chown -R "$(id -u):$(id -g)" .docker
fi
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Pack reusable C Data artifacts
shell: bash
run: |
tar -czf cdata-build.tgz \
.docker/cdata-build \
Expand All @@ -229,7 +203,7 @@ jobs:
retention-days: 1

test-cdata:
name: AMD64 Conda JNI JDK ${{ matrix.jdk }} Maven ${{ matrix.maven }}
name: AMD64 Conda JNI JDK ${{ matrix.jdk }} Maven 3.9.9
needs: build-cdata
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
Expand All @@ -238,10 +212,8 @@ jobs:
fail-fast: false
matrix:
jdk: [17, 21, 23]
maven: [3.9.9]
env:
JDK: ${{ matrix.jdk }}
MAVEN: ${{ matrix.maven }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v6
Expand All @@ -252,23 +224,20 @@ jobs:
uses: actions/cache@v5
with:
path: .docker/maven-cache
key: cdata-17-3.9.9-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-17-3.9.9-
key: cdata-17-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-17-${{ env.MAVEN }}-
- name: Download reusable C Data artifacts
uses: actions/download-artifact@v6
with:
name: cdata-build
- name: Restore reusable C Data artifacts
shell: bash
run: |
mkdir -p .docker
rm -rf \
.docker/cdata-build \
.docker/cdata-jni-dist \
.docker/maven-cache/repository/org/apache/arrow
tar -xzf cdata-build.tgz
- name: Test C Data without compiler lifecycle
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: |
Expand Down
6 changes: 1 addition & 5 deletions ci/scripts/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,7 @@ cp -r "${source_dir}/dev" "${build_dir}"
# crawl back up to the top. GNU realpath has --relative-to but this does not
# work on macOS

poms=$(
find "${source_dir}" \
\( -path "${source_dir}"/build -o -path "${source_dir}"/.docker \) -prune \
-o -type f -name pom.xml -print
)
poms=$(find "${source_dir}" \( -path "${source_dir}"/build -o -path "${source_dir}"/.docker \) -prune -o -type f -name pom.xml -print)
if [[ "$OSTYPE" == "darwin"* ]]; then
poms=$(echo "$poms" | xargs -n1 python -c "import sys; import os.path; print(os.path.relpath(sys.argv[1], '${source_dir}'))")
else
Expand Down
24 changes: 9 additions & 15 deletions ci/scripts/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.

set -ex
set -exo pipefail

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
exit
Expand All @@ -41,45 +41,41 @@ mvn=(
)

run_tests() {
local log_name=$1
shift

if [[ "${ARROW_JAVA_TEST_PREBUILT:-OFF}" = "ON" ]]; then
run_prebuilt_tests "${log_name}" "${@}" -DfailIfNoTests=false surefire:test
run_prebuilt_tests "${@}" -DfailIfNoTests=false surefire:test
else
"${@}" test
fi
}

run_prebuilt_tests() {
local log_name=$1
shift
local log
log=$(mktemp)

set -o pipefail
"${@}" | tee "${source_dir}/${log_name}"
"${@}" | tee "${log}"

if grep -E "Compiling [0-9]+ source files?" "${source_dir}/${log_name}"; then
if grep -E "Compiling [0-9]+ source files?" "${log}"; then
echo "Unexpected compilation occurred while running prebuilt tests."
exit 1
fi

if ! grep -q "Tests run:" "${source_dir}/${log_name}"; then
if ! grep -q "Tests run:" "${log}"; then
echo "No surefire test summary found; tests may have been skipped."
exit 1
fi

rm -f "${log}"
}

pushd "${build_dir}"

if [[ "${ARROW_JAVA_TEST_BASE:-ON}" = "ON" ]]; then
run_tests \
surefire.log \
"${mvn[@]}" \
-Darrow.test.dataRoot="${source_dir}/testing/data"

if [[ "${ARROW_JAVA_TEST_PREBUILT:-OFF}" = "ON" ]]; then
run_prebuilt_tests \
opens-surefire.log \
"${mvn[@]}" \
-DfailIfNoTests=false \
-pl memory/memory-core \
Expand All @@ -95,7 +91,6 @@ if [ "${ARROW_JAVA_JNI}" = "ON" ]; then
fi
if [ "${#projects[@]}" -gt 0 ]; then
run_tests \
jni-surefire.log \
"${mvn[@]}" \
-Parrow-jni \
-pl "$(
Expand All @@ -107,7 +102,6 @@ fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
run_tests \
cdata-surefire.log \
"${mvn[@]}" \
-Parrow-c-data \
-pl c \
Expand Down
Loading