Repository navigation
1120 lines (1045 loc) · 58.2 KB
/
Copy pathtest-native.yml
File metadata and controls
1120 lines (1045 loc) · 58.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# The native build: the `arcadedb` distribution (a GraalVM native-image shared library loaded with ctypes). This workflow builds the
# wheels and runs the parity suite against them; release-native.yml calls it on a version tag and publishes. The JVM build,
# `arcadedb-jvm`, is test-jvm.yml / release-jvm.yml.
#
# A native image build takes about 5 to 30 minutes per platform and the ArcadeData organisation shares a runner cap with
# upstream ArcadeDB, so the load is selective (scripts/select_native_matrix.py holds the legs, tests/test_select_native_matrix.py pins them):
# pull request only when a native path changed (the `paths` filter below); linux-x86_64, Python 3.12; not for a draft
# push to main only when a native path changed; all four platforms, Python 3.12 and 3.14
# nightly all four platforms, Python 3.12 and 3.14, but only when something besides docs changed in the last 25 hours (a full run on Mondays)
# dispatch all four platforms, Python 3.12 and 3.14
# workflow_call from release-native.yml with `release: true`: all four platforms, Python 3.11 to 3.14
# This workflow never publishes: it has no id-token permission and no environment.
#
# Beside the full parity runs, the `pyver` job installs the already built linux-x86_64 wheel (py3-none: no rebuild) into the Pythons the full
# runs of the event leave out (a pull request: 3.11, 3.13, 3.14; main, nightly, dispatch: 3.11, 3.13; a release: none), into 3.15 (always: the full suite
# needs jpype1 and ladybug, which have no cp315 wheel yet, so 3.15 runs without the test files that need jpype1) and into the free-threaded 3.14t, and runs a small subset of the tests there (about 20 s of tests per Python). A release needs it: a leg that passes uploads a marker (native-passed-pyver-py<version>), and release-native.yml
# publishes nothing unless the legs of a release (3.15 and the free-threaded 3.14t) passed. Only linux-x86_64 is checked on them.
# scripts/select_native_matrix.py decides its legs.
#
# The `examples` job runs every script of examples/ against the installed wheel of each platform (Python 3.12 and 3.14; a pull request: linux-x86_64, 3.12), through
# scripts/run_examples.sh, the runner test-python-examples.yml uses too. scripts/native_examples_skip.txt lists, with a reason, the examples that need the JVM package.
# `examples-summary` is the one check for all its legs. They do not decide what a release publishes (release-native.yml reads the markers and the wheels only).
#
# One wheel per platform, py3-none-<platform tag>:
# linux-x86_64 manylinux_2_28 container manylinux_2_28_x86_64 libarcadedb.so + libjvector-x86_64.so
# linux-aarch64 manylinux_2_28 aarch64 container manylinux_2_28_aarch64 libarcadedb.so + libjvector-aarch64.so
# macos-arm64 native, heap and threads capped macosx_<minos>_arm64 libarcadedb.dylib
# windows-x64 native, MSVC environment loaded win_amd64 libarcadedb.dll
# Only free GitHub-hosted standard runners. GraalVM Community Edition only (pinned by sha256 per platform), never Oracle GraalVM.
#
# Gates per platform: the glibc floor, `auditwheel show`, `twine check`, wheel name `arcadedb` and version == requested version, and the
# test suite of this repository run against the INSTALLED wheel (native/wheel/parity_wheel.py): no failure, error, crash, timeout, or skip
# besides the counted NOTYET and NA-JVM kinds (tests/conftest.py).
#
# A leg that passes uploads a marker (native-passed-<platform>-py<version>) and the wheel (native-wheel-<platform>); a `pyver` leg that
# passes uploads a marker too (native-passed-pyver-py<version>). release-native.yml reads them to decide what may be published. Do not
# rename these artifacts without changing release-native.yml.
#
# Crashes (ArcadeData/arcadedb-python#295): a test file whose process dies without a result is a CRASH row with its exit code, and fails the leg.
# On the Windows legs only, the harness runs such a file once more in a fresh process (--retry-crash): a retry that produces a result is a
# FLAKY-CRASH row, the leg may pass, and the `crash-report` job makes it loud (the job summary, and on a release or a dispatch one comment on
# issue 295); a second crash, a failing test, a timeout and every Linux or macOS crash stay failures. Each Windows test process also writes a
# faulthandler trace into the parity artifact (parity-wheel-xml/<file>.attemptN.fault.txt).
name: Test native (arcadedb)
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
# tests/ and src/arcadedb/ are shared by both backends: a change there can break the native parity suite, so it runs too.
paths: &native_paths
- "native/**"
- "src/arcadedb/**"
- "tests/**"
- "scripts/select_native_matrix.py"
- "scripts/smoke_wheel.py"
- "scripts/run_examples.sh"
- "scripts/native_examples_skip.txt"
- "examples/**"
- "!examples/README.md"
- "pyproject.toml"
- "setup.py"
- ".github/workflows/test-native.yml"
push:
branches: [main]
paths: *native_paths
schedule:
- cron: "40 3 * * *"
# Called by release-native.yml on a version tag.
workflow_call:
inputs:
version:
description: "Wheel version to build (PEP 440: A.B.C.devN, A.B.C or A.B.C.postN); the release passes the tag"
required: true
type: string
engine:
description: "Engine release (Maven Central) the image links; empty = the release segment of a final version, else the newest release"
required: false
default: ""
type: string
maven-engine:
description: "Timestamped snapshot (A.B.C-YYYYMMDD.HHMMSS-N) Maven resolves for the A.B.C-SNAPSHOT engine, so the build cannot move with later deploys; empty = the engine itself (floating); a .devN release always uses the pinned one"
required: false
default: ""
type: string
release:
description: "The release legs: all four platforms and Python 3.11 to 3.14 (otherwise the small main/nightly legs)"
required: false
default: false
type: boolean
workflow_dispatch:
inputs:
version:
description: "Wheel version to build (PEP 440: A.B.C.devN, A.B.C or A.B.C.postN)"
required: false
default: "26.12.1.dev0"
type: string
engine:
description: "Engine the image links: a release (Maven Central), or for a devN version A.B.C-SNAPSHOT (Sonatype snapshot repository); empty = the release segment of a final version, else DEV_ENGINE"
required: false
default: ""
type: string
maven-engine:
description: "Timestamped snapshot (A.B.C-YYYYMMDD.HHMMSS-N) Maven resolves for the A.B.C-SNAPSHOT engine, so the build cannot move with later deploys; empty = the engine itself (floating); a .devN release always uses the pinned one"
required: false
default: ""
type: string
legs:
description: "full = all four platforms, Python 3.12 and 3.14; pull-request = linux-x86_64 and Python 3.12 only (what a pull request runs)"
required: false
default: "full"
type: choice
options: [full, pull-request]
# The group names the workflow literally: github.workflow is the CALLER's name in a workflow_call, and the group of release-native.yml
# must differ from this one or the two would wait for each other.
#
# Per event: a pull request has one group per PR; a push to main and the nightly schedule each have their own group (a newer run
# supersedes an older one); a workflow_dispatch has a group of its own per run, so a manual run and a push can never cancel each other
# (they used to share the ref of main); a run for a tag (the release calls this workflow, and github.event_name is then the caller's
# push) is never cancelled, a half-done release is worse than a queued one.
concurrency:
group: >-
${{ github.event_name == 'pull_request' && format('test-native-{0}', github.event.pull_request.number)
|| github.event_name == 'workflow_dispatch' && format('test-native-dispatch-{0}', github.run_id)
|| github.event_name == 'schedule' && 'test-native-schedule'
|| format('test-native-{0}', github.ref) }}
cancel-in-progress: ${{ github.ref_type != 'tag' && github.event_name != 'workflow_dispatch' }}
permissions:
contents: read
env:
GRAALVM_VERSION: "25.4.4.1.1"
GRAALVM_BASE: https://github.com/graalvm/graalvm-ce-builds/releases/download/graal-25.4.4.1.1
UV_VERSION: "0.12.0"
PYTHONUTF8: "1" # Windows defaults to cp1252; the notices contain non-ASCII author names
# The version and engine of a run that does not pass them (pull request, push, nightly). The newest engine release on Maven Central is what an A.B.C.devN wheel links
# (the A.B.C release does not exist yet); raise it when a newer release is out.
DEFAULT_VERSION: "26.12.1.dev0"
DEV_ENGINE: "26.11.1"
jobs:
plan:
name: Plan the legs
# a draft pull request starts the platform legs when it is marked ready, as in test-jvm.yml
if: ${{ github.event_name != 'pull_request' || !github.event.pull_request.draft }}
runs-on: ubuntu-24.04
timeout-minutes: 10
outputs:
run: ${{ steps.plan.outputs.run }}
version: ${{ steps.ver.outputs.version }}
engine: ${{ steps.ver.outputs.engine }}
maven-engine: ${{ steps.ver.outputs.maven-engine }}
final: ${{ steps.ver.outputs.final }}
build: ${{ steps.plan.outputs.build }}
test: ${{ steps.plan.outputs.test }}
pyver: ${{ steps.plan.outputs.pyver }}
examples: ${{ steps.plan.outputs.examples }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: 50
- name: Resolve the version and the engine
id: ver
env:
INPUT_VERSION: ${{ inputs.version }}
INPUT_ENGINE: ${{ inputs.engine }}
INPUT_MAVEN_ENGINE: ${{ inputs.maven-engine }}
INPUT_RELEASE: ${{ inputs.release }}
run: |
set -euo pipefail
version="${INPUT_VERSION:-$DEFAULT_VERSION}"
[[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+(\.(dev|post)[0-9]+)?$ ]] || { echo "::error::Version '$version' is not A.B.C, A.B.C.devN or A.B.C.postN (no aN/bN/rcN)"; exit 1; }
base="${version%%.dev*}"
base="${base%%.post*}"
if [[ "$version" == *.dev* ]]; then
final=false
pin_engine=""
pin_snapshot=""
if [ "${INPUT_RELEASE:-}" = true ]; then
# a .devN RELEASE links the pinned snapshot, the one arcadedb-jvm embeds (scripts/dev_engine_pin.json), and nothing else
pin_env="$(python3 -I scripts/dev_engine_pin.py shell-env "$version")"
pin_engine="$(printf '%s\n' "$pin_env" | sed -n 's/^ENGINE=//p')"
[ -n "$pin_engine" ] || { echo "::error::no engine in the pin" >&2; exit 1; }
[ -z "$INPUT_ENGINE" ] || [ "$INPUT_ENGINE" = "$pin_engine" ] || { echo "::error::A dev release links the pinned engine $pin_engine, not $INPUT_ENGINE" >&2; exit 1; }
python3 -I scripts/dev_engine_pin.py check --tag "$version"
engine="$pin_engine"
# Maven resolves the pinned TIMESTAMPED snapshot, not the floating A.B.C-SNAPSHOT: upstream deploys a new snapshot most days
pin_snapshot="$(printf '%s\n' "$pin_env" | sed -n 's/^SNAPSHOT=//p')"
[ -n "$pin_snapshot" ] || { echo "::error::no snapshot in the pin" >&2; exit 1; }
else
engine="${INPUT_ENGINE:-$DEV_ENGINE}"
fi
else
final=true
engine="${INPUT_ENGINE:-$base}"
# a final or post wheel links exactly the engine release it is named after
[ "$engine" = "$base" ] || { echo "::error::Version $version links engine $base, not $engine" >&2; exit 1; }
fi
# the version Maven resolves: the timestamped snapshot of a dev release (the pin), or of a manual run that names one, else the engine
maven_engine="${pin_snapshot:-}"
if [ -n "${INPUT_MAVEN_ENGINE:-}" ]; then
[[ "$engine" == *-SNAPSHOT ]] || { echo "::error::maven-engine needs a SNAPSHOT engine, not $engine" >&2; exit 1; }
[[ "$INPUT_MAVEN_ENGINE" =~ ^${engine%-SNAPSHOT}-[0-9]{8}\.[0-9]{6}-[0-9]+$ ]] || { echo "::error::maven-engine '$INPUT_MAVEN_ENGINE' is not a timestamped snapshot of $engine" >&2; exit 1; }
[ -z "$maven_engine" ] || [ "$maven_engine" = "$INPUT_MAVEN_ENGINE" ] || { echo "::error::A dev release links the pinned snapshot $maven_engine, not $INPUT_MAVEN_ENGINE" >&2; exit 1; }
maven_engine="$INPUT_MAVEN_ENGINE"
fi
maven_engine="${maven_engine:-$engine}"
# A SNAPSHOT engine is for a devN version only: a final or post version is refused above (it links exactly its release). A release run
# (release-native.yml passes `release: true`) links a snapshot only for a .devN tag, and only the pinned one (scripts/dev_engine_pin.json,
# proven above); any other snapshot, and every snapshot of a final or post release, is refused. tests/test_dev_engine_pin.py runs this step.
if [[ "$engine" == *-SNAPSHOT ]]; then
[[ "$engine" =~ ^[0-9]+\.[0-9]+\.[0-9]+-SNAPSHOT$ ]] || { echo "::error::Engine '$engine' is not A.B.C or A.B.C-SNAPSHOT" >&2; exit 1; }
[ "$final" = false ] || { echo "::error::A SNAPSHOT engine is for a devN version only" >&2; exit 1; }
[ "${INPUT_RELEASE:-}" != true ] || [ "$engine" = "${pin_engine:-}" ] || { echo "::error::A release never links a SNAPSHOT engine ($engine) except the pinned one of a .devN tag" >&2; exit 1; }
fi
{ echo "version=$version"; echo "engine=$engine"; echo "maven-engine=$maven_engine"; echo "final=$final"; } >> "$GITHUB_OUTPUT"
echo "wheel version $version, engine $engine (Maven resolves $maven_engine), final=$final"
- name: Choose the legs
id: plan
env:
EVENT: ${{ github.event_name }}
RELEASE: ${{ inputs.release }}
LEGS: ${{ inputs.legs }}
run: |
set -euo pipefail
run=true
event="$EVENT"
if [ "$RELEASE" = true ]; then event=release; fi
# a manual run may ask for the legs of a pull request (one platform, one Python): the runner cap is shared with upstream
if [ "$EVENT" = workflow_dispatch ] && [ "${LEGS:-full}" = pull-request ]; then event=pull_request; fi
if [ "$EVENT" = "schedule" ]; then
changed="$(git log --since='25 hours ago' --format=%h -- src native tests scripts examples pyproject.toml .github/workflows/test-native.yml | wc -l)"
monday="$(date -u +%u)"
if [ "$changed" = 0 ] && [ "$monday" != 1 ]; then
run=false
echo "nothing changed in the last 25 hours and it is not Monday: no native run tonight"
fi
fi
echo "run=$run" >> "$GITHUB_OUTPUT"
python3 -I scripts/select_native_matrix.py "$event" >> "$GITHUB_OUTPUT"
build:
name: Image and wheel (${{ matrix.key }})
needs: plan
if: ${{ needs.plan.outputs.run == 'true' }}
runs-on: ${{ matrix.runner }}
timeout-minutes: ${{ matrix.build_minutes }}
defaults:
run:
shell: bash
strategy:
fail-fast: false # a failing platform must not stop the others from building
matrix:
include: ${{ fromJSON(needs.plan.outputs.build) }}
env:
KEY: ${{ matrix.key }}
KIND: ${{ matrix.kind }}
steps:
# Windows checks scripts out with CRLF by default, which breaks bash and mvnw: keep LF everywhere.
- name: Keep LF line endings
run: |
git config --global core.autocrlf false
git config --global core.eol lf
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# T: the runner temp directory in a form bash and the native tools both accept (D:/a/_temp on Windows)
- name: Paths
run: |
if [ "$RUNNER_OS" = Windows ]; then T="$(cygpath -m "$RUNNER_TEMP")"; else T="$RUNNER_TEMP"; fi
echo "T=$T" >> "$GITHUB_ENV"
- name: Install uv
if: ${{ matrix.kind != 'windows' }}
run: |
installer="$(mktemp)"
curl -LsSf --retry 5 --retry-all-errors --retry-delay 5 -o "$installer" "https://astral.sh/uv/${UV_VERSION}/install.sh"
sh "$installer"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Install uv (Windows)
if: ${{ matrix.kind == 'windows' }}
shell: pwsh
run: |
$env:UV_NO_MODIFY_PATH = "1"
Invoke-RestMethod "https://astral.sh/uv/$env:UV_VERSION/install.ps1" | Invoke-Expression
Add-Content -Path $env:GITHUB_PATH -Value "$env:USERPROFILE\.local\bin"
# The builder heap is 12g (8g ran out of memory on the laptop, 2026-10-25): swap keeps a 16 GB runner from being killed.
- name: Add swap
if: ${{ matrix.kind == 'linux' }}
run: |
set -euo pipefail
sudo fallocate -l 8G /swapfile.native
sudo chmod 600 /swapfile.native
sudo mkswap /swapfile.native
sudo swapon /swapfile.native
free -m
- name: Download GraalVM CE and verify its sha256
env:
GRAALVM_ASSET: ${{ matrix.graalvm_asset }}
GRAALVM_SHA256: ${{ matrix.graalvm_sha256 }}
run: |
set -euo pipefail
curl -fsSL --retry 5 --retry-all-errors --retry-delay 5 -o "$T/graalvm.archive" "$GRAALVM_BASE/$GRAALVM_ASSET"
if command -v sha256sum > /dev/null; then
echo "$GRAALVM_SHA256 $T/graalvm.archive" | sha256sum -c -
else
echo "$GRAALVM_SHA256 $T/graalvm.archive" | shasum -a 256 -c -
fi
mkdir -p "$T/graalvm"
case "$GRAALVM_ASSET" in
*.zip)
7z x -y -o"$T/graalvm-x" "$T/graalvm.archive" > /dev/null
mv "$T"/graalvm-x/*/* "$T/graalvm/"
;;
*) tar -xzf "$T/graalvm.archive" -C "$T/graalvm" --strip-components=1 ;;
esac
rm -rf "$T/graalvm.archive" "$T/graalvm-x"
home="$T/graalvm"
# the macOS archive is a bundle: the JDK home is below Contents/Home
if [ -f "$home/Contents/Home/release" ]; then home="$home/Contents/Home"; fi
echo "GRAALVM_HOME_P5=$home" >> "$GITHUB_ENV"
grep -F "GRAALVM_VERSION=\"$GRAALVM_VERSION\"" "$home/release"
ni="$home/bin/native-image"
if [ "$KIND" = windows ]; then ni="$ni.cmd"; fi
"$ni" --version
# native-image on Windows needs the MSVC toolchain on PATH and the INCLUDE/LIB variables (what the x64 Native Tools prompt sets).
- name: Load the MSVC environment
if: ${{ matrix.kind == 'windows' }}
shell: pwsh
run: |
$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
$vs = & $vswhere -latest -products '*' -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath
if (-not $vs) { throw "no Visual Studio with the C++ tools" }
$bat = Join-Path $vs 'VC\Auxiliary\Build\vcvars64.bat'
Write-Host "vcvars: $bat"
$keep = '^(PATH|INCLUDE|LIB|LIBPATH|VC.*|VS.*|WindowsSdk.*|WindowsSDK.*|UCRT.*|UniversalCRT.*|DevEnv.*|Framework.*|NETFXSDK.*|Platform|CommandPromptType)$'
cmd /c "`"$bat`" >nul && set" | ForEach-Object {
# the name and value are taken before the second -match, which would overwrite $Matches
$i = $_.IndexOf('=')
if ($i -gt 0) {
$name = $_.Substring(0, $i)
if ($name -match $keep) { Add-Content -Path $env:GITHUB_ENV -Value $_ }
}
}
- name: Hide Git's GNU link.exe (it shadows the MSVC linker)
if: ${{ matrix.kind == 'windows' }}
run: |
mv /usr/bin/link.exe /usr/bin/link-gnu.exe || true
echo "link is now: $(command -v link || echo none)"
# -march=compatibility, -J-Xmx12g, -H:-VectorAPISupport, fastops.mode 1, inside the manylinux_2_28 container (glibc 2.28) so the
# glibc floor of libarcadedb.so is a guarantee, not a property of the runner.
- name: Build the native image in manylinux_2_28
if: ${{ matrix.kind == 'linux' }}
env:
ENGINE: ${{ needs.plan.outputs.maven-engine }}
MANYLINUX_IMAGE: ${{ matrix.container }}
XMX: ${{ matrix.xmx }}
NI_EXTRA: ${{ matrix.ni_extra }}
run: |
set -euo pipefail
mkdir -p "$T/lib" "$T/m2"
docker run --rm \
-v "$GITHUB_WORKSPACE:/work" -v "$GRAALVM_HOME_P5:/graalvm:ro" -v "$T/lib:/out" -v "$T/m2:/root/.m2" \
-e GRAALVM_HOME_P5=/graalvm -e ARCADEDB_ENGINE_VERSION="$ENGINE" -e MARCH=compatibility -e XMX="$XMX" -e NI_EXTRA="$NI_EXTRA" \
"$MANYLINUX_IMAGE" bash -euo pipefail -c '
ldd --version | sed -n 1p # not `head -1`: under pipefail its early exit can SIGPIPE ldd
uname -m
# zlib-devel, unzip and which are in the manylinux_2_28 image already; dnf runs only for a missing one (the AlmaLinux 8 mirror
# metadata fails now and then). EPEL is not needed and its mirrors fail too.
if ! rpm -q zlib-devel unzip which >/dev/null; then
for i in 1 2 3 4 5; do
dnf install -y --disablerepo="epel*" --setopt=install_weak_deps=False zlib-devel unzip which >/dev/null && break
[ "$i" -lt 5 ] && { echo "dnf failed, retry $i in $((i * 20))s" >&2; sleep $((i * 20)); } || exit 1
done
fi
cd /work
bash native/scripts/build-image.sh /out
' || { tail -n 80 "$T/lib/build.log" || true; exit 1; }
# the container ran as root: give the files back to the runner user
sudo chown -R "$(id -u):$(id -g)" "$T/lib" "$GITHUB_WORKSPACE"
cat "$T/lib/build.done"
cat "$T/lib/sha256.txt"
ls -l "$T/lib"
# macOS and Windows build on the runner itself (no container): same script, capped heap/threads where the runner is small.
- name: Build the native image on the runner
if: ${{ matrix.kind != 'linux' }}
env:
ARCADEDB_ENGINE_VERSION: ${{ needs.plan.outputs.maven-engine }}
MARCH: compatibility
XMX: ${{ matrix.xmx }}
NI_EXTRA: ${{ matrix.ni_extra }}
MACOSX_DEPLOYMENT_TARGET: ${{ matrix.macos_target }}
run: |
set -euo pipefail
mkdir -p "$T/lib"
bash native/scripts/build-image.sh "$T/lib" || { tail -n 120 "$T/lib/build.log" || true; exit 1; }
cat "$T/lib/build.done"
cat "$T/lib/sha256.txt"
ls -l "$T/lib"
- name: Glibc floor of the libraries
if: ${{ matrix.kind == 'linux' }}
run: |
set -euo pipefail
for so in "$T"/lib/*.so; do
top="$(objdump -T "$so" | grep -Eo 'GLIBC_[0-9]+\.[0-9]+(\.[0-9]+)?' | sort -uV | tail -n 1)"
echo "$(basename "$so"): highest $top"
[ "$(printf '%s\nGLIBC_2.28\n' "$top" | sort -V | tail -n 1)" = "GLIBC_2.28" ] || { echo "$so needs $top, above the manylinux_2_28 floor" >&2; exit 1; }
done
- name: Build the wheel
env:
VERSION: ${{ needs.plan.outputs.version }}
ENGINE: ${{ needs.plan.outputs.engine }}
MAVEN_ENGINE: ${{ needs.plan.outputs.maven-engine }}
RELEASE: ${{ inputs.release }}
run: |
set -euo pipefail
# the Linux container logged /root/.m2/repository in the native-image classpath; on the runner that is $T/m2/repository
case "$KIND" in
linux) m2="$T/m2/repository" ;;
windows) m2="$(cygpath -m "$HOME")/.m2/repository" ;;
*) m2="$HOME/.m2/repository" ;;
esac
mkdir -p "$T/record"
# THIRD_PARTY_NOTICES.md names the engine release and its jars, so it cannot match a SNAPSHOT build (timestamped jars, no tag):
# a SNAPSHOT run skips that comparison.
extra=()
if [[ "$ENGINE" == *-SNAPSHOT ]]; then
extra=(--skip-notices-check)
if [ "${RELEASE:-}" = true ]; then
# a .devN release: the Sonatype jars the build resolved must still be the pinned ones (an upstream deploy during the build fails
# here, fail closed), and the wheel records the snapshot and the image digest it embeds
uv run --no-project --python 3.12 python -I scripts/dev_engine_pin.py check --tag "$VERSION"
extra+=(--engine-pin)
fi
fi
if [[ "$MAVEN_ENGINE" != "$ENGINE" ]]; then
# the arcadedb jars of the native-image classpath (build.log), read from the Maven repository, are the pinned jars byte for byte: a
# floating SNAPSHOT or a sibling module that Maven resolved on its own differs here (the file names cannot tell: Maven names a
# resolved snapshot jar with its base version)
uv run --no-project --python 3.12 python -I scripts/dev_engine_pin.py verify-build-log "$T/lib/build.log" "$m2"
fi
uv run --no-project --python 3.12 python native/wheel/build_wheel.py --lib-dir "$T/lib" --version "$VERSION" --engine "$ENGINE" \
--graalvm-home "$GRAALVM_HOME_P5" --m2-repo "$m2" \
--notices-out "$T/record/THIRD_PARTY_NOTICES.generated.md" --modules-out "$T/record/jdk-modules.txt" --out "$T/wheel" ${extra[@]+"${extra[@]}"}
ls -l "$T/wheel/dist"
shopt -s nullglob
wheels=("$T"/wheel/dist/*.whl)
[ "${#wheels[@]}" = 1 ]
- name: "Gate: auditwheel grants the manylinux tag, no ISA warning"
if: ${{ matrix.kind == 'linux' }}
env:
PLAT: ${{ matrix.plat_re }}
run: |
set -euo pipefail
w="$(ls "$T"/wheel/dist/*.whl)"
uvx auditwheel show "$w" | tee "$T/auditwheel.txt"
# auditwheel wraps its report to the terminal width, so match on one line with the whitespace collapsed
tr -s '[:space:]' ' ' < "$T/auditwheel.txt" > "$T/auditwheel.flat"
grep -F 'consistent with the following platform tag' "$T/auditwheel.flat"
# auditwheel names the LOWEST tag the wheel is consistent with. The wheel carries the tag of its build container (build_wheel.py
# --glibc-policy); the library may need less (its glibc symbols stop at 2.17, jvector's libstdc++ asks for more than manylinux2014),
# never more.
granted="$(grep -Eo 'consistent with the following platform tag: "manylinux_[0-9]+_[0-9]+' "$T/auditwheel.flat" | grep -Eo 'manylinux_[0-9]+_[0-9]+' || true)"
want="$(printf '%s' "$PLAT" | grep -Eo 'manylinux_[0-9]+_[0-9]+')"
echo "auditwheel grants ${granted:-an older policy}; the wheel is tagged $want"
[ "$(printf '%s\n%s\n' "${granted:-manylinux_2_5}" "$want" | sort -V | tail -n 1)" = "$want" ] || { echo "auditwheel needs a tag above $want" >&2; exit 1; }
if grep -Eiq 'instruction set|x86_64_v[0-9]' "$T/auditwheel.flat"; then
echo "auditwheel reports an ISA warning: the image is not baseline x86-64" >&2
exit 1
fi
# The tests run on the runner (glibc 2.39). This step runs the wheel where its tag says it must work: in the manylinux_2_28 container
# (glibc 2.28), on that image's Python 3.12, with nothing but the wheel installed.
- name: "Gate: the wheel installs and opens a database on glibc 2.28"
if: ${{ matrix.kind == 'linux' }}
env:
MANYLINUX_IMAGE: ${{ matrix.container }}
run: |
set -euo pipefail
# The runner's uv is mounted into the container (a glibc 2.17 build); the image's own Python is used, never downloaded.
# --no-index: nothing but the wheel is installed.
docker run --rm -e UV_PYTHON_DOWNLOADS=never -v "$(command -v uv):/usr/local/bin/uv:ro" -v "$T/wheel/dist:/wheel:ro" -v "$GITHUB_WORKSPACE/scripts/smoke_wheel.py:/smoke_wheel.py:ro" "$MANYLINUX_IMAGE" bash -euo pipefail -c '
ldd --version | sed -n 1p
cd /tmp
uv run --no-project --no-index --python /opt/python/cp312-cp312/bin/python --with /wheel/*.whl python /smoke_wheel.py
'
- name: "Gate: the macOS library links system libraries only"
if: ${{ matrix.kind == 'macos' }}
run: |
set -euo pipefail
w="$(ls "$T"/wheel/dist/*.whl)"
mkdir -p "$T/unz"
unzip -q -o "$w" -d "$T/unz"
dylib="$(ls "$T"/unz/arcadedb/_native/libarcadedb.dylib)"
otool -L "$dylib"
otool -l "$dylib" | grep -A4 -E 'LC_BUILD_VERSION|LC_VERSION_MIN' || true
lipo -info "$dylib"
# delocate lists the libraries the wheel would have to bundle: none may remain (system libraries are not listed)
deps="$(uvx --from delocate delocate-listdeps "$w")"
printf 'delocate-listdeps: [%s]\n' "$deps"
[ -z "$deps" ]
# Informational: the imported DLLs, so a dependency on the Visual C++ redistributable would be visible.
- name: List the DLLs the Windows library imports
if: ${{ matrix.kind == 'windows' }}
run: |
set -euo pipefail
dll="$(ls "$T"/lib/libarcadedb.dll)"
dumpbin //dependents "$dll" | tee "$T/dumpbin.txt"
- name: "Gate: a Linux wheel carries jvector's library for its architecture"
if: ${{ matrix.kind == 'linux' }}
env:
PLAT: ${{ matrix.plat_re }}
run: |
set -euo pipefail
w="$(ls "$T"/wheel/dist/*.whl)"
arch="${KEY#linux-}" # linux-x86_64 -> x86_64, linux-aarch64 -> aarch64 (${PLAT##*_} gave "64" for x86_64)
unzip -Z1 "$w" | tee "$T/wheel-files.txt" > /dev/null
grep -Fx "arcadedb/_native/libarcadedb.so" "$T/wheel-files.txt"
grep -Fx "arcadedb/_native/libjvector-${arch}.so" "$T/wheel-files.txt"
- name: "Gate: twine check"
run: uvx twine check --strict "$T"/wheel/dist/*.whl
- name: "Gate: the wheel is arcadedb, its version equals the requested (tag) version, the tag fits the platform"
env:
VERSION: ${{ needs.plan.outputs.version }}
PLAT_RE: ${{ matrix.plat_re }}
run: |
set -euo pipefail
w="$(ls "$T"/wheel/dist/*.whl)"
uv run --no-project --with packaging python - "$w" "$VERSION" "$PLAT_RE" <<'PY'
import re
import sys
from packaging.utils import parse_wheel_filename
from packaging.version import Version
name, version, _build, tags = parse_wheel_filename(sys.argv[1].replace("\\", "/").rsplit("/", 1)[-1])
want = Version(sys.argv[2])
print(f"wheel {name} {version} {sorted(map(str, tags))}; requested {want}")
assert str(name) == "arcadedb", name
assert str(want) == sys.argv[2], f"{sys.argv[2]} is not the canonical PEP 440 form {want}: fix the tag"
assert version == want, f"wheel version {version} != requested {want}"
assert len(tags) == 1, tags
(tag,) = tags
assert (tag.interpreter, tag.abi) == ("py3", "none"), tag
assert re.fullmatch(sys.argv[3], tag.platform), f"platform tag {tag.platform} does not match {sys.argv[3]}"
PY
# also on failure: the generated notices and the full build log are what a drift needs to be diagnosed
- name: Collect the build record
if: ${{ always() }}
run: |
mkdir -p "$T/dist-out" "$T/record"
cp "$T/lib/build.log" "$T/record/build.log" || true
cp "$T"/wheel/dist/*.whl "$T/dist-out/" || true
for f in auditwheel.txt dumpbin.txt; do
if [ -f "$T/$f" ]; then cp "$T/$f" "$T/record/"; fi
done
cp "$T/lib/sha256.txt" "$T/lib/build.done" "$T/lib/used_classes.txt" "$T/record/" || true
tail -n 400 "$T/lib/build.log" > "$T/record/build.log.tail" || true
(cd "$T/dist-out" && if command -v sha256sum > /dev/null; then sha256sum ./*.whl; else shasum -a 256 ./*.whl; fi) | tee "$T/record/wheel.sha256" || true
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: native-wheel-${{ matrix.key }}
path: ${{ runner.temp }}/dist-out/*.whl
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: ${{ always() }}
with:
name: native-build-record-${{ matrix.key }}
path: ${{ runner.temp }}/record/
if-no-files-found: ignore
retention-days: 30
test:
name: Parity against the installed wheel (${{ matrix.key }}, Python ${{ matrix.python }})
needs: [plan, build]
# A platform whose build failed has no wheel to download and shows red here; the other platforms still run.
if: ${{ !cancelled() && needs.plan.outputs.run == 'true' }}
runs-on: ${{ matrix.runner }}
timeout-minutes: ${{ matrix.test_minutes }}
defaults:
run:
shell: bash
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.plan.outputs.test) }}
env:
KEY: ${{ matrix.key }}
PY: ${{ matrix.python }}
steps:
- name: Keep LF line endings
run: |
git config --global core.autocrlf false
git config --global core.eol lf
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Paths
run: |
if [ "$RUNNER_OS" = Windows ]; then T="$(cygpath -m "$RUNNER_TEMP")"; else T="$RUNNER_TEMP"; fi
echo "T=$T" >> "$GITHUB_ENV"
- name: Install uv
if: ${{ runner.os != 'Windows' }}
run: |
installer="$(mktemp)"
curl -LsSf --retry 5 --retry-all-errors --retry-delay 5 -o "$installer" "https://astral.sh/uv/${UV_VERSION}/install.sh"
sh "$installer"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Install uv (Windows)
if: ${{ runner.os == 'Windows' }}
shell: pwsh
run: |
$env:UV_NO_MODIFY_PATH = "1"
Invoke-RestMethod "https://astral.sh/uv/$env:UV_VERSION/install.ps1" | Invoke-Expression
Add-Content -Path $env:GITHUB_PATH -Value "$env:USERPROFILE\.local\bin"
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: native-wheel-${{ matrix.key }}
path: ${{ runner.temp }}/wheel
# The tests of this checkout run in a clean environment that contains the wheel and the test dependencies, not the source tree.
# The repository tooling tests that are about the wheel template and the matrix run here too (they need the checkout).
- name: Run the parity suite
env:
FINAL: ${{ needs.plan.outputs.final }}
run: |
set -euo pipefail
w="$(ls "$T"/wheel/*.whl)"
mkdir -p "$T/parity"
cores="$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 2)"
limit=()
if [ "$FINAL" = true ]; then limit=(--max-notyet 0); fi
# A file whose process dies without a result is run once more on Windows only (the crashes of issue 295 are Windows only);
# Linux and macOS stay strict. The harness marks a passing retry FLAKY-CRASH and the crash-report job logs it.
retry=()
if [ "$RUNNER_OS" = Windows ]; then retry=(--retry-crash); fi
# macOS runs bash 3.2: under set -u an empty "${limit[@]}" is "unbound variable", so expand it only when set
uv run --no-project --python "$PY" --with "$w" --with pytest --with setuptools --with numpy --with requests --with jpype1 --with pyarrow \
--with polars --with pandas --with duckdb --with networkx --with "psycopg[binary]" --with adbc-driver-postgresql \
--with redis --with neo4j --with ladybug \
python native/wheel/parity_wheel.py --repo "$GITHUB_WORKSPACE" \
--out "$T/parity/parity-wheel.tsv" --cores "0-$((cores - 1))" ${limit[@]+"${limit[@]}"} ${retry[@]+"${retry[@]}"} | tee "$T/parity/parity-wheel.run.log"
- name: "Gate: no failure, error, crash, timeout or uncounted skip"
run: |
set -euo pipefail
tsv="$T/parity/parity-wheel.tsv"
# the pyver job has a copy of this awk: keep the two in sync
awk -F'\t' '
NR == 1 { next }
$1 == "TOTAL" { passed = $2; failed = $3; err = $4; notyet = $5; na = $6; skipped = $7; tot = 1; next }
$8 ~ /^FLAKY-CRASH/ { flaky = flaky " " $1 "(" $8 ")"; next }
$8 != "" { bad = bad " " $1 "(" $8 ")" }
END {
if (!tot) { print "no TOTAL row" > "/dev/stderr"; exit 1 }
printf "parity %s py%s: passed=%s failed=%s error=%s notyet=%s na_jvm=%s skipped=%s\n", ENVIRON["KEY"], ENVIRON["PY"], passed, failed, err, notyet, na, skipped
# a file that crashed and passed when run once more: the leg may pass, the summary says it did only because of the retry
if (flaky != "") printf "FLAKY-CRASH (died without a result, passed on the one retry; first exit code in brackets):%s\n", flaky
if (bad != "") { print "files with a status:" bad > "/dev/stderr"; exit 1 }
if (passed + 0 == 0 || failed + 0 != 0 || err + 0 != 0) exit 1
# this repository accepts no skips; NOTYET and NA-JVM are the two counted kinds of the native build
if (skipped + 0 != 0) { print "skipped tests that are neither NOTYET nor NA-JVM: " skipped > "/dev/stderr"; exit 1 }
}' "$tsv" | tee -a "$GITHUB_STEP_SUMMARY"
# Reached only when the gate passed: release-native.yml reads these markers (one per platform and Python, holding the wheel file name).
- name: Mark this leg as passed
run: |
set -euo pipefail
mkdir -p "$T/passed"
basename "$(ls "$T"/wheel/*.whl)" > "$T/passed/$KEY-py$PY.ok"
cat "$T/passed/$KEY-py$PY.ok"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: native-passed-${{ matrix.key }}-py${{ matrix.python }}
path: ${{ runner.temp }}/passed/${{ matrix.key }}-py${{ matrix.python }}.ok
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: ${{ !cancelled() }}
with:
name: native-parity-${{ matrix.key }}-py${{ matrix.python }}
path: |
${{ runner.temp }}/parity/parity-wheel.tsv
${{ runner.temp }}/parity/parity-wheel.meta.json
${{ runner.temp }}/parity/parity-wheel.run.log
${{ runner.temp }}/parity/parity-wheel-xml/
if-no-files-found: warn
retention-days: 30
# The one job that may write to an issue: after the parity legs it reads their meta files (parity-wheel.meta.json in each native-parity-*
# artifact) and lists every FLAKY-CRASH and CRASH in the job summary; on a release (a tag push) or a dispatch it also posts ONE comment on
# issue 295 (native/wheel/report_flaky_crash.py decides: never on a pull request, only in ArcadeData/arcadedb-python). A log is not a gate: a token
# that cannot write issues, a missing artifact or a gh failure print a warning and the job still ends green. release-native.yml grants the
# issues permission to its call of this workflow, because a called workflow cannot hold more permission than its caller gives it.
crash-report:
name: Report parity crashes
needs: [plan, test]
if: ${{ !cancelled() && needs.plan.outputs.run == 'true' }}
runs-on: ubuntu-24.04
timeout-minutes: 10
continue-on-error: true
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
continue-on-error: true
with:
pattern: native-parity-*
path: ${{ runner.temp }}/parity-artifacts
- name: List the crashes and log them on issue 295
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: python3 -I native/wheel/report_flaky_crash.py --artifacts "$RUNNER_TEMP/parity-artifacts"
# The examples of examples/ against the INSTALLED native wheel, so the scripts a new user copies are run on what `uv add arcadedb` installs.
# Nothing is built here: the wheel is the one the build job uploaded (native-wheel-<key>), the file the parity job installs. The platforms are
# those of the parity job and the Pythons are 3.12 and 3.14 (scripts/select_native_matrix.py, `examples`). The runner is scripts/run_examples.sh, the
# same script test-python-examples.yml calls for the JVM package, so the arguments, timeouts, datasets and answer checks are the same.
# An example that needs the JVM package is listed with its reason in scripts/native_examples_skip.txt; the step prints that list.
# A failing example turns this job, and so the workflow, red. release-native.yml reads only the native-passed-* markers and the native-wheel-*
# wheels, so these legs do not decide what a release publishes.
examples:
name: Examples on the installed wheel (${{ matrix.key }}, Python ${{ matrix.python }})
needs: [plan, build]
# A platform whose build failed has no wheel to download and shows red here; the other platforms still run.
if: ${{ !cancelled() && needs.plan.outputs.run == 'true' }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 240
defaults:
run:
shell: bash
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.plan.outputs.examples) }}
env:
KEY: ${{ matrix.key }}
PY: ${{ matrix.python }}
EXAMPLES: "0[1-9]_*.py 1[0-9]_*.py 2[0-9]_*.py 3[0-9]_*.py"
EXAMPLES_SKIP_FILE: scripts/native_examples_skip.txt
# The Hugging Face cache inside the workspace, as in test-python-examples.yml: one path on every platform, cacheable.
HF_HOME: ${{ github.workspace }}/.hf-cache
PYTHONIOENCODING: utf-8
steps:
- name: Keep LF line endings
run: |
git config --global core.autocrlf false
git config --global core.eol lf
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Paths
run: |
if [ "$RUNNER_OS" = Windows ]; then T="$(cygpath -m "$RUNNER_TEMP")"; else T="$RUNNER_TEMP"; fi
echo "T=$T" >> "$GITHUB_ENV"
- name: Install uv
if: ${{ runner.os != 'Windows' }}
run: |
installer="$(mktemp)"
curl -LsSf --retry 5 --retry-all-errors --retry-delay 5 -o "$installer" "https://astral.sh/uv/${UV_VERSION}/install.sh"
sh "$installer"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Install uv (Windows)
if: ${{ runner.os == 'Windows' }}
shell: pwsh
run: |
$env:UV_NO_MODIFY_PATH = "1"
Invoke-RestMethod "https://astral.sh/uv/$env:UV_VERSION/install.ps1" | Invoke-Expression
Add-Content -Path $env:GITHUB_PATH -Value "$env:USERPROFILE\.local\bin"
# The wheel of the build job of this platform, not a rebuild.
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: native-wheel-${{ matrix.key }}
path: ${{ runner.temp }}/wheel
# The wheel and the example dependencies live in a uv project of their own (examples-env/, as in test-python-examples.yml), synced with
# `uv sync`. The wheel has no dependencies (jpype1 is not installed: a user of `uv add arcadedb` does not have it). torch comes from the
# CPU-only PyTorch index (explicit: only torch is looked up there). numpy is not capped (see test-python-examples.yml).
- name: Install the wheel and the example dependencies
run: |
set -euo pipefail
wheels=("$T"/wheel/arcadedb-*.whl)
[ "${#wheels[@]}" -eq 1 ] || { echo "expected one wheel in $T/wheel, found ${#wheels[@]}" >&2; exit 1; }
mkdir -p examples-env
cat > examples-env/pyproject.toml <<TOML
[project]
name = "examples-env"
version = "0"
requires-python = ">=3.11"
dependencies = [
"arcadedb",
"torch",
"numpy",
"requests",
"sentence-transformers",
"pandas",
"polars",
"pyarrow",
"tqdm",
"py7zr",
"lxml",
]
[tool.uv.sources]
arcadedb = { path = "${wheels[0]}" }
torch = { index = "pytorch-cpu" }
[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
TOML
uv sync --project examples-env --python "$PY"
# The examples run with this interpreter directly, so a timeout kills the example itself and not a `uv run` wrapper.
ex_py="$(uv run --project examples-env --no-sync python -c 'import sys; print(sys.executable)')"
echo "EX_PY=$ex_py" >> "$GITHUB_ENV"
"$ex_py" -c "import arcadedb; print('arcadedb', arcadedb.__version__, arcadedb.__file__)"
- name: Cache Hugging Face model
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ github.workspace }}/.hf-cache
key: hf-all-MiniLM-L6-v2-${{ runner.os }}-${{ runner.arch }}
restore-keys: |
hf-all-MiniLM-L6-v2-${{ runner.os }}-
# Not fail-fast, as in test-python-examples.yml: a hub outage here must not hide behind a retry, the dataset step fails with its own message.
- name: Pre-warm embedding model
continue-on-error: true
run: |
for attempt in 1 2 3; do
if "$EX_PY" -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2'); print('model ready')"; then
echo "model cached on attempt $attempt"
exit 0
fi
echo "::warning::model fetch failed (attempt $attempt/3)"
[ "$attempt" -lt 3 ] && sleep $((attempt * 20))
done
echo "::warning::could not pre-warm the embedding model after 3 attempts; the dataset step will retry"
- name: Download datasets
run: bash scripts/run_examples.sh datasets
- name: Install timeout command (macOS only)
if: ${{ runner.os == 'macOS' }}
run: brew install coreutils
- name: Run all examples
id: run_examples
run: bash scripts/run_examples.sh run
- name: Generate test summary
if: ${{ always() }}
run: |
cd examples
{
echo "## Examples on the installed wheel ($KEY, Python $PY)"
echo
if [ "${{ steps.run_examples.outcome }}" = "success" ]; then
echo "**Status**: all examples passed (${{ steps.run_examples.outputs.passed || '0' }}/${{ steps.run_examples.outputs.total || '0' }})"
else
echo "**Status**: some examples failed (${{ steps.run_examples.outputs.passed || '0' }}/${{ steps.run_examples.outputs.total || '0' }} passed)"
fi
echo
echo "| Metric | Count |"
echo "|--------|------:|"
echo "| Total | ${{ steps.run_examples.outputs.total || '0' }} |"
echo "| Passed | ${{ steps.run_examples.outputs.passed || '0' }} |"
echo "| Failed | ${{ steps.run_examples.outputs.failed || '0' }} |"
echo "| Skipped | ${{ steps.run_examples.outputs.skipped || '0' }} |"
echo
echo "### Skip list (scripts/native_examples_skip.txt)"
echo
echo '```'
grep -v -e '^[[:space:]]*#' -e '^[[:space:]]*$' ../scripts/native_examples_skip.txt || echo "(empty: no example is skipped)"
echo '```'
if [ -f example-results.txt ]; then
echo
echo "### Detailed results"
echo
echo '```'
cat example-results.txt
echo '```'
fi
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: ${{ !cancelled() }}
with:
name: native-examples-${{ matrix.key }}-py${{ matrix.python }}
path: |
examples/*.log
examples/example-results.txt
if-no-files-found: warn
retention-days: 7
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: ${{ failure() }}
with:
name: native-examples-databases-${{ matrix.key }}-py${{ matrix.python }}
path: examples/my_test_databases/
if-no-files-found: ignore
retention-days: 3
# One check for the examples of every platform and Python: red when any leg of the `examples` job failed or the job did not run to the end.
examples-summary:
name: Examples summary
needs: [plan, examples]
if: ${{ always() && needs.plan.outputs.run == 'true' }}
runs-on: ubuntu-24.04
timeout-minutes: 5
steps:
- name: Check the examples legs
env:
RESULT: ${{ needs.examples.result }}
run: |
{
echo "## Examples on the installed wheel"
echo
if [ "$RESULT" = success ]; then
echo "All example legs passed."
else
echo "The examples job ended with: $RESULT. See the legs of the job \"Examples on the installed wheel\"."
fi
} >> "$GITHUB_STEP_SUMMARY"
[ "$RESULT" = success ]
# The cheap Python-version coverage of the wheel of linux-x86_64 (py3-none: the same file for every Python, nothing is rebuilt): a subset of
# the test files, not the whole suite. The legs are the supported Pythons the full parity runs of this event leave out, plus the
# free-threaded 3.14t (scripts/select_native_matrix.py).
# Part of the release gate (ArcadeData/arcadedb-python#349): a leg that passes uploads the marker native-passed-pyver-py<version> (the file name of
# the wheel it installed), and `verify` in release-native.yml requires the markers of the legs a release has (3.15 and 3.14t, see
# scripts/select_native_matrix.py), so a red leg stops the release. Only the linux-x86_64 wheel is checked on these Pythons.
# 3.14t runs without polars, and so without test_dataframe_ingest.py (it imports polars at module level): polars ships abi3 wheels only
# (polars and polars-runtime-32 have no cp314t wheel, and an abi3 wheel cannot be installed on a free-threaded build). The other
# dependencies of the subset (numpy, pandas, pyarrow, jpype1, requests) all have cp314t wheels. polars is an optional extra of the wheel.
pyver:
name: Python ${{ matrix.python }} on the wheel (linux-x86_64)
needs: [plan, build]
# The linux-x86_64 build may have failed: then there is no wheel to download and this shows red, as the test job does.
if: ${{ !cancelled() && needs.plan.outputs.run == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 20
defaults:
run:
shell: bash
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.plan.outputs.pyver) }}
env:
KEY: linux-x86_64
PY: ${{ matrix.python }}
WITH_POLARS: ${{ matrix.polars }}
WITH_JPYPE: ${{ matrix.jpype }}
FREE_THREADED: ${{ matrix.freethreaded }}
# The subset: ctypes layer and database lifecycle (test_core, test_native_udf), queries and results (test_resultset), parameters
# (test_parameter_binding), type conversion (test_type_conversion), transactions (test_transaction_config), async (test_async_executor),