Generated by scripts/bench_compare.py (N=20, warmup excluded). speedup = reference / ggml (>1 means ggml is faster). Absolute numbers are machine-specific.

• Machine: AMD Ryzen 9 9950X3D 16-Core Processor (Zen5, 2 CCDs)
• ggml side: facedetect-cli bench, device cpu, build -DGGML_NATIVE=OFF (AVX2/FMA + tinyBLAS)
• Reference: CPU onnxruntime (intra-op = threads, inter-op = 1), cv2.setNumThreads(threads), matched to the ggml --threads
• Versions: insightface 1.0.1, onnxruntime 1.27.0, opencv 4.13.0, numpy 2.4.6
• Image: tests/fixtures/face_a.jpg

insightface FaceAnalysis.get ALSO runs the genderage + 2d106/1k3d68 landmark heads, so the full-pipeline reference does strictly more work than the ggml detect->align->embed pipeline (noted in the table).

The process-global backend used to default to 1 thread (fd::global_backend() created Backend(1)), below even ggml's own default of 4. These conv/matmul forwards are compute-bound and scale near-linearly to a handful of threads, so every library/CLI consumer that did not explicitly pass --threads was paying a ~3.3x latency tax. The default is now min(hardware_concurrency, 8) with a FACEDETECT_THREADS env override; fd::set_num_threads() / CLI --threads still win at runtime. Parity is unchanged (ggml matmul reduction order is thread-count-independent; detector boxes/landmarks <= 1 px and embedding cosine 1.000000 hold at any thread count - re-gated at 8 threads).

The shared fd::conv2d (SCRFD detector + ArcFace recognizer, both dominated by 3x3 stride-1 convs) always used ggml_conv_2d (im2col + tinyBLAS mul_mat). im2col materializes a 9x patch buffer per conv - a large memory stream. ggml_conv_2d_direct has no patch buffer (far less memory traffic) but a less throughput-dense kernel. The two trade off by memory-bandwidth pressure, which is set by the thread count.

Measured crossover (buffalo_l 640x640 detect, f32, min-of-5 isolated runs, this 16-core box; the box was under heavy concurrent load so absolute numbers are noisy and reported as the least-contended minimum):

So the routing is now: direct for K>1 convs on CPU when n_threads >= 8, else im2col; 1x1 convs always stay im2col (pure GEMM, no 9x expansion); GPU always im2col (cuBLAS-class mul_mat beats the basic direct CUDA conv). The process-global default is min(hw,8), so an 8+-logical-core box (the common deployment target) lands in direct's win region while smaller boxes default below the crossover and keep im2col - no regression at any measured thread count. A/B via FACEDETECT_CONV=direct|im2col|auto.

Parity preserved (re-gated at 8 threads): scrfd raw heads max|d| ~1e-6 vs ONNX (1e-3 gate); detect boxes/landmarks <= 0.73 px (1 px gate); ArcFace stem/block + golden-landmark + isolated embed cosine = 1.000000 (>= 0.9999 gate). The direct kernel changes float accumulation order only (diffs ~1e-6), not the result.

• Blanket switch to ggml_conv_2d_direct for all CPU thread counts (the naive depth-anything recipe): NET NEGATIVE at low thread counts. direct regresses single-thread detect by +29% (594.8 -> 768.3 ms) and 2-thread by +17%, because with spare memory bandwidth im2col's tinyBLAS GEMM out-throughputs the direct kernel. The win only appears once threads saturate bandwidth (>= 8 here). Hence the thread-count gate, not a blanket switch. (depth-anything's DPT head convs run at full image resolution -> bandwidth-bound even single-thread, so direct won there unconditionally; SCRFD/ArcFace feature maps are smaller, so the crossover sits at a higher thread count.)

A second optimization round stacks two parity-safe levers on top of the pass-2 baseline (523aee1, the conv_2d_direct thread-gated routing): a custom AVX2 Winograd F(2x2,3x3) for the large 3x3 stride-1 convs, and an ArcFace BatchNorm fold to host constants. Measured contention-robust: back-to-back A/B (baseline binary vs optimized binary in the same session), median-of-N, on a loaded 16-core box. The absolute ms below are load-specific - treat the speedups, not the raw numbers, as the signal. Parity held after each lever (SCRFD boxes/landmarks <= 1 px; embedding cosine = 1.0 on the isolated and golden-landmark paths).

The headline: the custom AVX2 Winograd F(2x2,3x3) BEAT tinyBLAS sgemm, which refutes the doubt recorded in the prior pass (pass 2 deferred Winograd on the grounds that it "must beat llamafile's already-tuned sgemm, which even plain direct conv loses to" - this round implemented it and it won). It re-targets the 3x3 stride-1 pad-1 convs that dominate the SCRFD backbone/neck/head stems (min feature-map dim >= 80) plus the ArcFace 112x112 stem, with the transformed filter cached once per shape and tiles split across ggml threads. Detect goes 1.37x at 1 thread and ~1.7-2.05x at 8 threads; the full buffalo_l pipeline inherits 1.28x / 1.58-1.77x.

The ArcFace recognizer lever folds the inference-mode BatchNorm affine (scale = gamma/sqrt(var+eps), shift = beta - mean*scale) to host constants once per loaded model, collapsing a ~5-op live-node chain repeated ~25x per embed to a single mul + add per BN (algebraic constant folding, parity preserved). PReLU could NOT be fused without breaking the GPU path - this ggml exposes neither a per-channel PReLU nor a binary max op, so a single fused PReLU op would require a CPU-only custom op; left numerically identical (documented).

These deltas are measured vs our OWN prior ggml baseline (523aee1), NOT a fresh head-to-head vs onnxruntime this round. The wins NARROW the gap to onnxruntime substantially (SCRFD detect ~1.7-2x at 8 threads; sibling voice-detect.cpp ERes2Net ~1.3-1.6x at 8t), but on the conv-bound face / encoder models a residual gap remains intrinsic: ggml's im2col / Winograd conv on small/mid feature maps vs onnxruntime's tuned conv kernels. GEMM-bound transformer heads (e.g. the voice-detect.cpp wav2vec2 analyze heads) are the most likely to reach near-parity, especially on a gated AVX512 build. No model regressed.

• Absolute latency improved 3.2-3.9x by defaulting to 8 threads (e.g. SCRFD detect 515.0 -> 153.9 ms, full pipeline 667.9 -> 201.9 ms). For any consumer that took the library default this is a real, parity-safe, out-of-the-box win.
• The ratio vs the reference did NOT improve - onnxruntime/OpenCV multithread at least as well (the detector reference scales ~4.5x, ggml ~3.3x), so the gap is intrinsic, not a thread-count artifact. ggml's conv (im2col + mul_mat) stays ~2-6x behind onnxruntime's dedicated, cache-blocked conv kernels on CPU.
• Realistic CPU ceiling. At matched threads, ggml will not beat onnxruntime on these conv-heavy detectors/recognizers on CPU; the value of the port remains no-Python deployment, a small libfacedetect.so, GGUF quantization, and one code path across ggml GPU backends. The thread default removes the artificial 1-thread penalty; it does not change the per-core conv-kernel gap.

Time was attributed by thread-scaling (a forward that scales ~linearly with threads is compute-bound; host work does not scale) and by mode isolation (recognizer mode times the embed graph alone on a pre-resized crop).

• SCRFD detect (640x640): dominated by the conv graph compute (ggml_backend_graph_compute). 503 ms @ 1t -> ~128 ms @ 8t, plateauing at ~6-8 threads (the large im2col is memory-bandwidth bound). Host-side letterbox + anchor decode + NMS is negligible (it would not scale 4x otherwise; the anchor grids are tiny scalar loops over a few thousand candidates).
• ArcFace / SFace recognizer (112x112): compute-bound matmul/conv, 156 ms @ 1t -> ~44 ms @ 8t, also plateauing ~8 threads.
• Full pipeline / analyze: detect dominates (detect ~154 ms vs embed ~45 ms at 8t), so the detector's conv kernel is the binding constraint.

• -DGGML_NATIVE=ON (AVX512 on Zen5): NET NEGATIVE for this workload. It helps the small recognizer (embed 44 -> 33 ms @ 8t) but significantly regresses the dominant bandwidth-bound SCRFD detector (detect 128 -> 166 ms @ 8t; 503 -> 766 ms @ 1t) - classic AVX512 down-clocking hurting a memory-bound im2col conv while the detector dominates the pipeline. Kept the build at AVX2/FMA + tinyBLAS (GGML_LLAMAFILE), which is what the headline table above uses.

Appended by scripts/gpu_verify.sh on a CUDA host (not run here - no GPU).

Built -DFACEDETECT_GGML_CUDA=ON -DGGML_CUDA_NO_VMM=ON -DGGML_NATIVE=OFF -DCMAKE_CUDA_ARCHITECTURES=121 (ggml auto-promotes 121 -> 121a for GB10). Device forced via FACEDETECT_DEVICE=CUDA0; the CLI confirms fd::Backend using device: CUDA0 (GB10, compute 12.1, 122 GB unified VRAM). ggml-GPU and ggml-CPU measured on the SAME GB10 box (N=30, warmup excluded, default min-of-hw-and-8 threads). The reference column is the CPU onnxruntime numbers from the 8-thread CPU table above, measured on an AMD Ryzen 9 9950X3D - a DIFFERENT machine, so GPU-vs-reference is cross-machine and indicative only.

GPU is 4-12x faster than the same-box CPU and beats the cross-machine CPU onnxruntime reference (2.3x-5.1x). The GPU port is competitive.

NOTE: the committed test binaries hard-pin setenv(FACEDETECT_DEVICE,"cpu",overwrite=1), which would silently override the GPU device. For this GPU run the DGX test copies were patched to overwrite=0 (an explicit env wins, default stays cpu) so the gates actually exercise CUDA. This patch is NOT in the source tree and should be wired into the harness.

• buffalo_l ArcFace recognizer:
  • golden-landmark path (decode -> norm_crop -> embed): cosine 1.000000, max|d| <= 1.2e-4 on ALL THREE fixtures -> parity HOLDS (strict 1e-3 bound met).
  • golden-crop (isolated) path: cosine 1.000000, max|d| 1.0e-4 -> HOLDS.
  • full production path (detect -> align -> embed): cosine 0.999945 (gate >= 0.9999 PASS); max|d| 1.5e-3 is REPORTED-not-gated (sub-pixel landmark drift, same as on CPU).
• buffalo_l SCRFD detector: top score 0.90054 vs ref 0.90063; max box-corner err 0.0217 px, max landmark err 0.0193 px -> parity HOLDS (1 px gate).
• buffalo_m / buffalo_s / YuNet / SFace: no committed reference golden in-tree, so GPU was NOT parity-checked vs reference this run (benched only).
• Teardown SIGABRT (benign): test_embedding prints all parity comparisons as PASS, then aborts at process exit with CUDA error: driver shutting down in ~ggml_backend_cuda_buffer_context (ggml-cuda.cu:637). This is a static-destruction-ordering artifact (the CUDA driver tears down before the ggml buffer destructor runs); the voice tests avoid it by calling vd::shutdown_backend() explicitly. It does NOT affect the parity results, which are all emitted before the abort.

The GPU section above compares ggml-GPU against the reference on a DIFFERENT machine's CPU, which is not a fair test. This section measures the REFERENCE on the SAME NVIDIA GB10 (Grace-Blackwell, sm_121a, CUDA 13.0).

• Reference frameworks (all confirmed resident on the GB10 GPU):
  • insightface 1.0.1 FaceAnalysis(..., providers=["CUDAExecutionProvider"], ctx_id=0) for the buffalo_l / buffalo_s end-to-end pipelines. Per-model session.get_providers() printed ['CUDAExecutionProvider', ...] for detection / recognition / genderage / 2d106 (insightface keeps only the 3d68 landmark head on CPU).
  • onnxruntime-gpu 1.24.0 (CUDAExecutionProvider) for the strict per-model ONNX forwards (SCRFD det_10g, ArcFace w600k_r50, YuNet, SFace). Wheel: onnxruntime_gpu-1.24.0-cp312-linux_aarch64 from the NVIDIA jetson-ai-lab SBSA/cu130 index (PyPI onnxruntime-gpu is x86-only). CUDA EP asserted first provider per session.
  • opencv-python 4.13.0 cv2.FaceDetectorYN/FaceRecognizerSF (the YuNet+SFace cv2 reference) cannot run on the GPU: the pip opencv-python DNN module has no CUDA backend. So the YuNet+SFace GPU reference is the YuNet + SFace ONNX forwards run through ORT CUDA EP instead (align/NMS stay on host, as they do on both sides).
• Method: warmup 20, N=100 timed, per-call median (ms). Same image (face_a.jpg, 641x800). speedup = reference-GPU / ggml-GPU (>1 means ggml-GPU is faster).

On the SAME GB10 GPU and a like-for-like detect+align+embed workload, the reference is faster on every pipeline: ~2.1x on buffalo_l (SCRFD+ArcFace ONNX forwards 9.39 ms vs ggml 19.99 ms) and ~2.9x on YuNet+SFace (2.06 ms vs 5.89 ms). ggml-GPU only "beats" the insightface full pipeline (34.45 ms) because insightface additionally runs the genderage + 2d106 + 3d68 landmark heads (strictly more work); against a same-work GPU reference ggml-GPU does NOT win. cuDNN's conv kernels (ORT CUDA EP) still out-run the ggml CUDA conv graph on these conv-dominated detectors/recognizers. Note: the YuNet+SFace cv2 DNN reference could NOT be GPU-benched (opencv-python has no CUDA backend); the ORT-CUDA ONNX forwards are used as the GPU reference instead. All ONNX/insightface models DO run on Blackwell sm_121a (no model was blocked).