Visitar URL original
docs/known-bugs: gen_simple JIT-overhead known-cost (Phoenix-stable) · SonicField/cpython@12ed40d · GitHub
Skip to content

Commit 12ed40d

Browse files
committed
docs/known-bugs: gen_simple JIT-overhead known-cost (Phoenix-stable)
Stable 0.70-0.76x vs vanilla CPython on bench_gen_simple across 11+ days, 6+ commit surfaces, 9 logged ABBA runs since 2026-04-16. Not a recent regression. Mechanism candidate: 87,000 short-lived generators per rep × per- generator-creation JIT-prologue/epilogue cost. Confirmed cinderjit.auto IS the cost driver (Phoenix JIT_OFF 491ms vs JIT_ON 633ms vs vanilla 476ms; testkeeper /tmp/gen_simple_match_harness.py 2026-04-27T15:14:16Z). Doc supersedes the prior dismissal framing 'pre-existing structural causal absence' (caught by pythia python#195 + medic motivated-reasoning warning). Replacement framing: 'Phoenix-stable known-cost, fix-class workstream queued post-terminal-goal'. Anti-pattern guard + threshold signal preserved (drops below 0.70x or rises above 0.76x = NEW signal, investigate). Status: PARKED behind pure-C JIT terminal-goal completion per Alex priority order. Resumption gate enumerates 5-step procedure when fix-class work opens. Per supervisor 2026-04-27T15:15:20Z disposition (a) following testkeeper investigation 15:14:16Z + theologian patch-shape APPROVE 15:17:13Z.
1 parent 86a45e5 commit 12ed40d

1 file changed

Lines changed: 134 additions & 0 deletions

File tree

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# gen_simple — JIT generator-creation overhead (Phoenix-stable known-cost)
2+
3+
**Status:** KNOWN-COST, fix-class workstream queued post-terminal-goal
4+
pure-C completion (per Alex priority order). NOT a recent regression;
5+
stable across 11+ days, 6+ sessions, multiple unrelated commit
6+
surfaces. Documented per supervisor 2026-04-27T15:15:20Z disposition
7+
following pythia #195 catch on prior "pre-existing structural causal
8+
absence" framing.
9+
10+
## Symptom
11+
12+
`Tools/benchmark_phoenix.py jit --only=gen_simple` reports Phoenix
13+
(JIT_ON, `cinderjit.auto`) at ~0.70-0.76x of vanilla CPython on the
14+
4-bench ABBA harness — i.e. JIT is ~30-37% slower than vanilla on
15+
this single benchmark, while other benchmarks in the same run land at
16+
positive speedup (fibonacci ~2.27x, nqueens ~1.59x, func_calls
17+
~1.00x).
18+
19+
## Workload shape
20+
21+
`bench_gen_simple(n_iter=8_700_000)` per-rep (`PER_BENCH_ITERS` in
22+
`Tools/benchmark_phoenix.py`):
23+
24+
```python
25+
def bench_gen_simple(n_iter):
26+
def gen(n):
27+
for i in range(n):
28+
yield i
29+
total = 0
30+
for _ in range(n_iter // 100): # 87,000 outer iterations
31+
for v in gen(100): # 87,000 generator creations
32+
total += v # 8,700,000 inner yields
33+
return total
34+
```
35+
36+
87,000 short-lived generators are created and exhausted per rep. The
37+
inner `gen()` function runs ~100 yields each before the generator
38+
object is discarded.
39+
40+
## Reproduction (testkeeper 2026-04-27T15:14:16Z, `/tmp/gen_simple_match_harness.py`)
41+
42+
3-warmup + 5-measure pattern matching `_worker_jit` in
43+
`Tools/benchmark_phoenix.py`, `n_iter=8_700_000`, `mean_ms` of 5
44+
measurements:
45+
46+
| Configuration | mean_ms |
47+
|---|---|
48+
| vanilla CPython `-S` | 476.1 |
49+
| Phoenix `JIT_ENABLE=0 PYTHONJITAUTO=0 -S` | 491.2 |
50+
| Phoenix `cinderjit.auto()` `-S` | 633.2 |
51+
52+
`cinderjit.auto()` is the cost driver. Phoenix-with-JIT-disabled is
53+
within 3% of vanilla (harmless CPython-branch overhead). Phoenix-JIT
54+
adds 142ms (29% on top of JIT_OFF) on this workload.
55+
56+
## Hypothesis (mechanism candidate, not yet confirmed)
57+
58+
The JIT-compiled outer loop pays per-generator-creation prologue +
59+
epilogue cost on each of the 87,000 generators. Vanilla CPython's
60+
interpreter has a more compact path for short-lived generator
61+
dispatch. Specific candidates worth investigating when fix-class
62+
workstream opens:
63+
64+
- JIT generator frame setup vs interpreter `_PyEvalFramePushAndInit`
65+
- Reference-count / LOAD_DEREF cost in compiled generator frames
66+
- `GEN_START` opcode emit weight relative to interpreter
67+
- Inliner heuristic: does Phoenix inline `gen(100)` into the outer
68+
loop? If not, every `gen(100)` call is full prologue.
69+
70+
## Stability evidence (NOT a recent regression)
71+
72+
| Date | Benchmark log | gen_simple ratio |
73+
|---|---|---|
74+
| 2026-04-16 | `docs/benchmarks/abba_x86_64_20260416.md` | 0.73x |
75+
| 2026-04-16 | `docs/benchmarks/abba_arm64_20260416.md` | 0.76x |
76+
| 2026-04-22 | `docs/benchmarks/abba_push45_20260422_040152.md` | 0.72x |
77+
| 2026-04-22 | `docs/benchmarks/abba_pre_push46_20260422_044949.md` | 0.72-0.74x |
78+
| 2026-04-24 | `docs/benchmarks/abba_w27c2a_2026-04-24.md` | 0.74x |
79+
| 2026-04-25 | `docs/benchmarks/abba_24bench_push59_2026-04-25.md` | 0.70x |
80+
| 2026-04-27 | `docs/benchmarks/abba_blme_b2_2026-04-27.md` | 0.74x |
81+
| 2026-04-27 | `docs/benchmarks/abba_kwnames_probe_2026-04-27.md` | 0.75x |
82+
| 2026-04-27 | `docs/benchmarks/abba_preloader_probe_2026-04-27.md` | 0.73x |
83+
84+
Ratio range 0.70-0.76x over 11+ days, 9 logged runs, 6+ commit
85+
surfaces. The variance is consistent with same-session ABBA noise
86+
floor (~3-7% per `feedback_abba_cross_session.md`); no commit in this
87+
window introduced or removed the cost.
88+
89+
## Anti-pattern guard
90+
91+
When per-commit ABBA reports `gen_simple` in this 0.70-0.76x range,
92+
gate reports MUST cite this doc + use the framing
93+
`Phoenix-stable known-cost, fix-class workstream queued
94+
post-terminal-goal` rather than the prior dismissal language
95+
`pre-existing, structural causal absence` (which sounded like
96+
no-action accounting and was caught by pythia #195
97+
2026-04-27T15:10:01Z + medic 15:15:15Z motivated-reasoning warning).
98+
99+
If `gen_simple` drops BELOW 0.70x or rises ABOVE 0.76x in a single
100+
ABBA, that IS a new signal — investigate the proximate commit, do
101+
NOT cite this doc.
102+
103+
## Resumption gate (when fix-class work opens)
104+
105+
1. Re-baseline `gen_simple` standalone (vanilla / Phoenix-JIT-off /
106+
Phoenix-JIT-on) to confirm the ~0.73x ratio still holds; if not,
107+
reframe.
108+
2. Profile JIT-compiled `bench_gen_simple` under `perf record` or
109+
`cProfile` (note cProfile distorts vanilla baseline by ~3.8x; use
110+
`perf` for clean comparison).
111+
3. Diff JIT-emitted code for `bench_gen_simple` outer loop vs
112+
interpreter dispatch — look for per-iteration generator-frame
113+
prologue weight.
114+
4. Falsify each hypothesis-class candidate above; document which is
115+
load-bearing.
116+
5. Heavy-tier instrumentation (analogous to W-PYTORCH-CM (ii)
117+
tp_alloc-watchpoint pattern) only after lighter probes converge or
118+
exhaust.
119+
120+
## Related
121+
122+
- `feedback_assume_phoenix_regression.md` — Alex 2026-04-24 directive:
123+
all bugs presumed Phoenix-introduced. Satisfied here (cost IS
124+
Phoenix's, just not recent).
125+
- `feedback_falsifier_convergent_negative.md` — convergent-negative
126+
ABBA delta needs `investigated-no-quick-fix` framing, not
127+
`pre-existing` escape. This doc IS that framing.
128+
- `feedback_no_workarounds.md` — documenting + deferring fix is NOT a
129+
workaround; only bail-out / deopt is the prohibited pattern.
130+
Deferred-fix-with-workstream-anchor is transparent accounting.
131+
- pythia #195 (D-1777300601 2026-04-27T15:10:01Z) — caught the
132+
framing drift across 4 commits.
133+
- medic 2026-04-27T15:15:15Z [MEDIC-WARNING] — caught generalist
134+
follow-up framing as motivated reasoning vs pythia.

0 commit comments

Comments
 (0)