Repository navigation
Overlay: a course-owned row whose every alternative is unavailable in the lab must not become a one-answer gate question; guide: the suite-model row's live option contradicts the offline completion gate #38
Description
Found on owner review of PR #37 (the loop-engineering lab spec regenerated on guide 11b3673 / overlay f9cc86d). Fixed by hand in that spec; nothing here is applied to the guide or overlay.
1. Overlay — "mark, never delete" produces one-answer questions
OVERRIDE 2: "Where the environment makes one of its Options impractical, mark that Option in its own cell with the reason — never delete it." Applied to the regenerated spec it produced:
- D1 System of record: default = the course's in-app simulated Jira + Confluence; the only alternative = real Atlassian via REST + webhook, which needs an account and token the lab does not supply and the spec may never ask for. Customize would ask a question whose second option the learner cannot take.
- D3 Grader type, option (c) a human grader: cannot pass the offline suite that is the completion gate.
- Not Included items 1 and 7 pointed back at those rows ("owned-by-row D1/D3"), so the exclusion list and the Ledger referred to each other for a choice that existed in neither.
The rule's intent (never silently drop a course-owned decision) is right; its effect in this case is wrong. Proposed: when every alternative on a course-owned row is unavailable in this build mode, the row is not presented as a question: its default and invariant are pinned in §2 (or the owning rule) with the alternative recorded in Not Included as unavailable and the seam the course points at stated, so a standalone takeaway can restore it. An option that is unavailable while others remain is marked unavailable in its cell, and Not Included cross-references it as unavailable, never as owned-by-row. Hand-applied in PR #37; the gate there now asks four rows instead of five.
2. Guide §3 — the scripted-model row offers a live option that contradicts the completion gate
The new scripted-model rule says: "a realization row ('which model the acceptance suite runs against': scripted stand-in as Default; the course's live model in Options tagged '(course default)')". The row's subject is the suite, and the spec's completion gate is "the offline suite passes". Choosing the live option for the suite makes it non-deterministic and no longer offline, so the option changes the meaning of "complete". The regenerated spec's RR1 carried exactly that option; the hand edit annotated it. Proposed: split the subject — the suite always runs on the scripted stand-in (a rule, not a row); the course's live model is the realization for live use of the finished app and the live-tagged criteria, which is where its "(course default)" belongs. Evidence: spec.coding-agent-lab.md RR1 on course/loop-engineering-regen; the agnostic audits of PR #34 flagged the same tension (RR1 option (b) vs its Invariant) and it was reworded rather than resolved.
3. Review method worth keeping
The check that found these was one question per row, by eye: can the learner actually take each option in this build mode, and does the completion gate still mean the same thing if they do? Neither the checker nor the scorers ask it. Candidate §14 line for lab mode (ADD 3): every Option on every presented row is takeable in this environment without credentials the spec forbids asking for, and no Option changes what "complete" means.
Activity
Carr1005 commentedon Oct 12, 2026
Root cause — why a fresh generation on the confirmed guide still asked questions the learner cannot answer
Owner asked for this after reviewing PR #37/#39. Written to be read before the next pass is planned.
The regime measures only what it pins, and it pins after a human notices. Every property in every matrix (
guide-pins-v2,guide-sources-and-tiebreaks,guide-loop-engineering-residuals) encodes a previous failure: citations, declared fixtures, replay, reciprocity, row ids, blockquote shape. Eleven regens passed them all. "Can the learner take each option in this environment?" and "does an option change what complete means?" were never properties, so they were never checked, so they survived every round. The matrices name this ("protects exactly what it pins") and the response each time was more pins of the same kind.Correct rules compose into a wrong outcome; nothing tests compositions. Base: every Ledger row is one gate question. Overlay OVERRIDE 2: mark an impractical option, never delete it (added so the generator would stop silently dropping course-owned rows). Runtime: the lab supplies no Atlassian credentials; the spec never asks for keys. Together: a question with one takeable answer (D1). Each rule is right alone. The guide is ~14,400 words of such rules and has no step that asks what they do together in a given environment.
Audits are consistency audits. §14 and
references/audit-prompt.mdcheck the spec against itself and the guide's rules. Three rounds (17 → 7 → 3 hard) went to cross-AC state and fixture coverage. A one-answer question is not a rule violation, so no auditor reported it, though it was visible in round 1 to anyone reading as a learner.A promoted rule encoded a mistake, and its property validated the mistake. §3's scripted-model rule copied the keyed-tool shape (real / stand-in / omission) onto the suite row; for the suite, the live option contradicts "complete when the offline suite passes". P-F5 checked for exactly that shape, so nine regens passed by reproducing the contradiction.
Composition was recorded, never decided. The surfacing bar admits borderline rows; D1 entered through the structural litmus (store boxes change owner), true in the abstract, irrelevant where no alternative exists. Nobody has decided, subject by subject, which rows this course should ask; each regen re-derives from scratch (Next pass: Ledger composition variance; overlay silences from the loop-engineering confirmation regens #35).
Orchestration. Success was declared from checker lines and scorer quotes rather than from reading the spec as a learner. A two-question-per-row eye review found in minutes what ~4M tokens of regens and scorers did not, because it asked a different question. Rules added at the end of long sessions worsened the composition problem.
What the next pass should be (not more rules of the existing kind)
a. A learner-facing review step, in
audit-prompt.mdand as a §14 line: for every presented row, every option is takeable in this environment without credentials the spec forbids asking for, and no option changes what "complete" means. Two questions, by eye, per row.b. Overlay: when the environment removes every alternative on a course-owned row, resolve the row (pin in §2 / the owning rule; alternative to Not Included as
unavailablewith the seam stated) instead of asking it. An unavailable option beside available ones is taggedunavailable, neverowned-by-row.c. Guide §3: split the suite-model subject — the suite always runs on the scripted stand-in (a rule); the course's live model is the realization for live use and the
live-tagged criteria.d. Owner decision on the row set for this course, recorded in the matrix, so regens are scored against a decided composition rather than re-deriving it (closes the loop on #35's variance for this course).
e. The prose-economy pass (#26 A5) before any new rule. Rule count is now part of the problem.
Carr1005 commentedon Oct 12, 2026
Third item, from the final audit of the lab spec (post-#39):
4. Every non-default option must state the ACs it retires or alters
D2's "skills" and "fixed system prompt" options and D3's "deterministic checker" option were valid under their rows' Invariants but silent about consequences: choosing the system prompt retires the apply-endpoint ACs; choosing skills changes what the scripted review script keys on (it reads the always-loaded text), so one AC cannot pass as written; choosing the deterministic checker bypasses the scripted grader where the error/unevaluable/never-satisfied markers live, so five ACs cannot pass. A learner who customizes gets a spec whose completion gate silently moves. Two earlier lab regens (LC, LD) added per-option AC lists on their own during repair; it is a rule, not a courtesy. Candidate §5.5 sentence: each Options entry other than the Default states the acceptance criteria it retires or alters and how; an option that retires the completion gate's criteria says so. §14 line to match. Hand-applied in the lab spec on
course/loop-engineering-agnostic-regen.Also applied there by owner direction (items 1 and 2 above): D3's human-grader option leaves the Options cell (narrative + Not Included
unavailable); RR1 leaves the Ledger — the suite always runs scripted (§2 row), the course's live models are the live-use realization (R21). The gate now asks L1, D2, D3.Carr1005 commentedon Oct 12, 2026
Verification audit after the repairs (lab spec, branch
course/loop-engineering-agnostic-regen): 1 hard, 8 soft, all small and applied by hand. Two notes for the overlay/guide PR this issue will produce: (1) with the suite-model row pinned in §2 ("the suite always runs on the scripted stand-in"), the overlay's ADD 1 sentence "The acceptance-suite model row is course-owned and stays" and the ADD 3 line "No course-owned row was removed by this mode" now contradict the spec — the split proposed in item 2 above must also rewrite those two sentences; (2) the learner-facing check caught one more untakeable restoration (per-tool human approval blocks the synchronous offline runs the completion gate depends on) — theunavailableclass needs the wording "cannot be taken, or breaks the completion gate, in this build mode", not only "outside this build mode".Carr1005 commentedon Oct 12, 2026
From the agnostic spec's verification audit (PR #40): item 4 (every non-default option states its AC effect) applies to the learner rows too, in a different form — choosing another project, data source, rubric or provider re-authors the fixture corpus and the criteria that encode it, while the row's Invariant and the rules survive; the cell must say so, and keyed options carry their setup weight inline even on learner rows. Seven of the eight hard items in that audit were this. Also: an AC that compares a tool observation byte-for-byte with a file must account for the harness's rendering (the pinned harness's
read_filereturns line-numbered content) — a CTX-D fact the guide could require stating.Overlay: an option the learner cannot take in the lab is not presente…
Merge pull request #41 from https-deeplearning-ai/guide-untakeable-op…
Loop-engineering lab spec: clean-room regeneration on guide 11b3673 /…
Loop-engineering lab spec: clean-room regeneration on guide 4e1b71e /…
Carr1005 commentedon Oct 19, 2026
Closed in favour of one list. Resolved here: item 1 (PR #41) and item 4 (PR #58). Open items moved to #48 as N1–N5 (#48 (comment)); item 2 is U3.