Repository navigation
U1–U18: open findings from the 2026-10-14 handoff (frozen to-do list) #48
Description
Frozen 2026-10-16. Do not edit this body. Status, plans, and effects on other items live in comments only (format below). A U's current status is its latest comment.
To-do list for the 18 unresolved findings (U1–U18) in the handoff of 2026-10-14 (docs/handoff/2026-10-14/README.md §7, branch handoff/2026-10-14 @ 956ec58). Each cell has a source (collapsed sheet below). An independent Opus 5.5 check of all 72 cells found 66 OK, 5 wrong, 1 unsupported; the corrections are applied here.
Reading the table
- Guide: the spec-generation guide (
.claude/skills/generate-spec/references/spec-generation-guide.md). Every spec is generated from it. - Lab overlay, lab runtime file: lab-only add-ons to the guide (
spec-generation-guide.coding-agent-lab.md,….runtime.md). - Lab spec:
courses/loop-engineering-with-langchain-deepagents/spec.coding-agent-lab.md. Agnostic spec:…/spec.md. Both as onmain@89e4b8f. - Edited first: the file each listed choice edits first. "Or leave as is" appears only where a source offers that choice.
- Specs affected: the spec where the problem was seen, plus specs where the same subject was confirmed present. An edit reaches a spec only when that spec is regenerated. Guide edits reach every course's specs and overlay/runtime edits every lab spec; those other specs were not checked.
- Open issue: open issues on 2026-10-16 (Recording: Claude Code transcript compaction can erase the /prepare-build announcement bookend #19, Next pass: candidate slate (A) + measurement watchlist (B) #26, Guide:
live-<kind>tag for inputs that cannot be authored as fixtures (carried from #28 F2) #31, Next pass: Ledger composition variance; overlay silences from the loop-engineering confirmation regens #35, Overlay: a course-owned row whose every alternative is unavailable in the lab must not become a one-answer gate question; guide: the suite-model row's live option contradicts the offline completion gate #38), bodies and comments. Direct = names this exact problem. Related = same subject, different case. Applies = concerns the file or tool the fix would use.
Table
| # | Finding | Edited first | Specs affected | Open issue |
|---|---|---|---|---|
| U1 | The guide phrase "a maturity story" led a fresh agent-memory generation to mark conversation-only memory "narrative", dropping it as a choice. The shipped spec offers it as an option on row D7 | Guide (narrow the phrase), or leave as is | Agent-memory spec.md |
none |
| U2 | One lab generation dropped the grader-kind row, saying the alternatives were "presented, never argued or built". The orchestrator traced this to the guide's "may still enter through design-structural". The row came back in the next generation with no rule change | Guide (make the structural test mandatory), or leave as is | Lab spec (seen there). The agnostic spec has the same subject as row D2 | #35, #38 (related) |
| U3 | The lab spec's RR1 row offers the live model for the test suite. Choosing it makes the suite non-repeatable and no longer offline, which changes what "complete" means | Guide §3 rule (split the subject), which also requires rewriting an overlay sentence (ADD 1). Or hand-edit RR1 in the lab spec. Or leave as is | Rule: both specs carry the live option. In the agnostic spec it's only a second, non-blocking pass. Hand-edit: lab spec only | #38 (direct), #35 (related) |
| U4 | The overlay's rule for removing a question says "no alternative remains takeable", but means "no alternative to the Default". The last generation read it the intended way | Lab overlay (one word) | Lab spec; expected unchanged | #38 (related) |
| U5 | The web framework flips between Flask (the course's, unpinned) and FastAPI (preinstalled in the lab) across generations | Lab overlay (one sentence, either way) | Lab spec | #35 (direct) |
| U6 | The runtime file lists what the lab supplies (OpenAI and Anthropic keys) but never what it doesn't. The generator had to infer that Slack, Atlassian and LangSmith keys aren't supplied | Lab runtime file (state what isn't supplied). The generator suggested the overlay | Lab spec | #35, #38 (related) |
| U7 | No rule says which option gets the "(course default)" label when the course used both versions: Lesson 6's video uses real Slack and Jira, the shipped code a simulated tracker | Guide or lab overlay (one sentence) | Lab spec: the label flipped between generations D and E. Agnostic spec: on its next regeneration (U18) | #35 (related) |
| U8 | The lab spec removes the hosted-tracing question. The audit says a stand-in is still takeable, so removing it doesn't follow from the rule. Four generations removed it the same way | Owner's call. The handoff names the lab spec and the lab overlay as where a change would go | Lab spec. The agnostic spec still asks it, as row RR2 | #35, #38 (related) |
| U9 | A repair added a Revert button on the Learning page (rule R21, test AC50) | Lab spec (keep or strike) | Lab spec only | none |
| U10 | Repairs added rules and tests, although the audit prompt says rule meanings and test coverage belong to the generator | Audit prompt (tighten the wording, or allow additions explicitly) | None; it governs future repairs | #38 (related, applies) |
| U11 | The fake model is specified at implementation level (a nine-step turn policy, exact turn counts), which guide §2 forbids. Nearly all serious audit findings fall there | Guide §3 and §2: specify the fake model by the results it must produce | Lab spec (seen there). The agnostic spec has a fake model too (row RR1) | #35, #38 (related) |
| U12 | The agnostic spec's tests are tied to the course's example data, so a learner who swaps the project keeps the pattern but loses every test | Guide §5 / §8: tag each test as pattern-level or example-level | Agnostic spec only (in the lab, the project is fixed) | #38 (direct) |
| U13 | The lab spec asks about the loops piece by piece (only "is there a grader") and never about the learning loop. The agnostic spec asks about all four loops in one row | Lab overlay: ask about all loops the same way, or about none ("the project answer fixes the loop set", which also drops the no-grader option). Or leave as is | Lab spec | #35, #38 (related) |
| U14 | Possible new option: a fake chat page (simulated Slack) as a second takeable answer to the Lesson 6 event question, which the lab currently removes | Guide sentence (the handoff also lists the overlay), or leave as is | Lab spec: the event question would come back and need its own tests. Effect on the agnostic spec not assessed | #35, #38 (related) |
| U15 | The scope row is numbered L6; guide line 139 numbers rows within their group, so it should be L1 | Lab spec (renumber; cosmetic) | Lab spec only | none |
| U16 | The post-generation checkpoint is practice, not a step in the generate-spec skill |
generate-spec skill (one PR "if it keeps paying") |
None | none |
| U17 | No build or eval has run on this course | Nothing tracked. /prepare-build --env=coding-agent-lab creates local build and eval folders. Sonnet recommended, Haiku if cost matters most |
None | #19 (applies) |
| U18 | The agnostic spec predates the new Lesson 6 transcript. It's where the real-Jira option and the "which loops" question live | Agnostic spec (clean-room regeneration, checkpoint, stop) | Agnostic spec | #35 (related) |
#31 relates to none of these.
Comment format (append-only)
**U4, U8** · status: done | in progress | deferred | dropped
Change: PR #NN — <file · section, one line>
Effect on other U: U8 — <what changed>, <source>
One comment per event. Plans (clustering, ordering, deprioritizing) are comments too.
Source sheet — one source per cell
Source keys.
H§n—docs/handoff/2026-10-14/README.md§n @956ec58.G:n,O:n,RT:n,AP:n— line n ofspec-generation-guide.md,spec-generation-guide.coding-agent-lab.md,….runtime.md,audit-prompt.mdunder.claude/skills/generate-spec/references/@89e4b8f.LAB:n,AGN:n— line n of the lab / agnostic loop-engineering spec @89e4b8f.AM:n— line n ofcourses/agent-memory-building-memory-aware-agents/spec.md@89e4b8f.#n §x/#n cK— issue n, body section x / K-th comment, as of 2026-10-16.AUD-E1 #16,FIX-E1 #25—docs/handoff/2026-10-14/loop-engineering/audit-E-round1.mdfinding 16,fixlog-E-audit1.mditem 25.S:Ln— previous Claude Code session transcript (0345ced7…jsonl), 0-based JSONL line n. Local to the owner's machine; not in the repo.
U1. Finding: H§6 table row 1; G:166; S:L6788 (generator: "I marked it narrative"); AM:76. Edited first: H§7 U1 ("narrow … or accept"). Specs: H§6; AM:76; H§2 (loop-engineering rows "do not move on paper"). Issue: no hit in any open issue.
U2. Finding: H§4.1; H§4.2 row 4 (generator's reason "presented, never argued or built"; sentence behind it, the word "may"); G:157; S:L6223, S:L6449. Edited first: H§7 U2 ("leave … or make the structural test mandatory"). Specs: H§4.1; AGN:73 (D2 "Grader ownership": LLM grader vs deterministic checker). Issue: #35 §1 (row set varies between regenerations); #38 §1 (human-grader option on the grader-kind row), #38 c2 (the deterministic-checker option retires ACs), #38 c1 item d (owner to decide this course's row set).
U3. Finding: #38 §2. Edited first: H§7 U3 ("adopt a rule, hand-edit, or leave"); #38 §2, c1 item c; #38 c3 note (1): the split must also rewrite the overlay's ADD 1 sentence "The acceptance-suite model row is course-owned and stays" (O:192). The ADD 3 line #38 c3 quotes has since changed (now O:240, "No course-owned row left the Ledger except through the Environment Resolutions table"); check it still fits. Specs: LAB:76; AGN:74 (live option is a second, non-gating pass; the offline pass stays the gate). Issue: #38 §2 (direct); #35 §2 bullet "the served app's default model mode when RR1 covers only the suite" (related).
U4. Finding: H§7 U4; O:110; S:L6383 (generator: "I read 'alternative' as 'alternative to the Default'"); S:L6421. Edited first: H§7 U4 ("one word"). Specs: S:L6421 ("It read it the intended way"); S:L6449 ("The generator guessed right"). Issue: #38 §1 proposed the removal rule this wording belongs to (merged as #41, H§2).
U5. Finding: H§7 U5; S:L5366; S:L6449; RT:39. Edited first: H§7 U5 ("one sentence either way"). Specs: S:L6449. Issue: #35 §2 bullet "Whether a preinstalled web framework is preferred over the course's unpinned one" (direct).
U6. Finding: H§7 U6; RT:61; O:103; S:L6170. Edited first: H§7 U6 ("runtime file"); S:L6170 (generator: "The overlay should state the negative explicitly"). Specs: S:L6170. Issue: #35 §2 bullet on whether "never ask for a key" covers LangSmith; #38 §1 (the Atlassian token the lab does not supply).
U7. Finding: H§7 U7; S:L6170 item (3) ("unanticipated by either file"). Edited first: H§7 U7 ("overlay/guide", "one sentence"). Specs: generation D labelled the video's real integrations "(course default)" (S:L6170); shipped E labels the shipped simulated code (LAB:132, LAB:653); S:L6170 ("a standalone generation of this course would have to pick one"). Issue: #35 c4 bullet "keyed-tool Options assume the course's realization is the keyed one"; #35 §3 bullet 1 (a "(course default)" gap on the scope row).
U8. Finding: LAB:91; AUD-E1 #16 ("the stand-in and omission are takeable"); S:L6690 ("four generations"). Edited first: H§7 U8 (where: "spec + overlay"; decision: "owner's call"); AUD-E1 #16 ("owner decides"). Specs: LAB:91; AGN:75. Issue: #35 §1 (hosted tracing "row or Not Included"); #38 §1 (the removal rule), c1 item d.
U9. Finding: H§5; FIX-E1 #25; S:L6690. Edited first: H§7 U9 ("accept or strike"). Specs: H§7 U9. Issue: no hit.
U10. Finding: H§3 caveat 1; AP:63–66 ("Ledger rows, defaults, rule meanings and AC coverage are the generator's"); S:L6751. Edited first: H§7 U10. Specs: H§7 U10. Issue: #38 c2 (repairs added per-option AC lists; related); #38 c1 item a (proposes an audit-prompt.md edit; applies).
U11. Finding: H§3 caveat 2; H§5; S:L5696. Edited first: H§7 U11; S:L5696 ("specified by the outcomes it must produce"). Specs: S:L5696; AGN:74. Issue: #35 c4 closing observation ("the guide can still shrink the first-round count"); #38 c2 (the scripted review script makes ACs fail when an option changes its input; cf. S:L5696 "exact counts break the moment anyone touches the instructions or rubric").
U12. Finding: H§7 U12; H§5; S:L5696. Edited first: H§7 U12; S:L5696 ("criteria tagged as pattern-level … and example-level"). Specs: S:L5696 ("Project swapping is not a lab goal"; "it is the agnostic spec's problem to solve"). Issue: #38 c4 (choosing another project re-authors the fixture corpus and the criteria; direct).
U13. Finding: H§7 U13; H§5; S:L5966; AGN:72 (D1 "Loop composition"). Edited first: H§7 U13 ("overlay / project answer — ask all loops the same way, or none"); S:L5966 ("Accept it" / "One overlay sentence: … the project answer fixes the loop set … would also remove D2's plain-loop option"); S:L5696 item 4 ("Reconsider loop composition as a lab row"). Specs: S:L5966. Issue: #35 §1 ("what a triggered run executes" varies); #38 c1 item d (owner to decide the row set).
U14. Finding: H§5; S:L5584; S:L6002. Edited first: S:L6002 ("Option 4 needs a guide sentence"); H§7 U14 ("overlay/guide keyed-tool stand-in", "owner decision"); H§5 ("not started"). Specs: S:L5584; S:L6002 ("It also changes what gets built, since the chat surface needs its own contracts and criteria"). Issue: #35 §1 (event-intake surface varies), #35 c4 keyed-tool bullet; #38 §1 (system-of-record row whose only alternative needed an Atlassian token).
U15. Finding: H§4.1; G:139 ("numbered within their Category group"); LAB:72; S:L6421. Edited first: H§7 U15 ("cosmetic"). Specs: H§7 U15. Issue: no hit.
U16. Finding: H§3; H§7 U16; S:L6449 ("practice, not yet in the skill"). Edited first: H§7 U16. Specs: H§7 U16. Issue: no link (#38 c1 item a proposes a different review step).
U17. Finding: H§0; H§7 U17; S:L5696. Edited first: H§7 U17; CLAUDE.md (per-run layout; builds/, evals/ gitignored); S:L6278 (Sonnet / Haiku). Specs: CLAUDE.md. Issue: #19 body and comments (compaction erased the extractor's opening marker on run-08 and run-09; applies).
U18. Finding: H§0; H§7 U18; AGN:9 ("guide @ 11b3673"); S:L6278 item 5. Edited first: H§7 U18; H§8 option 4. Specs: H§7 U18. Issue: #35 c2 (an agnostic regeneration of this course demoted the course's web app; related).
Activity
Carr1005 commentedon Oct 18, 2026
U1, U2, U13 · status: done
Change: PR #52 — guide §5.5 litmus rewritten as an ordered test (failure → working, incl. stages of the course's progression → otherwise; structural subjects stay rows with never-shown options marked in their cell; one subject, one row); PR #55 — definitions of row / bake-in / narrative aligned with it.
Effect on other U:
unavailablein the others; still open.mainstill says L6; unchanged.spec.mdonmainis unchanged; still open.Runs: clean room, Opus 5.5; outputs local under
experiments/*/guide-stepping-stones/(untracked).Carr1005 commentedon Oct 18, 2026
U16, U17 · status: context (finding, no change)
Change: none.
handoff/2026-10-14, §1 and §3) and in the orchestrator's local memory.generate-spec/SKILL.mdonmainhas no clean-room isolation and no checkpoint, runs audit + repair automatically (step 4b), and does not fix the generation model. Owner rule since 2026-10-18: all runs and experiments on Opus 5.5 (promote-to-guide's "Opus-class" now agrees)./generate-specfollows a different process from the specs produced here; comparing spec quality across contributors would mix process with guide.Effect on other U: U16 widens from "add the checkpoint to the skill" to "write the process into the skill, marked as current practice under evaluation".
Carr1005 commentedon Oct 18, 2026
New finding (not in U1–U18): rows that stop mattering · status: in progress (Part 2)
Change: none yet.
91a3718; the gate was made strict after agents buried rows,e934873), and the guide has never required an option to name what it retires (Overlay: a course-owned row whose every alternative is unavailable in the lab must not become a one-answer gate question; guide: the suite-model row's live option contradicts the offline completion gate #38 comment 2, item 4).Plan (Part 2): (1) every non-default option states the rules and tests it retires or alters and the rows it makes moot; (2) no open-ended options; (3) consistent option labels; (4) "applies only when" conditions plus a checker check; (5) decide later whether the gate may skip declared-dependent rows. Regenerate and adopt the agnostic specs after (1)–(4) — owner decision 2026-10-19, which replaces D4.
Carr1005 commentedon Oct 19, 2026
U7 · status: done
Change: PR #57 — gate labels. When the course used two realizations of one row, the shipped-code one carries "(course default)" and the shown-only one (video demo, slide) says so in its own cell.
Also in #57 (not U items): "(Recommended)" now marks the row's Default (the generator's most-efficient pick, what the recommended baseline builds), written in the Default's cell; "(course default)" is written on the course's option in every row where the course made a choice; model choice moves to a confirmed step-5 checklist line. Checked by 3 regenerations and 3 gate probes.
Effect on other U:
Carr1005 commentedon Oct 19, 2026
Follow-up on the keyed-tool note (from the U7 comment) · status: no guide change
Change: none. An independent Opus 5.5 review (
experiments/review-rule2-keyednote.md, local) found the note is not the cause.774a12a, lab) gave its reason for leaving out the manual-only and no-learning-loop options as "the sides are not exclusive" (its spec, T4 and T13). It never cited the keyed-tool note.2095adfoffered both options. G3 is the only one that did not.Also dropped: the planned Part 2 rule (2), "no open-ended options", as a standalone rule. 0 of 25 recent regenerations had an "any subset / any combination" option. It becomes one clause in rule (4): "each Options entry is one concrete build, including a never-built one (§5.5 step iii); a mix goes to free text."
Effect on other U: none.
Carr1005 commentedon Oct 19, 2026
New finding (not in U1–U18): rows that stop mattering · status: in progress (Part 5 next)
Change: PR #58 — Part 2 rules 3 + 4. Each option has an id (
D1a); a row that stops mattering under an earlier option says "Applies only when<row>is<id>"; §7 "Tests by option" table lists exact retired/changed rules and tests and moot rows; step-5 checklist copies the chosen option's line; eval-spec-vs-build readsresolved-decisions.mdand lists retired tests as "Not applicable". Checked by 3 regenerations and 1 gate probe (Opus 5.5).Effect on other U: none.
Carr1005 commentedon Oct 19, 2026
New finding (not in U1–U18): rows that stop mattering · status: Part 5 in review
Change: PR for Part 5 — the gate skips a row only when its own "Applies only when" condition is false; a skip resolves only that row; checklist mark
not applicable (…; <row> = <answer>). 4 gate probes and 4 regenerations (Opus 5.5).Follow-up found (not a U item): when an option removes a model role (e.g. no grader), its "Tests by option" line does not say what happens to that role's "model setting required" rule and test. Seen in I2 (R23/AC34) and twice in K2 probes (R24/AC38). Not fixed.
Effect on other U: none.
Carr1005 commentedon Oct 19, 2026
N1–N5 (carried from #38) · status: open
Change: none. #38 is closed in favour of this list; an item's status is its latest comment here. Checked item by item against
main@bcc9859(Opus 5.5 verifier, line quotes spot-checked).unavailableclass should read "cannot be taken, or breaks the completion gate, in this build mode"read_file) — a CTX-D factResolved in #38 (not carried): item 1, a row whose every option is untakeable in the lab is not asked (PR #41); item 4, each non-Default option states the tests it retires or changes (PR #58).
Effect on other U:
Carr1005 commentedon Oct 19, 2026
N6–N20 (carried from #35) · status: open
Change: none. #35 is closed in favour of this list. Checked item by item against
main@bcc9859(Opus 5.5 verifier).rowin one mode is notnarrativein the other.envloaded with override clobbers the lab's keys — overlay should state itAGENTS.mdname collision explicitlyResolved in #35 (not carried): the model-roles line (step-5 model line, PR #57); the scope row's "(course default)" target (PR #57,
no course choice); the blockquote pin (PR #36); the §4 residuals (nothing owed).Closed as moot: "grep added guide lines for course terms in every promotion pass" — it belongs to
/promote-to-guide, no longer maintained (#53).Effect on other U:
Carr1005 commentedon Oct 19, 2026
N21–N29 (carried from #26) · status: open
Change: none. #26 is closed in favour of this list. Checked item by item against
main@bcc9859(Opus 5.5 verifier).resolved-decisions.md?live (net)criteriaResolved in #26 (not carried): A2, A3, A4 (PR #36; noted in #26's 2026-10-12 comment).
Effect on other U: U17 — its build can answer N27.
Carr1005 commentedon Oct 19, 2026
New finding (not in U1–U18): rows that stop mattering · status: done
Change: PR #58 (option ids, "Applies only when", §7 "Tests by option") and PR #59 (the gate skips a row only when its own written condition is false; a skip resolves only that row; checklist mark
not applicable (…; <row> = <answer>)). Checked by 7 regenerations and 5 gate probes (Opus 5.5).Next: regenerate and adopt the agnostic specs (owner decision 2026-10-19, replaces D4).
Effect on other U: U18 — the loop-engineering agnostic regeneration is that step.
Carr1005 commentedon Oct 19, 2026
U10 · status: decided (owner, 2026-10-19), wording change pending
Change: none yet. Decision (a): a repair must not add a rule (§4
R<n>) or a test (§5AC<n>), and must not add app behavior. A repair may fix an existing rule or test so it agrees with the rest of the spec (counts, fixtures, citations). A missing rule or test found by an audit becomes a guide finding here, not an edit.Enforced from the next adoption by a script that compares the spec before and after each repair: Ledger rows, options, defaults, conditions, and the sets of
R/ACids must be unchanged.Effect on other U:
Carr1005 commentedon Oct 20, 2026
N30–N36 (open points of the agnostic specs adopted on guide
5caba53) · status: openChange: none. Found by the adoption audits (Opus 5.5) and left unrepaired, because each needs a decision: a Ledger cell, a disposition, or a new test. Listed in PR #63 (agent-memory) and PR #65 (loop-engineering).
narrativefor a stage shown working; T9 a lever whose "manual" side removes the automatic offload; T24/T25 name no tie-breakerEffect on other U: U8 — loop-engineering's RR2 hosted-tracing row offers no stand-in (the keyed-tool rule wants real / stand-in / omission); found again by the audits.
Merge pull request #66 from https-deeplearning-ai/lab-u6-no-other-keys
Merge pull request #67 from https-deeplearning-ai/lab-u3-suite-row-re…
Merge pull request #68 from https-deeplearning-ai/lab-u3-suite-row-re…