crewAI course: problems found in specs and builds (v2.6–v2.8) #97

Problems found in the crewAI course's specs and builds (2026-10-27/28): specs #86 (v2.6), #90 (v2.7), #94 and #96 (v2.8), 8 + 8 + 6 + 3 test generations, and five clean-room builds (v2.6 pinned, v2.7 probe, v2.7, v2.8 recommended, v2.8 own job). Each item says whether it is a guide question, a spec-test gap, or a framework fact.

1. Tests pin behaviour the current framework does not have (guide)

The spec turns things the course left to the framework into exact test contracts, invented "so behaviour is testable and the same across frameworks" (#96, D9): a 15-step limit that forces an answer, exact error strings (Tool error: no coworker with role <role>), tools with "one text argument", results passed on "each labelled with the task name", a run-wide cache. Current crewAI (1.15) does these differently, so every builder replaced crewAI's executor loop, delegation, cache and context passing, and subclassed private methods (Crew._get_context, Task._execute_core). The "use current releases" rule (4.7) and "names that go stale" cover names and versions, not behaviour. Effects: the build is mostly custom code around crewAI; it can break on the next crewAI release; two builds differ. Direction: test what the pattern guarantees and leave framework-controlled details to the current framework, recorded by the build agent (like 4.6 does for models).

2. "Don't make up information" is not tested (spec tests)

Every real run adds unsupported résumé claims (e.g. "Designed and implemented backend API functionality", "Relational data modeling"). The spec checks only the instruction text (T16); builders added their own live checks, which cover employers and named skills only. The spec's own scripted résumé lists a skill taken from the code profile, so "facts" are undefined. In the pinned v2.6 build the résumé invented Docker and dropped the employers (fixed by current releases).

3. Parts from other lessons become questions (guide 3.3)

Most generations ask "pause for your feedback" and "JSON output" (Lesson 12), which are not on the card's "Offered from other lessons" (memory, reviewer only); #86 kept them as notes. Cause: 3.3 also admits a part that "changes one of the stages". Some runs add more: framework choice, notifications, "stop the run when a tool fails" (the course says not to), a hand-made manager. Question counts range 9–17.

4. Tools under 4.4 (guide)

A course tool that only needs the key the build already has (résumé search): 9 of 10 kept it, 1 left it out under 4.4's tool bullet ("a heavy tool ... otherwise it is left out"). Row 35's "a key the build does not already need" covers heavy parts but excludes tools. #94 removed the choice entirely (always on).

5. The own-job path (C1b) is résumé-shaped in some generations (guide/spec)

• crewAI course: agnostic spec on guide v2.8 #94: the team format allowed only 4 tools, two of them résumé-only.
• Candidate 3: asks "What format is your résumé (or other local document)?" and "search your résumé" of a learner with no résumé (conditioned on how documents arrive, not on the job).
• crewAI course: agnostic spec on guide v2.8 (owner-picked, built twice) #96: résumé questions skip cleanly, custom tools go into a tool contract; but its web-search option says agents "read the résumé", and "written to contract D" has no clear target for the build agent.
• Builder checks: builders reword résumé examples for an own job well (2 of 2), but cannot remove a question that should not be asked.
• C1b gives no inputs for the sender or offer, so an outreach email ends up with placeholders.
• An own-job build keeps the résumé crew's reference tests (T1–T21), because the spec both says "leave out what is not chosen" and "C1b adds the same tests".

6. Offline tests vs the real model (spec)

• The scripted stand-in uses the framework's text tool format; the real model uses native tool calls, so some failures appear only in live runs.
• "Label every stand-in output" conflicts with T10 (result files must equal the scripted text exactly).
• The sample addresses are *.example.test, so the spec's "real run on the sample" cannot fetch them; builders add a sample-pages flag.

7. Learner-facing wording

• "code-profile address/page" for the GitHub profile link (5 of 9 v2.7 specs; crewAI course: agnostic spec on guide v2.8 (owner-picked, built twice) #96 still says "code profile" in places). See Spec opening note: what learners read first — open concerns after v2.8 #92.
• Opening note: see Spec opening note: what learners read first — open concerns after v2.8 #92.

8. Framework side effects the spec never mentions (framework facts)

• crewAI writes to the home folder (~/.config/crewai, ~/Library/Application Support/crewai/credentials/secret.key), keeps a SQLite file of task outputs (including the résumé text), checks PyPI for updates, and sends telemetry unless switched off.
• crewAI stops a run after 5 identical tool calls, and in older versions the agent loop expected a text reply format current models skip (the pinned v2.6 build's agents never read the résumé).
• Delegation default changed from on (course era) to off; specs set it explicitly.

9. Not yet checked

• Real job boards (login walls, JavaScript-heavy pages) with the plain page fetcher.
• How a learner swaps in their own résumé after the recommended build (a setting/path).
• A build by an agent other than Opus.

Related: #87, #91, #92, #96.