One-shot survey forecasts with EDSL and Expected Parrot

21 distinct models, 22 configurations, the same 100 survey questions. Each model predicts the whole answer distribution in one response through production Expected Parrot. Aaru’s published predictions are scored against the same published survey shares. Lower total variation distance (TVD) is better. The main ranking contains 17 configurations with all 100 valid distributions; 5 incomplete runs are reported separately below.

GPT-6 Astra (medium) has the lowest mean one-shot error among complete runs in this sample: 8.85 TVD, versus 7.50 for Aaru. These are descriptive rankings, with one retained response per question and configuration. The intervals do not adjust for comparing many models or selecting the best result. They do not establish a stable ranking across repeated runs.

Requires Git and uv. No credential or model call is needed. The first run downloads locked dependencies.

git clone https://github.com/expectedparrot/aaru-edsl-benchmark.git
cd aaru-edsl-benchmark
uv run --frozen aaru-edsl reproduce --check
uv run --frozen pytest -q

Included: scores, summary, response archive, cost reconciliation, examples and source links, and PDF plots. Checksums protect file integrity; they are not independent attestations of inference provenance.

import json
from edsl import Agent, Model, QuestionDistribution, Survey
from aaru_edsl.protocol import SYSTEM, MODELS, context

sample = json.load(open("data/sample.json"))
questions = [
    QuestionDistribution(
        question_name=row["question_name"],
        question_text=json.dumps(context(row), ensure_ascii=False),
        question_options=[o["label"] for o in row["options"]],
        include_comment=False,
    )
    for row in sample
]
agent = Agent(traits={"role": "forecaster"}, instruction=SYSTEM,
              traits_presentation_template="")
spec = MODELS["astra"]  # choose any configuration in data/protocol.json
model = Model(spec["model"], service_name=spec["service"], **spec["parameters"])
jobs = Survey(questions).by(agent).by(model)

Every model receives identical rendered prompts: population, date, question, instructions, and answer labels. Each interview has empty cross-question memory. Native QuestionDistribution validation and an independent archive check require label-keyed probabilities summing to one. No uniform fallback, normalization, retrieval, persona simulation, or fine-tuning is used.

From the repository directory, save an EP key in the local .env file by completing the browser login:

uv run --frozen ep auth login
uv run --frozen ep check

With EDSL already installed, the command is simply ep auth login. Only an EP key is needed; no direct provider key is used.

uv run --frozen aaru-edsl infer --stage smoke --allow-paid-inference
uv run --frozen aaru-edsl status
# Repeat status until smoke jobs are saved; inspect failures and cost audits.
uv run --frozen aaru-edsl infer --stage full --allow-paid-inference
uv run --frozen aaru-edsl status
# Repeat status until all jobs are saved.
# If needed, retry only failed questions, once:
uv run --frozen aaru-edsl infer --stage retry --allow-paid-inference
uv run --frozen aaru-edsl status
# Repeat status until retry jobs are saved.
uv run --frozen aaru-edsl archive
uv run --frozen aaru-edsl reproduce

Smoke runs ask the first sampled question; full runs ask the other 99. Add --models astra astra_medium (or other keys below) to submit a subset. Full submission requires a valid smoke answer for every selected configuration. Existing receipts prevent duplicate submissions; ambiguous submissions block resubmission until reconciled. Raw Results remain under ignored runs/. Remote caching is enabled; the single bounded retry bypasses the cache and preserves identical prompts and settings. Successful answers are never rerun based on accuracy. Archiving requires all 100 questions to be accounted for by a valid forecast or a documented final failure after the bounded retry, including retained arms. It replaces the included response archive: use a new branch for a fresh experiment. --check verifies the published archive’s numerical results; new stochastic runs need not reproduce those numbers.

Selection uses random.Random(20260927).sample(sorted(eligible, key=id), 100) from 2,808 eligible categorical questions. Eligibility excludes 143 noncategorical entries and 42 strict duplicates from the 2,993-record export. The seed fixes question selection, not stochastic inference. These 100 questions span 40 operational survey families.

The expansion covers 6 EP inference services and multiple model makers. Service is the hosting route, not necessarily the maker. Models were chosen for provider and size diversity from EP’s text-working catalog with non-fallback prices, then checked for compatibility on one question. This is not an exhaustive or random sample of models. Existing Astra, Fable, and Gemini responses were retained.

All configurations request an 8,192-token output cap; provider enforcement and whether reasoning counts toward the cap can vary. Reasoning controls and temperatures differ across providers: these are not equal compute budgets. Exact parameters and rendered prompts are frozen in protocol and data/*_jobs.json / data/*_prompts.json. The expansion catalog snapshot records availability and listed prices at selection. EDSL is pinned to public commit cfef949e4ea87ab0bf6e998ab64b54a1bfcec6e4.

Reference shares, Aaru forecasts, source URLs, and other survey answers are excluded from prompts. Audience descriptions come from the benchmark and may contain substantive context; excluding numeric targets does not establish freedom from leakage.

Only questions missing a valid distribution receive one explicitly requested retry with unchanged settings. EP may also retry internally. Failed attempts are preserved in failures, with error-type diagnostics; no invalid answer is repaired into a forecast. Missing answers can reflect validation errors, rate limits, or infrastructure failures. Backend reports do not always include raw failed responses. Kimi’s full job was marked completed but omitted 11 validated distributions; the runner checks every question independently of job status.

Sensitivity check: 11 questions had valid first-submission answers from every configuration. This small subset is selected by output validity and may differ systematically from other questions. Restricting every method to it gives:

Observed-only means use each model’s valid questions and may be biased by selective failure. The bounds instead use all 100 targets, allowing each missing answer any TVD from 0 to 100; they do not impute a forecast.

These configurations were not expanded beyond the first question. Decisions used format, infrastructure, and billing checks, without scoring accuracy. One smoke failure does not establish a model’s general inability to answer the task. Smoke answers and cost records are preserved in screening.json; settings are in protocol.json.

High minus medium mean TVD is +0.20 (95% paired family-bootstrap interval [-0.30, +0.64]). High wins on 37/100 questions, medium on 43/100, with 20 ties. The interval includes zero; this sample does not establish an accuracy advantage for either effort level.

The only configured difference is reasoning_effort. Medium was added after inspecting high/Fable/Gemini results; retained responses were collected at different times. This is an exploratory paired comparison, not a randomized repeated experiment.

The separate 2,808-question medium-effort analysis used different prompts and reported GPT-6 TVD 10.11 versus Aaru 7.71. Those full-corpus numbers are not the results of this 100-question experiment.

Smoke-only excluded configurations add $0.14990 in reported EP charges.

Total retained-run accounting: $22.97 in Results and $23.06 reported by EP. These are observed costs, not fixed prices for future runs.

Results costs use recorded token-price metadata and exclude cache hits. Reported EP costs are job-accounting records, not independently verified account debits. CostReconciliationWarning flags differences exceeding $0.0002 per job, allowing two billing line items to round to $0.0001. Answers are saved before warning; a billing discrepancy never triggers a paid retry. See EDSL #2668. A separate token-accounting check flags cases where inclusive completion counts appear to be charged again as reasoning tokens. Results/EP agreement alone cannot detect that shared error. These flags are evidence for investigation, not independently verified account overcharges; provider estimates may also use different prices or discounts.

A separate Astra openai_v2 adapter diagnostic returned $0.04981 in Results versus $0.61990 reported by EP. It is excluded from the accuracy comparison and cost table; its record is preserved. Add that charge when totaling all experiment spending. We chose the catalog-listed openai route on billing compatibility before full inference.

TVD is 50 * sum(abs(predicted_share - survey_share)) for shares on a 0–1 scale, expressed in percentage points. Ten TVD points means ten percentage points of probability mass must move to match the reference. It is not a classification error rate. Questions have equal weight; reference and Aaru shares are scored as exported.

Intervals use 20,000 paired bootstrap draws over survey families (seed 20260927). Connected families share cluster, series, canonical source, or sufficiently long identical question/option text, with a documented Conference Board series join. This operational grouping is not proof of independence. Intervals exclude stochastic reruns, model selection, multiple-comparison adjustment, and human-survey sampling error.

This retrospective benchmark uses published surveys that may occur in model training data. It does not establish prospective performance on novel questions, individual-agent validity, joint distributions, or how Aaru generated its predictions. Results apply to this seeded sample.

Examples show Astra medium/high, Fable, and Gemini 3.8 Flash for legibility. They are the first six sampled questions with at most four options and ASCII wording, selected without reference to errors. All model forecasts and original source links are in the archive.

Source: Aaru’s publication and question export. A byte-preserving source snapshot and provenance are included. NOTICE.md describes third-party materials. MIT covers original code only.

GitHub Actions runs tests and offline reproduction without credentials. Public response records contain answers, usage, price metadata, cache flags, and prompts; they omit provider/account identifiers and reasoning content. EP job UUIDs identify supporting private accounting records.