Publish llms.txt and Markdown copies of the docs for coding agents #74376

Coding agents (Claude Code, Cursor, Copilot, OpenCode) read our docs to write Dags, and today they get only HTML. A provider guide page is about 150 KB of HTML for about 20 KB of content, and fetch tools often summarise it down to a paragraph, losing the code samples. Projects such as Temporal, Pydantic AI, Ray and Camel publish an llms.txt index and a Markdown copy of each page. https://airflow.apache.org/llms.txt and every /docs/.../llms.txt return 404.

This issue covers doing the same for Airflow's docs, starting with the Common AI provider (apache-airflow-providers-common-ai), in three parts. A working proof of concept exists for the first two, and the third was tested on the staging site.

1. Docs build (this repo)

Proof of concept: astronomer/airflow branch llms-txt-poc, one commit.

• sphinx-llm runs a second Sphinx build with a Markdown builder and writes page.html.md next to every page.html. It also adds <link rel="alternate" type="text/markdown" href="page.html.md"> to each HTML page, which Codex CLI follows.
• A new extension, devel-common/src/sphinx_exts/airflow_llms_txt.py, enabled for Common AI only through PROVIDER_PACKAGES_WITH_LLMS_TXT in devel-common/src/docs/provider_conf.py:
  • Fixes the Markdown for Airflow docs: mermaid diagrams, code-block captions, admonitions, compact tables, same-page links, license-header comments, and |version| in URLs.
  • Links each exampleinclude excerpt to the full example file on GitHub, because the excerpts leave out imports.
  • Writes llms.txt grouped by the docs navigation, with absolute versioned URLs and a description per page.
  • Writes llms-full.txt: the same pages in one file.
• Build time for Common AI is unchanged at about 45 s locally, and nothing changes for other packages.

Before merging:

• Cut the extension down. About 90 of its ~360 lines use private internals of sphinx-markdown-builder (tables, admonitions, same-page links) or work around a sphinx-llm bug (in sequential mode the Markdown build reuses the HTML build's doctree cache). Upstream those to sphinx-markdown-builder and sphinx-llm, and keep only the Airflow-specific parts that use public Sphinx APIs here.
• Write a .html.md for each redirect stub generated from docs/redirects.txt (seven in Common AI), saying where the page moved.
• Fail the build if a published .html page has no .html.md, excluding _modules/, genindex, search and py-modindex. The serving rule in part 3 relies on every other page having one.
• Add unit tests for the extension.

2. Publishing (dev/breeze)

• Stop stable/ going backwards. stable/ is a full copy (aws s3 sync --delete) of whichever version was published last. Republishing or patching an older version with the default skip-write-to-stable-folder: false replaces stable/ with an older build, and docs_publisher.py writes stable.txt without comparing versions. Make the publisher refuse to overwrite stable/ with a version lower than stable.txt.
• Add a post-publish check that stable/index.html.md returns 200 with a text/markdown content type. S3 stores the content type the AWS CLI guesses from the file extension, and older Python mimetypes tables have no entry for .md, which leaves binary/octet-stream.

3. Serving: Accept: text/markdown on airflow.apache.org (apache/airflow-site)

Claude Code sends Accept: text/markdown, text/html, */* on every fetch; Cursor, Copilot and OpenCode send similar headers (support table). If airflow.apache.org answers those requests with the Markdown copy, agents get it from the URLs they already use, with no need to find llms.txt. Read the Docs and Cloudflare-hosted sites already do this. No ASF project does it on purpose yet.

/docs is proxied to the docs CloudFront in landing-pages/site/static/.htaccess, so this is a .htaccess change placed before that proxy rule:

# Phase 1: the Common AI provider's stable docs only. Deploy this only after
# https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/index.html.md returns 200.
RewriteCond %{HTTP:Accept} text/markdown [NC]
RewriteCond %{HTTP:Accept} !text/markdown\s*;\s*q=0(\.0*)?\s*(,|$) [NC]
# Match the raw request line, so the rewrite and the Vary header below decide on the same string.
RewriteCond %{THE_REQUEST} " /docs/apache-airflow-providers-common-ai/stable/"
RewriteCond %{THE_REQUEST} !"/(_modules/|genindex\.html|search\.html|py-modindex\.html)"
RewriteRule ^(docs/apache-airflow-providers-common-ai/stable/.+\.html)$ https://d7fnmbhf26p21.cloudfront.net/$1.md [P,L]
RewriteCond %{HTTP:Accept} text/markdown [NC]
RewriteCond %{HTTP:Accept} !text/markdown\s*;\s*q=0(\.0*)?\s*(,|$) [NC]
RewriteCond %{THE_REQUEST} " /docs/apache-airflow-providers-common-ai/(stable/(?!_modules/)(.+/)?)?[ ?]"
RewriteRule ^docs/apache-airflow-providers-common-ai/(?:(stable/(?:.+/)?))?$ https://d7fnmbhf26p21.cloudfront.net/docs/apache-airflow-providers-common-ai/stable/$1index.html.md [P,L]
<IfModule mod_headers.c>
    Header merge Vary Accept "expr=%{THE_REQUEST} =~ m# /docs/apache-airflow-providers-common-ai/#"
    Header set X-Robots-Tag "noindex" "expr=%{THE_REQUEST} =~ m#\.html\.md[ ?]#"
    Header edit Content-Type "^(text/(markdown|plain))$" "$1; charset=utf-8" "expr=%{THE_REQUEST} =~ m# /docs/#"
</IfModule>

What was tested on airflow.staged.apache.org, with each page's _sources/*.rst.txt standing in for the not-yet-published .html.md:

• Agents: a request with Accept: text/markdown got the text copy with charset=utf-8.
• Browsers: the same URL returned HTML.
• Vary: every response for the package carried Vary: Accept-Encoding,Accept.
• Other packages: unchanged.
• End to end: Claude Code's fetch tool received the text copy from an ordinary .html URL.

ASF's Fastly cache honours Vary: Accept; the same check on apisix.apache.org returned separate cache hits for Markdown and HTML requests.

That staging test used an earlier version of the rule. The rule above adds fixes for problems it had:

• Crafted URLs bypassed Vary. …/common-%61i/… and //docs/… matched the rewrite but not the Vary condition, so Fastly could have cached Markdown for browsers.
• The directory rule lacked the _modules/ exclusion.
• text/markdown;q=0 still matched, even though a client sending it is refusing Markdown.

Re-test the rule above on staging before production.

Rollout constraints:

• A missing .md returns S3's 404, and Fastly caches it for up to an hour. Publishing invalidates CloudFront, not Fastly. So turn the rule on for a package only once its stable/ has .md copies, and keep an explicit package list: removed providers (sqoop, plexus, qubole) still serve stable/ and will never get .md files.
• Fastly doesn't normalise Accept, so Vary: Accept stores one cache entry per distinct Accept string, up to 50 per URL. That's fine for one package. Talk to ASF Infra before widening the rule to all of /docs.
• Rollback is reverting the .htaccess commit; Fastly may take up to an hour to drop cached copies.

Later

• A root https://airflow.apache.org/llms.txt in airflow-site, pointing at each package's stable/llms.txt.
• The same for the core apache-airflow docs, which get far more traffic than any provider.
• Versioned offline bundles (a zip of every .md per release) for agents without internet access, as Camel does.

References

• llms.txt format
• Read the Docs: Markdown for agents
• Ray's in-repo Sphinx llms.txt extension
• SPARK-53528: Spark's SPIP for llms.txt in Spark docs, which also uses Sphinx
• INFRA-26812: Fastly caching in front of Airflow's /docs proxy