Visitar URL original
Add a knowledge indexing generator for an LLM-friendly docs index by deruyter92 · Pull Request #3453 · DeepLabCut/DeepLabCut · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2b7bce5
Tools: add knowledge indexing generator for an LLM-friendly docs index
deruyter92 Aug 12, 2026
c37bd0d
fix default output dir: `_build`
deruyter92 Aug 26, 2026
a2a5e0b
use full commit hash of repo
deruyter92 Aug 26, 2026
fe0fbfb
add duplicate check
deruyter92 Aug 26, 2026
65cc178
move knowledge index to JSONL + llms.txt
deruyter92 Aug 26, 2026
75ef519
add content hash and distinct `package_version` (in addition to docs …
deruyter92 Aug 26, 2026
9baf364
update pyproject.toml: fix missing `griffelib` dependency for knowled…
deruyter92 Aug 26, 2026
3ca5c90
allow separate adjustment of doc index and dev-docs api index
deruyter92 Aug 26, 2026
e309fcd
add CI workflows for knowlede-index deployment
deruyter92 Aug 26, 2026
cc46137
update docstrings and readme
deruyter92 Aug 26, 2026
6dc52fb
separate dependencies for leaner CI build&deployment of knowledge-index
deruyter92 Aug 26, 2026
6237e28
simplify llms.txt no per-page summary of doc sections. Prominent ref…
deruyter92 Aug 26, 2026
72f5494
update uv.lock
deruyter92 Aug 26, 2026
21d68b9
validate version labels + tighten manifest reading
deruyter92 Aug 27, 2026
c2db3dd
CI: harden reusable docs workflows against script injection
deruyter92 Aug 27, 2026
cf1f87d
fix delete version + refuse delete "main".
deruyter92 Aug 27, 2026
bf00732
Fix bug in checkout from tag merging into pre-existing checkout
deruyter92 Aug 27, 2026
06967d3
knowledge-index Fix bug in checkout from tag merging into pre-existin…
deruyter92 Aug 27, 2026
8b23cf9
check if dev-docs version is deployed before attempting a delete
deruyter92 Aug 27, 2026
908c6dc
mark dev-docs and knowledge-index dependencies as dependecy-group (ra…
deruyter92 Aug 27, 2026
67111b8
update uv.lock
deruyter92 Sep 4, 2026
e7ab837
add tests for tools/knowledge-indexing
deruyter92 Sep 4, 2026
ff047dd
set `ignore_errors=False` on `delete_version` -> raise if directory i…
deruyter92 Sep 7, 2026
0630f83
fix docs_index frontmatter matching via regex
deruyter92 Sep 7, 2026
6b618fb
update test_write: use real API dataclasses instead of `NameSpace` + …
deruyter92 Sep 7, 2026
e10d279
add tests for frontmatter matching
deruyter92 Sep 7, 2026
76cb4ae
conditional guard dependency install knowledge-indexing: deployment-o…
deruyter92 Sep 7, 2026
58461ed
Move intermediate nodes to schemas (importable without dependencies)
deruyter92 Sep 7, 2026
8ce5334
remove tests/tools/knowledge_indexing/conftest.py
deruyter92 Sep 7, 2026
fcad299
update pyproject.toml and uv lock - add markdow-it-py as dev dependency
deruyter92 Sep 7, 2026
e5bcc48
lazy import docutils (so it is not required at test-time)
deruyter92 Sep 7, 2026
57a9f93
Only the first H1 titles a docs page
deruyter92 Sep 7, 2026
2c884cb
Harden docs frontmatter and record ids
deruyter92 Sep 7, 2026
7864a65
Read docs section anchors from the built site instead of markdown
deruyter92 Sep 7, 2026
b5fb2a0
Resolve api.latest through the latest-release alias
deruyter92 Sep 7, 2026
4453c30
Separate symbol ids from submodule ids with `#`
deruyter92 Sep 7, 2026
02b4f95
Index class methods as their own api records
deruyter92 Sep 7, 2026
3eaa5e1
install knowlegde-index dependency group for testing
deruyter92 Sep 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Read docs section anchors from the built site instead of markdown
Anchors were derived from markdown with make_id(), but this did not reliably handle repeats. e.g. many repeated "Overview" headings were deep-linked to the
wrong section, silently.

--docs-html points at a built site and anchors are derived from its <section id>
attributes, keyed by (heading text, occurrence). If no built site is available, it falls back to link the page instead of the section anchor. CI passes the artifact  (which is already published anyway). A page missing from --docs-html fails the build.

test_docs_anchors_regression.py builds a page with the repo's _config.yml
and asserts every anchor lands on its own heading. It is skipped when no jupyter-book is available so it does not run in CI
  • Loading branch information
deruyter92 committed Sep 8, 2026
commit 7864a65ab7506de0cc746e801d851a23cf1ede81
2 changes: 2 additions & 0 deletions .github/workflows/deploy-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,5 +61,7 @@ jobs:
action: deploy
version_label: main
skip_api: true
# Same artifact deploy-main-docs published; anchors are read from it.
docs_html_artifact: built-book
python-version: "3.10"
secrets: inherit
28 changes: 28 additions & 0 deletions .github/workflows/deploy-knowledge-index.yml
Comment thread
deruyter92 marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,16 @@ on:
default: false
type: boolean

docs_html_artifact:
description: >-
Artifact holding the built user docs, from a job earlier in this run.
Section anchors are read from it, since only the docs build knows what
id it gave a heading. Without it, docs sections link to their page
rather than to a heading.
required: false
default: ""
type: string

python-version:
description: Python version used to build the knowledge index.
required: false
Expand Down Expand Up @@ -117,20 +127,38 @@ jobs:
sparse-checkout: |
knowledge

# Section anchors are read from the built pages rather than derived from
# the markdown: only the docs build knows which id a heading was given.
- name: Download built user docs
if: inputs.action == 'deploy' && !inputs.skip_docs && inputs.docs_html_artifact != ''
uses: actions/download-artifact@v4
with:
name: ${{ inputs.docs_html_artifact }}
path: _docs-html

- name: Build knowledge index
if: inputs.action == 'deploy'
shell: bash
env:
REVISION: ${{ steps.revision.outputs.sha }}
DOCS_HTML_ARTIFACT: ${{ inputs.docs_html_artifact }}
run: |
mkdir -p _build/knowledge-index
if [ -d gh-pages/knowledge ]; then
cp -r gh-pages/knowledge _build/knowledge-index/
fi
# Passed whenever an artifact was named, so a download that produced
# the wrong layout fails the build instead of quietly publishing an
# index with no anchors.
docs_html=()
if [ -n "$DOCS_HTML_ARTIFACT" ]; then
docs_html=(--docs-html _docs-html)
fi
python -m tools.knowledge_indexing \
--output _build/knowledge-index \
--version-label "$VERSION_LABEL" \
--revision "$REVISION" \
"${docs_html[@]}" \
${{ inputs.skip_api && '--skip-api' || '' }} \
${{ inputs.skip_docs && '--skip-docs' || '' }}

Expand Down
145 changes: 145 additions & 0 deletions tests/tools/knowledge_indexing/test_docs_anchors_regression.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
"""Regression check: every anchor we publish exists in the real docs build.

The unit tests check `docs_index` against a hand-written HTML fixture; this
checks that fixture against the toolchain, by building a page with the repo's
own `_config.yml`. A docs upgrade that changes how headings become ids fails
here rather than silently shipping links to the wrong section.

Marked `functional` and skipped without jupyter-book, which lives in the `docs`
extra rather than the `dev` group.
"""

from __future__ import annotations

import re
import shutil
import subprocess
from html import unescape
from pathlib import Path

import pytest
import yaml
from markdown_it import MarkdownIt

from tools.knowledge_indexing.docs_index import _read_structure, read_published_anchors

pytestmark = [
pytest.mark.functional,
pytest.mark.skipif(shutil.which("jupyter-book") is None, reason="jupyter-book (the `docs` extra) not installed"),
]

PAGE = """\
# Sample page

Lead paragraph.

## Overview

First.

## Code example

Second.

## Overview

Third, a repeat.

## Code example:

Fourth, colliding only through punctuation.

(sec:labelled-section)=
## Labelled section

Fifth, carrying an explicit MyST target.

## Output & directory structure

Sixth, with an entity in the heading.

## Overview

Seventh, a third repeat.
"""


def _build(tmp_path: Path) -> str:
"""Build PAGE with the repo's config and return the generated HTML."""
repo = Path(__file__).resolve().parents[3]
config = yaml.safe_load((repo / "_config.yml").read_text(encoding="utf-8"))
# These point at files under docs/ that this minimal book does not carry.
config.pop("logo", None)
config.get("sphinx", {}).get("config", {}).pop("html_static_path", None)
config.get("sphinx", {}).get("config", {}).pop("html_css_files", None)

book = tmp_path / "book"
book.mkdir()
(book / "_config.yml").write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8")
(book / "_toc.yml").write_text("format: jb-book\nroot: page\n", encoding="utf-8")
(book / "page.md").write_text(PAGE, encoding="utf-8")

result = subprocess.run(
[shutil.which("jupyter-book"), "build", str(book)],
capture_output=True,
text=True,
timeout=900,
)
html = book / "_build" / "html" / "page.html"
if not html.is_file():
pytest.fail(f"jupyter-book produced no page.html\n{result.stdout[-2000:]}\n{result.stderr[-2000:]}")
return html.read_text(encoding="utf-8")


def _indexed(html: str):
tokens = MarkdownIt("commonmark").parse(PAGE)
anchors = read_published_anchors(html)
_, _, sections = _read_structure(tokens, "docs:page", "https://example.test/page.html", anchors)
return sections


_TAG = re.compile(r"<[^>]+>")
_HEADING = re.compile(r"<(h[1-6])[^>]*>(.*?)</\1>", re.S)


def _heading_at(page: str, anchor: str) -> str:
"""Text of the heading a fragment lands on, read independently of `docs_index`."""
start = page.find(f'id="{anchor}"')
assert start != -1, f"#{anchor} is not on the page"
match = _HEADING.search(page, start)
assert match, f"no heading follows #{anchor}"
return " ".join(unescape(_TAG.sub("", match.group(2))).split()).rstrip("#").strip()


def test_every_anchor_lands_on_its_own_heading(tmp_path: Path):
html = _build(tmp_path)
sections = _indexed(html)

assert sections, "no sections were indexed"
for section in sections:
assert section.anchor, f"{section.title!r} got no anchor from a page that was built"
assert _heading_at(html, section.anchor) == section.title


def test_repeated_headings_resolve_to_distinct_anchors(tmp_path: Path):
html = _build(tmp_path)
overviews = [s for s in _indexed(html) if s.title == "Overview"]

assert len(overviews) == 3
anchors = [s.anchor for s in overviews]
# All three would otherwise publish #overview, deep-linking to the first.
assert len(set(anchors)) == 3, f"repeated headings share an anchor: {anchors}"
assert anchors[0] == "overview"


def test_headings_needing_more_than_a_slug_are_resolved(tmp_path: Path):
html = _build(tmp_path)
by_title = {s.title: s for s in _indexed(html)}

# An entity in the heading must not leave the section unanchored.
assert by_title["Output & directory structure"].anchor == "output-directory-structure"
# make_id strips the colon, so this collides and cannot keep that slug.
assert by_title["Code example:"].anchor != by_title["Code example"].anchor
# The MyST target case: whichever id the build chose, it must be the one
# that lands on this heading and not merely an id present on the page.
assert _heading_at(html, by_title["Labelled section"].anchor) == "Labelled section"
112 changes: 104 additions & 8 deletions tests/tools/knowledge_indexing/test_docs_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,100 @@
import pytest
from markdown_it import MarkdownIt

from tools.knowledge_indexing.docs_index import _parse_page, _read_structure, _split_frontmatter
from tools.knowledge_indexing.docs_index import (
_page_anchors,
_read_structure,
_split_frontmatter,
read_published_anchors,
)
from tools.knowledge_indexing.toc import TocEntry

PAGE_URL = "https://example.test/docs/page.html"


def _sections(markdown: str):
def _sections(markdown: str, anchors: dict[tuple[str, int], str] | None = None):
tokens = MarkdownIt("commonmark").parse(markdown)
_, _, sections = _read_structure(tokens, "docs:page", PAGE_URL)
_, _, sections = _read_structure(tokens, "docs:page", PAGE_URL, anchors)
return sections


# A built page as Sphinx emits it: repeated headings get auto ids, and each
# heading carries a "#" permalink that is not part of its text.
BUILT_HTML = """
<section id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Link">#</a></h2>
<p>First.</p>
<section id="output-directory-structure">
<h3>Output &amp; directory structure<a class="headerlink" href="#output-directory-structure">#</a></h3>
</section>
</section>
<section id="id1">
<h2>Overview<a class="headerlink" href="#id1" title="Link">#</a></h2>
<p>Second.</p>
</section>
"""


def test_read_published_anchors_keys_by_text_and_occurrence():
anchors = read_published_anchors(BUILT_HTML)
assert anchors[("Overview", 1)] == "overview"
assert anchors[("Overview", 2)] == "id1"


def test_read_published_anchors_strips_the_permalink_and_resolves_entities():
anchors = read_published_anchors(BUILT_HTML)
# Both have to match the markdown heading text for the lookup to hit.
assert anchors[("Output & directory structure", 1)] == "output-directory-structure"


def test_sections_use_the_published_anchor():
markdown = "# Title\n\n## Overview\n\nFirst.\n\n## Overview\n\nSecond.\n"
first, second = _sections(markdown, read_published_anchors(BUILT_HTML))

assert first.anchor == "overview"
assert first.docs_url == f"{PAGE_URL}#overview"
# The repeat gets its own published id, not the first heading's anchor.
assert second.anchor == "id1"
assert second.docs_url == f"{PAGE_URL}#id1"


def test_sections_fall_back_to_the_page_without_a_build():
first, second = _sections("# Title\n\n## Overview\n\nA.\n\n## Overview\n\nB.\n")

# No anchors available: link to the page rather than guess a fragment.
for section in (first, second):
assert section.anchor == ""
assert section.docs_url == PAGE_URL


def test_heading_the_build_publishes_no_section_for_gets_no_anchor():
markdown = "# Title\n\n## Overview\n\nA.\n\n## Not A Section\n\nB.\n"
_, orphan = _sections(markdown, read_published_anchors(BUILT_HTML))

assert orphan.title == "Not A Section"
assert orphan.anchor == ""
assert orphan.docs_url == PAGE_URL


def test_record_ids_are_stable_with_and_without_a_build():
markdown = "# Title\n\n## Overview\n\nA.\n\n## Overview\n\nB.\n"
# Identity comes from the source, so it does not depend on the build.
with_build = [s.id for s in _sections(markdown, read_published_anchors(BUILT_HTML))]
without_build = [s.id for s in _sections(markdown)]

assert with_build == without_build == ["docs:page#overview", "docs:page#overview~2"]


def test_page_title_h1_does_not_consume_a_later_heading_s_occurrence():
# The build counts the title when numbering repeats, so the lookup must too;
# otherwise the h2 asks for ("Overview", 1) and gets the title's anchor.
html = '<section id="overview"><h2>Overview</h2></section>'
anchors = read_published_anchors('<section id="intro"><h1>Overview</h1></section>' + html)

(section,) = _sections("# Overview\n\nLead.\n\n## Overview\n\nBody.\n", anchors)
assert section.anchor == "overview"


def test_only_the_first_h1_is_the_page_title():
# Some pages use h1 throughout, so only the first may be taken as the title.
sections = _sections("# Title\n\nLead.\n\n# Second\n\nA.\n\n# Third\n\nB.\n")
Expand All @@ -34,12 +116,26 @@ def test_heading_that_slugs_to_nothing_still_gets_a_usable_id():
assert first.id != second.id


def test_non_mapping_audit_frontmatter_is_rejected(tmp_path: Path):
page = tmp_path / "page.md"
page.write_text("---\ndeeplabcut: true\n---\n# Title\n", encoding="utf-8")
def test_page_anchors_reads_the_html_mirroring_the_toc_path(tmp_path: Path):
# Pins the layout production expects of a downloaded docs artifact:
# <html_dir>/<toc entry>.html, source layout mirrored.
page = tmp_path / "docs" / "main-workflows" / "user-guide.html"
page.parent.mkdir(parents=True)
page.write_text(BUILT_HTML, encoding="utf-8")

entry = TocEntry(file="docs/main-workflows/user-guide")
assert _page_anchors(tmp_path, entry)[("Overview", 1)] == "overview"


def test_page_anchors_raises_when_a_requested_page_is_not_built(tmp_path: Path):
# Asking for anchors and silently getting page-level links back is the
# failure this design exists to prevent, so a missing page must not degrade.
with pytest.raises(FileNotFoundError, match="every page in"):
_page_anchors(tmp_path, TocEntry(file="docs/missing"))


with pytest.raises(ValueError, match="must be a mapping"):
_parse_page(page, TocEntry(file="docs/page"), "")
def test_page_anchors_without_a_build_is_not_an_error():
assert _page_anchors(None, TocEntry(file="docs/missing")) == {}


def test_frontmatter_is_split_from_body():
Expand Down
27 changes: 22 additions & 5 deletions tools/knowledge_indexing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ python -m tools.knowledge_indexing
| `--output` | Directory mirroring the gh-pages site root to write into (default: `<repo>/_build/knowledge-index`) |
| `--repo` | Repository root, containing `_toc.yml`, `docs/` and `deeplabcut/` (default: cwd) |
| `--version-label` | Developer-docs version label the API URLs point at, one of the labels mike deploys (default: `main`) |
| `--docs-html` | Root of a built docs site (e.g. `_build/html`) to read published section anchors from. Without it, docs sections link to their page instead of to a heading |
| `--revision` | Commit the source was read at, recorded in the manifest (default: `HEAD` of `--repo`) |
| `--skip-api` | Don't rebuild `api.jsonl` this run; keep it and its manifest provenance as already on disk |
| `--skip-docs` | Don't rebuild `docs.jsonl`/`llms.txt` this run; keep them and their manifest provenance as already on disk |
Expand Down Expand Up @@ -113,9 +114,21 @@ tool generates no concepts at all — see "Known gaps".

Each page is one `type: "page"` row in `docs.jsonl`; each heading below its title
is its own `type: "section"` row, carrying an `anchor` and a `url` pointing at
it, plus a `summary`: the first paragraph of prose under the heading. Anchors
come from `docutils.nodes.make_id`, the function docutils uses to build section
ids. A heading whose body is only a subheading, a figure or code has no summary.
it, plus a `summary`: the first paragraph of prose under the heading. A heading
whose body is only a subheading, a figure or code has no summary.

Anchors are **read from the built docs, not derived from the markdown**. Which id
a heading gets is decided by Sphinx and docutils together: `make_id(title)` for
the first heading of a name, a document-wide `id1`, `id2`, … for every repeat
(including headings differing only in punctuation, which `make_id` strips), an
explicit id where a MyST target supplies one, and no section at all where a title
is promoted to the page title. Those depend on toolchain settings, so deriving
them from source would break silently whenever the docs stack changes. Pass
`--docs-html` to get the published anchors; without it, sections link to their
page. Record `id`s come from the source either way, so they match across modes.
`tests/tools/knowledge_indexing/test_docs_anchors_regression.py` builds a page
with the repo's own `_config.yml` and checks every anchor we publish is really
on it.
A section row denormalises its page's `section` (toc part) so it stands on its
own, but everything else about the page — `status`, `parent`, `children`,
`related_pages`, `labels` — lives only on the page row, reachable via the
Expand Down Expand Up @@ -200,8 +213,12 @@ user-docs half can never get built by the wrong trigger:
- **Notebooks are not indexed.** `_toc.yml` lists notebooks under `examples/`,
which would need a reader for `.ipynb` markdown cells.
- **Validation is minimal.** Duplicate record ids abort the write, but there is
no `--check` mode, so anchors, dangling references and unresolvable URLs are
not caught.
no `--check` mode, so dangling references and unresolvable URLs are not
caught.
- **Section anchors need the built docs.** A run without `--docs-html` publishes
page-level links for every section. That is correct but less precise, and the
two modes produce different `url`s (record ids are unaffected). CI passes the
built book, so the deployed index always has anchors.
- **`knowledge/manifest.json` is rebuilt by rescanning disk**, not by tracking
state across separate builds -- see "Deployment" below for how CI seeds that
rescan. `api.latest` currently always points at `main`; resolving "latest
Expand Down
Loading