Tools: add knowledge indexing generator for an LLM-friendly docs index

Adds tools/knowledge_indexing, which reads the repository and writes a directory of small YAML files describing the codebase and user docs, one node per file, so an agent can load them lazily by id. Everything comes from the checkout, not from built or deployed docs: griffe reads the source statically for API signatures and docstrings, markdown-it-py parses the Markdown, and _toc.yml defines which pages are published along with their part/parent/child hierarchy. On the current checkout that is 270 nodes: 207 modules with 685 documented symbols, and 63 pages with 656 sections. Docs pages carry sections rather than concepts. A section is a retrievable part of one document, with an anchor, an absolute docs_url and the first paragraph of prose under the heading; anchors come from docutils.nodes.make_id, so they match the ids Sphinx emits. Concepts are reusable entities discussed across documents, which slugged headings are not, so none are generated. symbols.yaml maps every symbol id to the module node documenting it, for lookups that start from a bare name. index.yaml records the schema version, git revision and the two site base URLs the
deruyter92 · Sep 8, 2026 · Aug 12, 2026 · Aug 26, 2026 · Aug 26, 2026 · Aug 26, 2026
commit 2b7bce5ecef5fceea9f71bdb1e8262c8ea0c9859
diff --git a/tools/knowledge_indexing/README.md b/tools/knowledge_indexing/README.md
@@ -0,0 +1,126 @@
+# Knowledge indexing
+
+Generates an LLM-friendly knowledge index from the DeepLabCut source and user
+documentation. The output is a directory of small YAML files, one per node, that
+an agent can load lazily by id.
+
+## Usage
+
+```bash
+python -m tools.knowledge_indexing
+```
+
+| Option | Meaning |
+|---|---|
+| `--output` | Directory to write the index into (default: `<repo>/build/knowledge-index/<version>`) |
+| `--repo` | Repository root, containing `_toc.yml`, `docs/` and `deeplabcut/` (default: cwd) |
+| `--version` | Developer-docs version the API URLs point at, one of the versions mike deploys (default: `main`) |
+
+Everything is read from the repository, so no built or deployed documentation is
+needed. Both documentation extras are required: `griffe` comes with `dev-docs`,
+`docutils` with `docs`, and `markdown-it-py` with either. `pyyaml` is a core
+dependency.
+
+```bash
+pip install -e ".[docs,dev-docs]"
+```
+
+## Output
+
+```text
+build/knowledge-index/main/
+├── index.yaml        manifest: schema, provenance, and every node id with its file
+├── symbols.yaml      flat symbol id -> module node lookup
+├── apis/             one file per published module
+└── docs-pages/       one file per published docs page, with its sections
+```
+
+Read `index.yaml` first; it maps each node id to its file, which matters because
+ids are namespaced and docs-page ids are nested (`docs:pytorch/user_guide`) while
+the directories are flat.
+
+Ids are namespaced by type — `docs:`, `docs:<page>#<anchor>`, `api:` — so a
+reference is unambiguous about what it points at.
+
+Every `docs_url` is absolute, so a node carries a working link on its own. The two
+sites they are built against are recorded under `base_urls` in the manifest: the
+user docs are unversioned at the site root, while the developer docs are deployed
+per version by mike under `dev/<version>/`, which is the segment `--version`
+fills in. Both bases are defined in `__main__.py`.
+
+## Where the data comes from
+
+### APIs
+
+[griffe](https://mkdocstrings.github.io/griffe/) is the static analysis library
+`mkdocstrings` uses to build the developer docs. Reading the source with the same
+tool that renders it means signatures and docstrings match what is published, and
+nothing has to be imported — no torch, no GPU, no optional dependencies.
+
+Each symbol gets a `source` (`file.py:lineno`) and a `docs_url` derived from its
+dotted path, following the layout `mkdocs-api-autonav` generates. Only documented
+symbols are indexed, because `mkdocstrings` omits undocumented members from the
+reference pages, so an entry for one would have neither a summary nor a URL that
+resolves.
+
+API nodes are grouped per module, which keeps each file worth loading, so
+`symbols.yaml` maps every symbol id to the module node documenting it for lookups
+that start from a bare name.
+
+Two constants in `api_index.py` mirror `dev-docs/mkdocs.yml` and must be kept in
+sync with it: `EXCLUDED_MODULES` and `API_ROOT_URI`.
+
+### Docs pages
+
+`_toc.yml` defines the scope. It lists the pages that are actually published —
+`docs/` holds others that it deliberately leaves out — and it is the only place
+the part / chapter / section hierarchy exists, so it also supplies each page's
+`part`, `parent` and `children`.
+
+Parsing uses `markdown-it-py`, the parser Jupyter Book itself uses, because the
+docs nest code fences up to five backticks deep; matching fences with regular
+expressions instead lets shell comments inside them parse as headings.
+
+**Headings become sections, not concepts.** A section is a retrievable part of
+one document; a concept is a reusable domain entity discussed across documents.
+Slugged headings are the former pretending to be the latter, which is why this
+tool generates no concepts at all — see "Known gaps".
+
+Each section carries an `anchor` and a `docs_url` pointing at it, plus an
+`excerpt`: the first paragraph of prose under the heading. Anchors come from
+`docutils.nodes.make_id`, the function docutils uses to build section ids. A
+heading whose body is only a subheading, a figure or code has no excerpt.
+
+Admonition-style MyST directives (`{note}`, `{important}`, `{tip}`, …) are fenced
+blocks, and their bodies are parsed for prose rather than skipped as code.
+
+Pages link to each other mostly through MyST labels rather than file paths, so
+parsing happens in two passes: collect the labels each page defines, then resolve
+every link and `{ref}` against them. Each node publishes its own `labels`, which
+is what a `{ref}` elsewhere in the docs resolves against.
+
+`status`, `visibility` and `last_verified` are copied from each page's audit
+frontmatter. Nothing is filtered on them; only an explicit `ignore: true` skips a
+page.
+
+## Module layout
+
+| File | Responsibility |
+|---|---|
+| `schemas.py` | The on-disk schema: every published file and node type |
+| `toc.py` | `_toc.yml` → the set of published pages and their hierarchy |
+| `api_index.py` | Source tree → API nodes, via griffe |
+| `docs_index.py` | Markdown → docs-page nodes with sections |
+| `write.py` | Nodes → YAML files, symbol table and manifest |
+| `__main__.py` | CLI |
+
+## Known gaps
+
+- **No concepts or workflows.** These need to be authored rather than derived
+  from headings.
+- **Notebooks are not indexed.** `_toc.yml` lists notebooks under `examples/`,
+  which would need a reader for `.ipynb` markdown cells.
+- **Nothing validates the index.** There is no `--check` mode, so anchors,
+  dangling references, duplicate ids and unresolvable URLs are not caught.
+- **The user docs are not versioned** upstream, so `docs-pages/` is identical
+  across index versions and only stamped with the revision it came from.
diff --git a/tools/knowledge_indexing/__init__.py b/tools/knowledge_indexing/__init__.py
@@ -0,0 +1,39 @@
+"""Generate an LLM-friendly knowledge index from the DeepLabCut documentation.
+
+The index is a directory of small YAML files, one per node, that an agent can
+load lazily by id. See README.md for the layout and the CLI.
+"""
+
+from __future__ import annotations
+
+from .api_index import build_api_nodes
+from .docs_index import build_docs_nodes
+from .schemas import (
+    ApiNode,
+    DocsPageNode,
+    Manifest,
+    Node,
+    Section,
+    Symbol,
+    SymbolEntry,
+    SymbolTable,
+)
+from .toc import TocEntry, read_toc
+from .write import write_index, write_symbol_table
+
+__all__ = [
+    "ApiNode",
+    "DocsPageNode",
+    "Manifest",
+    "Node",
+    "Section",
+    "Symbol",
+    "SymbolEntry",
+    "SymbolTable",
+    "TocEntry",
+    "build_api_nodes",
+    "build_docs_nodes",
+    "read_toc",
+    "write_index",
+    "write_symbol_table",
+]
diff --git a/tools/knowledge_indexing/__main__.py b/tools/knowledge_indexing/__main__.py
@@ -0,0 +1,115 @@
+"""Command line entry point.
+
+    python -m tools.knowledge_indexing
+
+See README.md for the output layout.
+"""
+
+from __future__ import annotations
+
+import argparse
+import subprocess
+import sys
+from pathlib import Path
+
+from .api_index import build_api_nodes
+from .docs_index import build_docs_nodes
+from .schemas import API_DIRECTORY, DOCS_DIRECTORY
+from .toc import TOC_FILE
+from .write import write_index, write_symbol_table
+
+PACKAGE = "deeplabcut"
+
+# Default output, relative to --repo and per version, so the index is a build
+# artifact of the repository it describes. `build/` is already gitignored.
+DEFAULT_OUTPUT_ROOT = Path("build/knowledge-index")
+
+# The published sites every `docs_url` is built against. The user docs are
+# unversioned and live at the site root, while the developer docs are deployed per
+# version by mike under `dev/` (`plugins.mike` in dev-docs/mkdocs.yml), which is
+# the segment --version fills in.
+DOCS_BASE_URL = "https://deeplabcut.github.io/DeepLabCut/"
+API_BASE_URL = "https://deeplabcut.github.io/DeepLabCut/dev/{version}/"
+
+
+def _parse_args(argv: list[str] | None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        prog="python -m tools.knowledge_indexing",
+        description="Generate an LLM-friendly knowledge index from the DeepLabCut docs.",
+    )
+    parser.add_argument(
+        "--output",
+        type=Path,
+        help=f"Directory to write the index into (default: <repo>/{DEFAULT_OUTPUT_ROOT}/<version>).",
+    )
+    parser.add_argument(
+        "--repo",
+        type=Path,
+        default=Path(),
+        help=f"Repository root, containing {TOC_FILE}, docs/ and {PACKAGE}/ (default: cwd).",
+    )
+    parser.add_argument(
+        "--version",
+        default="main",
+        help=(
+            "Developer-docs version the API URLs point at, also recorded in the "
+            "manifest. One of the versions mike deploys, e.g. main or 3.0 (default: main)."
+        ),
+    )
+    return parser.parse_args(argv)
+
+
+def _git_revision(repo: Path) -> str:
+    """Short commit hash of `repo`, or "" if it is not a git checkout."""
+    try:
+        result = subprocess.run(
+            ["git", "-C", str(repo), "rev-parse", "--short", "HEAD"],
+            capture_output=True,
+            text=True,
+            timeout=10,
+            check=True,
+        )
+    except (OSError, subprocess.SubprocessError):
+        return ""
+    return result.stdout.strip()
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = _parse_args(argv)
+    repo: Path = args.repo
+
+    for required in (repo / TOC_FILE, repo / PACKAGE):
+        if not required.exists():
+            print(f"Error: {required} not found; is --repo the repository root?", file=sys.stderr)
+            return 1
+
+    output: Path = args.output or repo / DEFAULT_OUTPUT_ROOT / args.version
+    base_urls = {
+        API_DIRECTORY: API_BASE_URL.format(version=args.version),
+        DOCS_DIRECTORY: DOCS_BASE_URL,
+    }
+
+    print(f"Reading API from {repo / PACKAGE} ...")
+    apis = build_api_nodes(PACKAGE, repo, base_urls[API_DIRECTORY])
+    print(f"  {len(apis)} modules, {sum(len(node.symbols) for node in apis)} documented symbols")
+
+    print(f"Reading user docs listed in {repo / TOC_FILE} ...")
+    docs_pages = build_docs_nodes(repo, base_urls[DOCS_DIRECTORY])
+    print(f"  {len(docs_pages)} pages, {sum(len(page.sections) for page in docs_pages)} sections")
+
+    groups = {API_DIRECTORY: apis, DOCS_DIRECTORY: docs_pages}
+    write_index(
+        output,
+        groups,
+        version=args.version,
+        revision=_git_revision(repo),
+        base_urls=base_urls,
+    )
+    symbols = write_symbol_table(output, apis)
+
+    print(f"Wrote {len(apis) + len(docs_pages)} nodes and {symbols} symbol entries to {output}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())