Visitar URL original
Add a knowledge indexing generator for an LLM-friendly docs index by deruyter92 · Pull Request #3453 · DeepLabCut/DeepLabCut · GitHub
Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2b7bce5
Tools: add knowledge indexing generator for an LLM-friendly docs index
deruyter92 Aug 12, 2026
c37bd0d
fix default output dir: `_build`
deruyter92 Aug 26, 2026
a2a5e0b
use full commit hash of repo
deruyter92 Aug 26, 2026
fe0fbfb
add duplicate check
deruyter92 Aug 26, 2026
65cc178
move knowledge index to JSONL + llms.txt
deruyter92 Aug 26, 2026
75ef519
add content hash and distinct `package_version` (in addition to docs …
deruyter92 Aug 26, 2026
9baf364
update pyproject.toml: fix missing `griffelib` dependency for knowled…
deruyter92 Aug 26, 2026
3ca5c90
allow separate adjustment of doc index and dev-docs api index
deruyter92 Aug 26, 2026
e309fcd
add CI workflows for knowlede-index deployment
deruyter92 Aug 26, 2026
cc46137
update docstrings and readme
deruyter92 Aug 26, 2026
6dc52fb
separate dependencies for leaner CI build&deployment of knowledge-index
deruyter92 Aug 26, 2026
6237e28
simplify llms.txt no per-page summary of doc sections. Prominent ref…
deruyter92 Aug 26, 2026
72f5494
update uv.lock
deruyter92 Aug 26, 2026
21d68b9
validate version labels + tighten manifest reading
deruyter92 Aug 27, 2026
c2db3dd
CI: harden reusable docs workflows against script injection
deruyter92 Aug 27, 2026
cf1f87d
fix delete version + refuse delete "main".
deruyter92 Aug 27, 2026
bf00732
Fix bug in checkout from tag merging into pre-existing checkout
deruyter92 Aug 27, 2026
06967d3
knowledge-index Fix bug in checkout from tag merging into pre-existin…
deruyter92 Aug 27, 2026
8b23cf9
check if dev-docs version is deployed before attempting a delete
deruyter92 Aug 27, 2026
908c6dc
mark dev-docs and knowledge-index dependencies as dependecy-group (ra…
deruyter92 Aug 27, 2026
67111b8
update uv.lock
deruyter92 Sep 4, 2026
e7ab837
add tests for tools/knowledge-indexing
deruyter92 Sep 4, 2026
ff047dd
set `ignore_errors=False` on `delete_version` -> raise if directory i…
deruyter92 Sep 7, 2026
0630f83
fix docs_index frontmatter matching via regex
deruyter92 Sep 7, 2026
6b618fb
update test_write: use real API dataclasses instead of `NameSpace` + …
deruyter92 Sep 7, 2026
e10d279
add tests for frontmatter matching
deruyter92 Sep 7, 2026
76cb4ae
conditional guard dependency install knowledge-indexing: deployment-o…
deruyter92 Sep 7, 2026
58461ed
Move intermediate nodes to schemas (importable without dependencies)
deruyter92 Sep 7, 2026
8ce5334
remove tests/tools/knowledge_indexing/conftest.py
deruyter92 Sep 7, 2026
fcad299
update pyproject.toml and uv lock - add markdow-it-py as dev dependency
deruyter92 Sep 7, 2026
e5bcc48
lazy import docutils (so it is not required at test-time)
deruyter92 Sep 7, 2026
57a9f93
Only the first H1 titles a docs page
deruyter92 Sep 7, 2026
2c884cb
Harden docs frontmatter and record ids
deruyter92 Sep 7, 2026
7864a65
Read docs section anchors from the built site instead of markdown
deruyter92 Sep 7, 2026
b5fb2a0
Resolve api.latest through the latest-release alias
deruyter92 Sep 7, 2026
4453c30
Separate symbol ids from submodule ids with `#`
deruyter92 Sep 7, 2026
02b4f95
Index class methods as their own api records
deruyter92 Sep 7, 2026
3eaa5e1
install knowlegde-index dependency group for testing
deruyter92 Sep 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix docs_index frontmatter matching via regex
  • Loading branch information
deruyter92 committed Sep 8, 2026
commit 0630f832442dc5e20a27069e4d8ffa36367c50ff
17 changes: 10 additions & 7 deletions tools/knowledge_indexing/docs_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -358,18 +358,21 @@ def _shorten(text: str, limit: int = EXCERPT_MAX_CHARS) -> str:
return text[:limit].rsplit(" ", 1)[0].rstrip(",;:") + "…"


_FRONTMATTER = re.compile(r"\A---[^\S\n]*\n(.*?)^---[^\S\n]*(?:\n|\Z)", re.DOTALL | re.MULTILINE)


def _split_frontmatter(text: str) -> tuple[dict[str, Any], str]:
"""Split YAML frontmatter from the markdown body."""
if not text.startswith("---"):
return {}, text
parts = text.split("---", 2)
if len(parts) < 3:
match = _FRONTMATTER.match(text)
if match is None:
return {}, text
try:
frontmatter = yaml.safe_load(parts[1])
frontmatter = yaml.safe_load(match.group(1))
except yaml.YAMLError:
return {}, parts[2]
return (frontmatter if isinstance(frontmatter, dict) else {}), parts[2]
return {}, text
if not isinstance(frontmatter, dict):
return {}, text
return frontmatter, text[match.end() :]


def _local_id(toc_file: str) -> str:
Expand Down