anony-mate-api

AnonyMate is the backend of a modern web application for redacting sensitive information from free-form text and PDF documents. Built with FastAPI and Python 3.15, it converts documents to text (Docling), detects sensitive entities (GLiNER), and writes redaction marks back onto PDFs for an editor such as Kofax Power PDF or Acrobat to apply.

This repository contains the backend API; the frontend is built with Nuxt 4 and available at https://github.com/DCC-BS/anony-mate-frontend.

DCC Documentation & Guidelines | DCC Website

• Document Conversion: Uploads (.docx, .txt, PDF, scans) are converted to reviewable text via Docling — with OCR for scanned pages and per-word boxes for PDF marking.
• Entity Detection: GLiNER finds sensitive entities for a caller-defined schema of entity types (label plus natural-language description), running as async tasks with polling so long scans cannot be cut off.
• Redaction: Detected occurrences are replaced from per-type templates ({name}, {label}, {subject}, {occurrence}), with a never-redact (blacklist) exclusion list honoured throughout.
• PDF Marking: Writes one redaction annotation (ISO 32000) per kept mark onto the original PDF, named by its mark id and carrying the placeholder as its overlay text — nothing is removed; the editor applies the marks.
• Stateless Task Store: The API holds no client state — documents and decisions live in the browser (IndexedDB). Task results are polled, TTL-bounded, and dropped when a caller vanishes.
• Backpressure & Fairness: Bounded conversion and redaction queues with 429 refusals instead of unbounded upstream queues; abandoned tasks are dropped before a worker picks them up.
• Entity Type Presets: Serves curated preset schemas (default, legal, full) the frontend seeds into its local groups.

• Framework: FastAPI (ASGI via Uvicorn)
• Python: 3.15, packaged with Hatchling + version-pioneer
• Package & Environment Manager: uv
• Document Conversion: Docling / docling-serve
• Entity Detection: GLiNER via dcc-gliner-api
• PDF Handling: pypdf
• Dependency Injection: dependency-injector
• Logging & Tracing: structlog, Logfire
• Environment & Secrets: Varlock with Proton Pass plugin
• Tool Version Manager: Mise
• Linter, Formatter & Type Check: Ruff, ty
• Testing: pytest (incl. doctests)

We recommend using mise to automatically manage tool versions (uv, Varlock, pass-cli):

mise install

Create a .env file in the project root (see .env.example):

APP_MODE=dev      # dev, ci, build, prod

Environment variables are validated using Varlock schemas (.env.schema, .env.dev.schema, .env.azure.schema). Highlights:

We use varlock for schema-based validation and optional secret retrieval from Proton Pass:

# Validate and load secrets into the environment
mise run env

Start the FastAPI dev server with auto-reload at http://localhost:8000:

# Using mise
mise run dev

# Or using the Makefile
make dev

mise run run

The API documentation (OpenAPI/Swagger) is served at http://localhost:8000/docs.

The client library ships as its own lightweight package, anony-mate-lib (a standalone project in lib/; it needs only httpx and pydantic). Its API is documented in lib/README.md:

uv add "anony-mate-lib @ git+https://github.com/DCC-BS/anony-mate-api#subdirectory=lib"

On entering it loads the default presets of entity types and blacklist, so redaction works without settings of its own; the async submissions poll the task and collect the result as part of the call:

from anony_mate_lib import AnonyMateClient, RedactInput

async with AnonyMateClient("http://localhost:8000") as client:
    result = await client.anonymize_text(RedactInput(text="Anna wohnt in Berlin."))
    print(result.text)  # [person:1] wohnt in [ort:1].

The request and response models live in anony_mate_lib.models; the building blocks the client itself is made of — preset and task polling helpers for use with an own httpx client — one level below.

The detection and conversion services run as sidecar containers via Docker Compose (GLiNER API, docling-serve; optional vLLM for the LLM):

mise run docker:up

mise run docker:logs

mise run docker:down

Marking scanned PDFs needs a docling-serve built with PP-OCRv6, PP-DocLayoutV3 and the word-boxes plugin (see dcc-docling-serve/plugins/word_boxes.py); the published upstream image does not yet provide these.

src/anony_mate_api/
├── app.py                  # FastAPI application factory & lifespan
├── container.py            # Dependency-injector wiring
├── routers/                # HTTP endpoints
│   ├── conversion_router.py    # Document upload & conversion
│   ├── redact_router.py        # Text redaction
│   ├── task_router.py          # Task status polling
│   ├── entity_types_router.py  # Entity type presets & schemas
│   └── blacklist_router.py     # Never-redact terms
├── services/               # Business logic
│   ├── document_conversion_service.py  # Docling submission & polling
│   ├── redact_service.py               # GLiNER detection & replacement
│   ├── pdf_marking_service.py          # Redaction annotations on PDFs
│   ├── pdf_marks/                      # Mark geometry & word boxes
│   ├── text_matching.py                # Occurrence matching
│   └── task_store.py                   # In-memory task store (TTL, backpressure)
├── models/                 # Pydantic request/response models
└── utils/                  # Configuration & helpers
tests/                      # pytest suite (incl. doctests)
lib/                        # anony-mate-lib client package (API docs: lib/README.md)
tools/                      # Load testing

# Run the test suite (including doctests), API server and client library
mise run test:unit

# Verify lockfile, format, lint, type-check, and scan for secrets
mise run check

# Same, without modifying files (CI mode)
mise run ci-check

# Everything: checks + tests
mise run ci

# Build the wheel file
mise run build

Build and run using the multi-stage Dockerfile:

# Build Docker image
docker build -t anony-mate-api .

# Run container
docker run -p 8000:8000 anony-mate-api

MIT © Data Competence Center Basel-Stadt

Datenwissenschaften und KI
Developed with ❤️ by DCC - Data Competence Center