Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
60 lines (46 loc) · 4 KB

ARCHITECTURE.md

File metadata and controls

60 lines (46 loc) · 4 KB
Raw

Architecture

Pipeline

┌─────────┐   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐
│  text   ├──▶│ 70 regex /   ├──▶│  GLiNER NER  ├──▶│  anonymizer  │──▶ tokenised text + vault
│         │   │  checksum    │   │  (PII labels │   │  (6 ops)     │
│         │   │  recognizers │   │  + DE clin.) │   │              │
└─────────┘   └──────────────┘   └──────────────┘   └──────────────┘
                     │                  │                   │
                     ▼                  ▼                   ▼
              ENGLISH/EU IDs       PERSON/ORG/LOC      Replace, Redact,
              IBAN, phone,         AGE, PROFESSION,    Mask, Hash,
              email, SSN,          (multilingual)      Encrypt, Synthesize
              passport, …

The detection bias is recall > precision: anonde would rather over-tokenise (safe) than miss a PHI span (a leak). The bench tracks this explicitly via the leak_rate metric (lower = better).

Directory layout

anonde/
├── analyzer/                  # recognizer registry + parallel dispatch
│   ├── analyzer.go            # AnalyzerEngine.Analyze: filter → dispatch → conflict resolve
│   ├── result.go              # RecognizerResult + RemoveConflicts (NER-preference rule)
│   └── recognizers/           # 70 pattern + GLiNER NER recognizers
│       ├── *Recognizer.go     # per-region pattern recognizers
│       └── ner_gliner.go      # `-tags ner`: GLiNER (open-set NER) via yalue/onnxruntime_go
├── anonymizer/                # apply operators to detected spans
│   ├── anonymizer.go          # mergeAdjacentSameType + dispatch to operators
│   └── operators/             # Replace, Redact, Mask, Hash, Encrypt, Synthesize, Keep
├── cmd/anonde/              # HTTP service
├── internal/api/              # transport (Connect + gRPC + REST gateway)
└── bench/                     # single bench harness
    ├── Makefile               # top-level `make matrix`, `make matrix-de`, `make matrix-en`, …
    ├── corpora/<NAME>/        # per-corpus Makefile + loader + data + gold
    ├── runners/               # one Go runner, three Python sidecars (gliner, openai_pf, presidio)
    ├── probes/                # diagnostic loaders for hugot, gliner
    └── scoring/               # compare.py, render_matrix.py, label_map.yaml

Conflict resolution (the non-obvious part)

RemoveConflicts keeps the highest-scoring span when two overlap, except for entity types where NER is more reliable than heuristic patterns (PERSON, ORGANIZATION, LOCATION, AGE, PROFESSION, NRP). For those, an NER finding beats a pattern finding regardless of score.

Why: pattern recognizers like DEAnomalyRecognizer produce fixed scores (0.85); GLiNER produces sigmoid floats (0.4–0.85). Without this rule, patterns always won and the NER's contextual judgement was wasted.

For structured types (IBAN, PHONE_NUMBER, DATE_TIME, EMAIL_ADDRESS, …) the score-only rule still applies; regex+checksum precision matters more than NER context there.

See analyzer/result.go, shouldReplace.

Per-recognizer error visibility

Silent failures in the analyzer pipeline are logged via analyzer: recognizer error (swallowed) so a broken NER backend can't masquerade as patterns-only. CI also asserts that NER cells produce a non-zero number of NER-attributable findings (see DEPLOYMENT.md).

In-memory vault

The anonde server ships an in-memory vault (token ↔ cleartext) with configurable TTL; no DB required for ephemeral workloads. TTLs and the request-size cap are env-tunable; see DEPLOYMENT.md.