|
| 1 | +# DataFog benchmarks |
| 2 | + |
| 3 | +Every performance number DataFog publishes should be reproducible by a |
| 4 | +skeptic with one command. This directory is that command: |
| 5 | + |
| 6 | +```bash |
| 7 | +pip install -e . -r benchmarks/requirements.txt |
| 8 | +python -m spacy download en_core_web_sm # comparison suites only |
| 9 | +python benchmarks/run.py |
| 10 | +``` |
| 11 | + |
| 12 | +Suites that are missing optional dependencies are skipped with an install |
| 13 | +hint — `python benchmarks/run.py --suite core,hook` runs with nothing but |
| 14 | +datafog itself installed. `--json results.json` writes machine-readable |
| 15 | +output; `--quick` is a fast smoke run (fewer repeats, large payloads |
| 16 | +skipped). |
| 17 | + |
| 18 | +## Methodology |
| 19 | + |
| 20 | +- **In-process timings** use stdlib `timeit`: `Timer.autorange()` picks an |
| 21 | + inner loop count so each timed repeat runs ≥0.2s, then 5 repeats (3 in |
| 22 | + comparison suites) are reported as median ± stdev. No mocking — every |
| 23 | + call goes through the same public code path a user hits. |
| 24 | +- **The hook suite measures wall-clock subprocess time**: a fresh |
| 25 | + `datafog-hook` process per iteration, JSON payload on stdin, exactly as |
| 26 | + Claude Code invokes it. 3 warmup runs, 30 timed runs, median and p90 |
| 27 | + reported. Python interpreter startup is *included* — that is the honest |
| 28 | + cost of a per-tool-call hook. |
| 29 | +- **Pinned payloads** live in [payloads/](payloads/) and are checked in, |
| 30 | + not generated at runtime. [payloads/manifest.json](payloads/manifest.json) |
| 31 | + records the entity counts the regex engine must find in each payload; the |
| 32 | + runner verifies those counts *before* timing anything, so the suite fails |
| 33 | + loudly rather than silently benchmarking an engine that stopped |
| 34 | + detecting. All PII values are industry-standard synthetic test data |
| 35 | + (reserved `.invalid`/`example.com` domains, fictional 555-01XX phone |
| 36 | + numbers, public Luhn-valid test cards, SSA example SSNs). |
| 37 | + |
| 38 | +| Payload | Size | Content | |
| 39 | +| --- | --- | --- | |
| 40 | +| `core_1kb_dense.txt` | 1.2 KB | support ticket, 8 entities (PII-dense) | |
| 41 | +| `core_10kb_mixed.txt` | 10 KB | business document, 12 entities in prose | |
| 42 | +| `core_100kb_sparse.txt` | 100 KB | machine logs, 18 entities among UUIDs/timestamps (false-positive pressure) | |
| 43 | +| `chat_request_clean.json` | — | 2-message LiteLLM chat body, no PII | |
| 44 | +| `chat_request_pii.json` | — | 2-message LiteLLM chat body, email + card + phone | |
| 45 | +| `hook_pretooluse.json` | — | Claude Code `PreToolUse` payload, `curl` command carrying PII | |
| 46 | + |
| 47 | +All suites use the production default entity set |
| 48 | +(`EMAIL, PHONE, CREDIT_CARD, SSN`). |
| 49 | + |
| 50 | +## Reference results |
| 51 | + |
| 52 | +Apple M5 Pro, macOS, CPython 3.13, datafog 4.7.0, litellm 1.91.0, |
| 53 | +presidio-analyzer 2.2.363, spaCy 3.8 (`en_core_web_sm`). Medians of the |
| 54 | +full (non-quick) run; expect different absolute numbers on different |
| 55 | +hardware, but the ratios and orders of magnitude should hold. |
| 56 | + |
| 57 | +### core — `datafog.scan` / `datafog.redact`, regex engine |
| 58 | + |
| 59 | +| Operation | Median | Throughput | |
| 60 | +| --- | --- | --- | |
| 61 | +| scan 1.2 KB dense | 231 µs | 5.3 MB/s | |
| 62 | +| redact 1.2 KB dense | 240 µs | 5.1 MB/s | |
| 63 | +| scan 10 KB mixed | 1.20 ms | 8.6 MB/s | |
| 64 | +| redact 10 KB mixed | 1.19 ms | 8.6 MB/s | |
| 65 | +| scan 100 KB sparse | 11.9 ms | 8.4 MB/s | |
| 66 | +| redact 100 KB sparse | 12.1 ms | 8.3 MB/s | |
| 67 | + |
| 68 | +### guardrail — `DataFogGuardrail` (LiteLLM), in-process |
| 69 | + |
| 70 | +| Operation | Median | |
| 71 | +| --- | --- | |
| 72 | +| single message redact | 43 µs | |
| 73 | +| `pre_call`, clean 2-message request | 118 µs | |
| 74 | +| `pre_call`, PII 2-message request (redact) | 222 µs | |
| 75 | +| event-loop dispatch (harness overhead, included above) | 27 µs | |
| 76 | + |
| 77 | +The PII-request figure includes litellm's own guardrail-intervention |
| 78 | +logging (~85 µs), not just detection. In a live proxy the event loop is |
| 79 | +already running, so the ~27 µs dispatch overhead in the two per-request |
| 80 | +rows is an artifact of benchmarking with `run_until_complete`. |
| 81 | + |
| 82 | +### hook — `datafog-hook` (Claude Code) |
| 83 | + |
| 84 | +| Operation | Median | |
| 85 | +| --- | --- | |
| 86 | +| end-to-end subprocess, incl. Python startup | 69–89 ms across runs (p90 ~77–96 ms) | |
| 87 | +| the scan itself (in-process) | 75 µs | |
| 88 | + |
| 89 | +The end-to-end cost is ~99.9% Python interpreter + import startup; the |
| 90 | +scan is microseconds. Cold (first-ever) invocations can spike to several |
| 91 | +hundred ms while the OS warms caches — the warmup runs exclude that, the |
| 92 | +p90 shows steady-state spread. |
| 93 | + |
| 94 | +### Comparisons — same payloads, same four entity types |
| 95 | + |
| 96 | +Both comparison targets are pinned to `en_core_web_sm`, the *smallest* |
| 97 | +English spaCy model — deliberately favorable to them (smaller model = |
| 98 | +faster inference), so these ratios are lower bounds. Model/engine setup |
| 99 | +time (~0.1–1 s) is excluded from per-scan figures. Presidio runs in-process |
| 100 | +via `AnalyzerEngine` with a documented `NlpEngineProvider` config — no |
| 101 | +sidecar, no network hop, which again flatters Presidio relative to its |
| 102 | +usual proxy deployment. |
| 103 | + |
| 104 | +| Payload | datafog regex | Presidio | spaCy NER | datafog vs Presidio | datafog vs spaCy | |
| 105 | +| --- | --- | --- | --- | --- | --- | |
| 106 | +| 1.2 KB dense | 222–251 µs | 22.8 ms | 28.6 ms | 103x | 114x | |
| 107 | +| 10 KB mixed | 1.2 ms | 174 ms | 171 ms | 148x | 140x | |
| 108 | +| 100 KB sparse | 11.9–12.2 ms | 2.02 s | 1.60 s | 170x | 131x | |
| 109 | + |
| 110 | +Speed is not the only axis: the runner records what each engine actually |
| 111 | +detected on each payload (see the `detected` metadata in the output). |
| 112 | +Recall differs in both directions — e.g. Presidio's email recognizer does |
| 113 | +not match addresses on the reserved `.invalid` domain, and context-based |
| 114 | +scoring drops some entities the regex engine finds; conversely NER models |
| 115 | +detect names and locations the regex engine does not target at all. The |
| 116 | +comparison is a fair *speed* comparison on identical inputs and entity |
| 117 | +types, not an accuracy study. |
| 118 | + |
| 119 | +## How these map to published claims |
| 120 | + |
| 121 | +- **"~31µs per request" (LiteLLM guardrail, project README / |
| 122 | + datafog.ai)** — not reproduced *as worded*. ~31–43 µs is the cost of |
| 123 | + scanning/redacting **one short message**; a full 2-message request |
| 124 | + through `async_pre_call_hook` is ~120 µs clean / ~220 µs with redaction |
| 125 | + and litellm logging. Still 100–1000x below a sidecar's network hop, but |
| 126 | + the claim should be reworded to per-message, e.g. *"~40µs per message — |
| 127 | + a request clears the guardrail in well under a millisecond"*. Reproduce: |
| 128 | + `python benchmarks/run.py --suite guardrail`. |
| 129 | +- **"~70ms per invocation including process startup" (Claude Code |
| 130 | + hook)** — reproduced: 69–89 ms median across runs on this machine |
| 131 | + (~70 ms idle, higher under load), dominated by interpreter startup. |
| 132 | + *"~70–90ms"* (or "under 100ms") is the defensible phrasing. Reproduce: `python benchmarks/run.py --suite hook`. |
| 133 | +- **"190x performance advantage" (PyPI tagline)** — not reproduced at |
| 134 | + 190x by this suite. Measured: 103–170x vs Presidio and 114–140x vs spaCy |
| 135 | + NER, varying by payload. The historical 190x came from a different |
| 136 | + (13.3 KB, entity-dense) document and engine configuration. Honest |
| 137 | + phrasings this suite supports: *"100x+ faster than NER-based PII |
| 138 | + detection"* or *"up to 170x faster than Presidio on identical |
| 139 | + payloads"*. Reproduce: `python benchmarks/run.py --suite spacy,presidio`. |
| 140 | + |
| 141 | +## Fairness notes / known limitations |
| 142 | + |
| 143 | +- Single-threaded, single-machine, synthetic payloads. Real chat traffic |
| 144 | + and real documents will differ; the payloads are designed to bracket the |
| 145 | + realistic range (dense, mixed, sparse). |
| 146 | +- `timeit` medians hide GC pauses and tail latency by design; the hook |
| 147 | + suite's p90 is the only tail-latency figure here. |
| 148 | +- The comparison suites measure the engines *as configured here* |
| 149 | + (documented model, in-process, setup excluded). Different Presidio |
| 150 | + recognizer registries or larger spaCy models change both speed and |
| 151 | + recall — in Presidio's favor on recall, against it on speed. |
0 commit comments