You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add first-class support for structured and semi-structured data (tables, CSV/TSV, Parquet, JSON, database result sets) to the Phileas engine. Today Phileas is focused on free text and PDF. The Phileas vs Presidio comparison work conceded that Presidio is currently ahead on structured and tabular data, and this is the gap that closes it.
Because Phileas is the engine underneath Philter, structured support belongs here first; the Philter-level exposure and high-throughput columnar I/O are tracked separately.
Why it matters
A large share of real PII lives in tables and columns, not prose: exports, database dumps, spreadsheets, and training datasets. Structured support is table stakes for AI training-data de-identification and removes a concrete reason a prospect would pick Presidio over Phileas. It also makes the comparison page honest in our favor rather than against us.
Acceptance Criteria
Phileas can ingest tabular and semi-structured input (at minimum CSV/TSV and JSON; Parquet and database result sets identified as follow-ups if not in scope)
Redaction is column-aware: a policy can target specific columns by name or position and apply a chosen filter and strategy per column
Content-based detection still works within cells (a column not explicitly typed is still scanned), so structure augments rather than replaces detection
Output preserves the input structure: redacting a CSV yields valid CSV, JSON yields valid JSON, with the same shape and column order
Referential consistency is maintained across rows and columns: consistent pseudonymization keeps keys and joins intact (the same value maps to the same replacement within the configured scope)
Column-name targeting and content detection can be combined in one policy (for example, always redact the email column, and also redact any email found elsewhere)
The policy schema is extended to express column and structure rules, and PhiSQL coverage is considered (or a follow-up issue filed)
Behavior is documented under the engine's docs/docs/, including examples for tabular input
Unit tests cover tabular redaction, structure preservation, and cross-row consistency; a gold-standard evaluation case is added
A parity plan for the Python and .NET ports is noted (engine-first in Java, ports to follow)
Relationship to Philter-level exposure and columnar I/O/throughput is recorded so they can build on this
Summary
Add first-class support for structured and semi-structured data (tables, CSV/TSV, Parquet, JSON, database result sets) to the Phileas engine. Today Phileas is focused on free text and PDF. The Phileas vs Presidio comparison work conceded that Presidio is currently ahead on structured and tabular data, and this is the gap that closes it.
Because Phileas is the engine underneath Philter, structured support belongs here first; the Philter-level exposure and high-throughput columnar I/O are tracked separately.
Why it matters
A large share of real PII lives in tables and columns, not prose: exports, database dumps, spreadsheets, and training datasets. Structured support is table stakes for AI training-data de-identification and removes a concrete reason a prospect would pick Presidio over Phileas. It also makes the comparison page honest in our favor rather than against us.
Acceptance Criteria
emailcolumn, and also redact any email found elsewhere)docs/docs/, including examples for tabular input