Skip to content

Add structured and semi-structured data support (tables, CSV, JSON) #311

Description

@jzonthemtn

Summary

Add first-class support for structured and semi-structured data (tables, CSV/TSV, Parquet, JSON, database result sets) to the Phileas engine. Today Phileas is focused on free text and PDF. The Phileas vs Presidio comparison work conceded that Presidio is currently ahead on structured and tabular data, and this is the gap that closes it.

Because Phileas is the engine underneath Philter, structured support belongs here first; the Philter-level exposure and high-throughput columnar I/O are tracked separately.

Why it matters

A large share of real PII lives in tables and columns, not prose: exports, database dumps, spreadsheets, and training datasets. Structured support is table stakes for AI training-data de-identification and removes a concrete reason a prospect would pick Presidio over Phileas. It also makes the comparison page honest in our favor rather than against us.

Acceptance Criteria

  • Phileas can ingest tabular and semi-structured input (at minimum CSV/TSV and JSON; Parquet and database result sets identified as follow-ups if not in scope)
  • Redaction is column-aware: a policy can target specific columns by name or position and apply a chosen filter and strategy per column
  • Content-based detection still works within cells (a column not explicitly typed is still scanned), so structure augments rather than replaces detection
  • Output preserves the input structure: redacting a CSV yields valid CSV, JSON yields valid JSON, with the same shape and column order
  • Referential consistency is maintained across rows and columns: consistent pseudonymization keeps keys and joins intact (the same value maps to the same replacement within the configured scope)
  • Column-name targeting and content detection can be combined in one policy (for example, always redact the email column, and also redact any email found elsewhere)
  • The policy schema is extended to express column and structure rules, and PhiSQL coverage is considered (or a follow-up issue filed)
  • Behavior is documented under the engine's docs/docs/, including examples for tabular input
  • Unit tests cover tabular redaction, structure preservation, and cross-row consistency; a gold-standard evaluation case is added
  • A parity plan for the Python and .NET ports is noted (engine-first in Java, ports to follow)
  • Relationship to Philter-level exposure and columnar I/O/throughput is recorded so they can build on this

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions