Skip to content

RETENTION_AND_PII.md

Latest commit

 

History

History
44 lines (33 loc) · 1.97 KB

RETENTION_AND_PII.md

File metadata and controls

44 lines (33 loc) · 1.97 KB
Raw

Open-web retention + PII minimization (VAL-HARDEN-011 / 012)

Retention window (concrete)

TTL: 30 days (RETENTION_TTL_SECONDS = 30 * 24 * 3600).

Task rows, canary rows, and scrape-attempt records (including raw scraped content and any residual PII-bearing payloads) are retained for 30 days from created_at, then deleted by the retention purge. In-window rows remain for scoring, dispute, and audit.

  • Code constant: relay.policy.RETENTION_TTL_DAYS / RETENTION_TTL_SECONDS
  • HTTP: POST /internal/v1/policy/retention/purge (Bearer + slug)
  • Published on: GET /internal/v1/policy → retention_ttl_days, retention_policy

There is no indefinite retention of scrape content / PII.

PII minimization

Pre-seal screening rejects targets that solicit PII in the URL (query keys / SSN patterns). After a scrape lands, relay.policy.redact_pii runs on every payload written to scrape_attempts so synthetic and accidental markers are masked before storage:

Marker class Replacement
email [REDACTED_EMAIL]
phone [REDACTED_PHONE]
SSN ###-##-#### [REDACTED_SSN]
national-ID shape [REDACTED_NATIONAL_ID]

A grep of scrape_attempts rows and any relay-served artifact after intake must find no un-redacted synthetic marker (VAL-HARDEN-011).

Policy gate authority (VAL-HARDEN-009 / 010 / 013)

  • robots.txt-disallowed and ToS-restricted paths are rejected pre-seal with reasons robots_disallowed / tos_restricted and never create a tasks row.
  • Deny-list matches run on a canonicalized URL (case / trailing-dot / default port / punycode / confusable / IP-literal / redirect-to-denied) so those evasions cannot smuggle a denied host.
  • A miner-substituted URL for a task_id is rejected at L0 as url_not_policy_allowed or request_hash_mismatch; only relay-gated targets enter the sealed work-unit universe.