fix(worker): decrypt CRM regulatory_fields + related_parties PII - #318
Merged
Conversation
The plugin_crm extraction path decrypted contact, banking_details, legal_person and natural_person fields but not regulatory_fields or related_parties, so participant_document (a CNPJ) and related_parties[].document were projected as ciphertext into the snapshot -- the root cause of CCS-0011 in br-ccs batch generation. Add decryptRegulatoryFields and decryptRelatedParties, mirroring the existing per-object decryptors and reusing the same crypto instance; wire both into decryptNestedFields. Verified against the live dev-st CRM (KMS_VENDOR=none): all values are legacy AES-GCM (no tink: format). Fail-closed on undecryptable values; name/role/dates stay plaintext. X-Lerian-Ref: 0x1
Contributor
Lerian Library Version Check
0 outdated | 4 current | 1 skipped | 0 unknown |
📝 WalkthroughWalkthroughPlugin CRM nested decryption now handles ChangesCRM nested decryption
Sequence Diagram(s)sequenceDiagram
participant CRMRecord
participant decryptNestedFields
participant decryptRegulatoryFields
participant decryptRelatedParties
participant decryptFieldValue
CRMRecord->>decryptNestedFields: provide nested fields
decryptNestedFields->>decryptRegulatoryFields: process regulatory_fields
decryptRegulatoryFields->>decryptFieldValue: decrypt participant_document
decryptNestedFields->>decryptRelatedParties: process related_parties
decryptRelatedParties->>decryptFieldValue: decrypt each document
decryptNestedFields-->>CRMRecord: return updated record or contextual error
Possibly related PRs
✨ Finishing Touches✨ Simplify code
Comment |
lerian-studio
added
component: worker
Contributor
🔍 PR Validation Summary✅ PR Mergeable — no blocking failures
|
Contributor
🔒 Security Scan Results —
|
| Stage | Status | Blocking? |
|---|---|---|
| Filesystem Scan | ✅ Clean | — |
| Docker Image Scan | ✅ Clean | — |
| Docker Hub Health Score | ✅ Clean | — |
| Pre-release Version Check | 🟡 No (advisory) |
Trivy
Filesystem Scan
✅ No vulnerabilities or secrets found.
Docker Image Scan
✅ No vulnerabilities found.
Docker Hub Health Score Compliance
✅ Policies — 4/4 met
| Policy | Status |
|---|---|
| Default non-root user | ✅ Passed |
| No fixable critical/high CVEs | ✅ Passed |
| No high-profile vulnerabilities | ✅ Passed |
| No AGPL v3 licenses | ✅ Passed |
Pre-release Version Check
🚫 Found 1 unstable version pin(s). Only stable releases (x.y.z) and SHA-based pins are allowed.
| File | Line | Content |
|---|---|---|
./go.mod |
68 | github.com/LerianStudio/fetcher/pkg/engine v1.1.0-beta.1 |
Replace pre-release suffixes (
-alpha,-beta,-rc,-dev, etc.) with stable releases.
Contributor
📊 Unit Test Coverage Report:
|
| Metric | Value |
|---|---|
| Overall Coverage | 89.7% ✅ PASS |
| Threshold | 80% |
Coverage by Package
| Package | Coverage |
|---|---|
github.com/LerianStudio/fetcher/v2/components/manager/cmd/huma-spec |
33.4% |
github.com/LerianStudio/fetcher/v2/components/manager/internal/adapters/cache |
100.0% |
github.com/LerianStudio/fetcher/v2/components/manager/internal/adapters/http/in |
89.4% |
github.com/LerianStudio/fetcher/v2/components/manager/internal/services/command |
92.8% |
github.com/LerianStudio/fetcher/v2/components/manager/internal/services/query |
96.4% |
Generated by Go PR Analysis workflow
Contributor
📊 Unit Test Coverage Report:
|
| Metric | Value |
|---|---|
| Overall Coverage | 87.4% ✅ PASS |
| Threshold | 80% |
Coverage by Package
| Package | Coverage |
|---|---|
github.com/LerianStudio/fetcher/v2/components/worker/internal/services |
90.0% |
Generated by Go PR Analysis workflow
Contributor
🔒 Security Scan Results —
|
| Stage | Status | Blocking? |
|---|---|---|
| Filesystem Scan | ✅ Clean | — |
| Docker Image Scan | ✅ Clean | — |
| Docker Hub Health Score | ✅ Clean | — |
| Pre-release Version Check | 🟡 No (advisory) |
Trivy
Filesystem Scan
✅ No vulnerabilities or secrets found.
Docker Image Scan
✅ No vulnerabilities found.
Docker Hub Health Score Compliance
✅ Policies — 4/4 met
| Policy | Status |
|---|---|
| Default non-root user | ✅ Passed |
| No fixable critical/high CVEs | ✅ Passed |
| No high-profile vulnerabilities | ✅ Passed |
| No AGPL v3 licenses | ✅ Passed |
Pre-release Version Check
🚫 Found 1 unstable version pin(s). Only stable releases (x.y.z) and SHA-based pins are allowed.
| File | Line | Content |
|---|---|---|
./go.mod |
68 | github.com/LerianStudio/fetcher/pkg/engine v1.1.0-beta.1 |
Replace pre-release suffixes (
-alpha,-beta,-rc,-dev, etc.) with stable releases.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/worker/internal/services/extract_crm_data.go`:
- Around line 589-617: Extract the repeated map-container field decryption flow
into a shared helper, then update decryptRegulatoryFields and
decryptBankingDetailsFields (plus the existing contact sibling if applicable) to
use it. The helper should type-assert the nested map, iterate the supplied field
names, call decryptFieldValue, wrap errors with the nested field context, and
write the updated map back while preserving current behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 45bbc56c-38de-4cd9-a226-42b9cf35795c
📒 Files selected for processing (2)
components/worker/internal/services/extract_crm_data.gocomponents/worker/internal/services/extract_crm_data_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds decryption for two CRM PII fields on the
plugin_crmextraction path (Worker) that were previously projected into the snapshot as ciphertext:regulatory_fields.participant_document(a CNPJ) — newdecryptRegulatoryFieldsrelated_parties[].document— newdecryptRelatedParties(slice-shaped)Both mirror the existing per-object decryptors, reuse the same
*libCrypto.Cryptoinstance, and are wired intodecryptNestedFields.name/role/ dates stay plaintext.Why: this ciphertext-in-snapshot gap is the root cause of CCS-0011 in br-ccs batch generation — br-ccs received an encrypted CNPJ where it expected a plaintext value.
plugin_crmbypassespkg/engineand runs the legacyextract_crm_data.gopath, so the fix is local to that file.Verified against the live dev-st CRM (
midaz-crm,KMS_VENDOR=none): 15/15participant_documentand 15/15related_parties[].documentare legacy AES-GCM (0tink:-format), decryptable with the key the Fetcher already holds. Fail-closed on undecryptable values; errors name the field/index, never the raw value (no PII in logs/errors).Type of Change
fix: Bug fixBreaking Changes
None.
Testing
make testpassesmake lintpassesmake secpasses (full monorepo, exit 0)make test-int— n/a (no integration paths exercised by this change)Also green:
go build ./...,gofmt,-raceon the changed package; coverage 87.3%.make vulncheck(trivy) runs in CI.Test evidence: Gate 0 TDD RED→GREEN per task; Gate 8 review — Epic 1.1 (regulatory_fields) 9/9 reviewers, Epic 2.1 (related_parties) focused 5-reviewer pool; cycle-close
/code-review(high) + whole-diff HIGH-effort +/go-comment-reviewer.Architectural Checklist
panic()in production paths%wtime.Now().UTC()(n/a — no timestamps added)internal/bootstrap(n/a — Worker decrypt path, not HTTP handlers)Related Issues
Root cause of CCS-0011 in br-ccs batch generation (no GitHub issue tracked; see fetcher-033).
Follow-ups (not in this PR)
decryptPluginCRMDataloop (one undecryptable record currently fails the whole collection — intentional fail-closed for now).tink:-format handling (other CRM orgs use Tink in the write path).newTestCrypto(t)helper; split the oversizedextract_crm_data_test.go.