Skip to content

feat(core): add masked_connector_response with per-connector key allowlist #2050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shuklatushar226 wants to merge 15 commits into
base: main
Choose a base branch
from
Open

feat(core): add masked_connector_response with per-connector key allowlist #2050

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
6e2166f
feat(core): add unmasked_connector_response with per-connector key al…
shuklatushar226 Aug 4, 2026
b4431bc
test(core): remove unmasked_connector_response unit tests
shuklatushar226 Aug 4, 2026
cb64e5c
refactor(core): drop the response size cap and seed a single connector
shuklatushar226 Aug 4, 2026
d00b920
fix(core): resolve CI failures on unmasked_connector_response
shuklatushar226 Aug 4, 2026
0cd38c0
Merge remote-tracking branch 'origin/main' into feat/unmasked-connect…
shuklatushar226 Aug 4, 2026
6909fa0
chore: merge main, cover new call sites, trim comments
shuklatushar226 Aug 4, 2026
73ee3ff
perf(core): check the allowlist before the denylist in allowed()
shuklatushar226 Aug 4, 2026
95ecc2d
fix(core): handle newline-separated and BOM-prefixed connector responses
shuklatushar226 Aug 4, 2026
c8ac48f
fix(core): resolve the masking connector by name, not a ConnectorEnum…
shuklatushar226 Aug 6, 2026
709b850
refactor(core): rename unmasked_connector_response to masked_connecto…
shuklatushar226 Aug 6, 2026
d2eeb6a
fix(core): mask response paths that have no key to gate on
shuklatushar226 Aug 6, 2026
52de180
chore(config): default connector response masking off outside develop…
shuklatushar226 Aug 6, 2026
e5169b4
build(core): gate connector response masking behind a Cargo feature
shuklatushar226 Aug 7, 2026
4452706
fix(core): honour log_to_span across every gRPC-level log sink
shuklatushar226 Aug 7, 2026
2a98da5
feat(events): publish masked_connector_response on the connector event
shuklatushar226 Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 18 additions & 0 deletions config/development.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,3 +237,21 @@ enqueue_timeout_ms = 5000

[unmasked_headers]
keys = ["x-request-id","x-merchant-id","x-lineage-ids","x-reference-id","x-connector","x-tenant-id","x-shadow-mode","x-proxy-name"]

# Selectively-masked view of the connector response, exposed as
# `unmasked_connector_response`. Every key is preserved; a value is shown only if
# that connector's list below names it. Gated separately from
# `common.return_raw_connector_data`, so this can stay on where raw capture is off.
[connector_response_masking]
enabled = true

# Per-connector unmask lists, comma-separated and case-insensitive.
# A connector with no entry gets every value masked (keys still visible).
# Full PAN, CVV, expiry and credentials stay masked regardless of what is listed here.
# Truncated values (cardSummary, last4, cardBin) are not covered — name them if needed.
#
# Only one entry is seeded, as a worked example for testing. Add a line per
# connector as you need its fields visible; an unknown connector name here will
# abort startup rather than be ignored.
[connector_response_masking.connector_keys]
adyen = "pspreference,resultcode,merchantreference,refusalreason,eventcode,success"
18 changes: 18 additions & 0 deletions config/production.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,3 +178,21 @@ keys = ["x-request-id","x-merchant-id","x-lineage-ids","x-reference-id","x-conne
# Connectors that require an external API call for webhook source verification
[webhook_source_verification_call] # comma-separated list of connector names (case-insensitive)
connectors_with_webhook_source_verification_call = "paypal, truelayer"

# Selectively-masked view of the connector response, exposed as
# `unmasked_connector_response`. Every key is preserved; a value is shown only if
# that connector's list below names it. Gated separately from
# `common.return_raw_connector_data`, so this can stay on where raw capture is off.
[connector_response_masking]
enabled = true

@JeevaRamu0104 JeevaRamu0104 Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enabled = true here and in sandbox.toml, gated independently of return_raw_connector_data, means the masked view reaches callers from the first deploy — which makes the bypasses I've flagged on connector_response_masking.rs live rather than latent. Can we default it false until those are closed and covered? The struct Default is already false, so it's config-only.

The comment block above also promises PAN/CVV/expiry stay masked regardless — true for keyed fields, not for the unkeyed paths.

@shuklatushar226 shuklatushar226 Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taking it. enabled = false in both sandbox.toml and production.toml, development.toml stays true. Config-only, as you note — the struct Default was already false.

I'd have argued the containment case is weaker now that the bypasses are closed in this same PR, but that's the wrong way round: the reason to ship it off is that turning it on returns connector response bytes to the caller, so a deployment should opt in once it has chosen its key lists — independent of whether any particular bypass is open. Added that as the comment above the flag.

On the comment block — you're right that it was a claim about keyed fields dressed up as an unconditional one. Rewritten to say what naming a key actually grants, and to name the unkeyed case explicitly rather than leave it implied:

# Naming a key here reveals only that key's own value: an object below it is re-decided key by
# key, and an array below it stays masked, since its elements have no key you could name.
# A key whose name looks like a full PAN, CVV, expiry or credential stays masked even if listed.
# Truncated values (cardSummary, last4, cardBin) are not covered — name them if needed.
# A body that is not JSON, XML or form-encoded has no keys to gate on, so it is replaced
# wholesale by a stub carrying only its size.

Applied to all three config files. Fixed in 52de180.


# Per-connector unmask lists, comma-separated and case-insensitive.
# A connector with no entry gets every value masked (keys still visible).
# Full PAN, CVV, expiry and credentials stay masked regardless of what is listed here.
# Truncated values (cardSummary, last4, cardBin) are not covered — name them if needed.
#
# Only one entry is seeded, as a worked example for testing. Add a line per
# connector as you need its fields visible; an unknown connector name here will
# abort startup rather than be ignored.
[connector_response_masking.connector_keys]
adyen = "pspreference,resultcode,merchantreference,refusalreason,eventcode,success"
18 changes: 18 additions & 0 deletions config/sandbox.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,3 +179,21 @@ keys = ["x-request-id","x-merchant-id","x-lineage-ids","x-reference-id","x-conne
# Connectors that require an external API call for webhook source verification
[webhook_source_verification_call] # comma-separated list of connector names (case-insensitive)
connectors_with_webhook_source_verification_call = "paypal, truelayer"

# Selectively-masked view of the connector response, exposed as
# `unmasked_connector_response`. Every key is preserved; a value is shown only if
# that connector's list below names it. Gated separately from
# `common.return_raw_connector_data`, so this can stay on where raw capture is off.
[connector_response_masking]
enabled = true

# Per-connector unmask lists, comma-separated and case-insensitive.
# A connector with no entry gets every value masked (keys still visible).
# Full PAN, CVV, expiry and credentials stay masked regardless of what is listed here.
# Truncated values (cardSummary, last4, cardBin) are not covered — name them if needed.
#
# Only one entry is seeded, as a worked example for testing. Add a line per
# connector as you need its fields visible; an unknown connector name here will
# abort startup rather than be ignored.
[connector_response_masking.connector_keys]
adyen = "pspreference,resultcode,merchantreference,refusalreason,eventcode,success"
71 changes: 71 additions & 0 deletions crates/common/external-services/src/service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,48 @@ fn flow_status_label(flow_status: &domain_types::router_data::FlowStatus) -> Str
}
}

/// Build the selectively-masked view of a connector response and stash it on the flow data.
///
/// Reads the untouched response bytes, so this works whether or not `raw_connector_response`
/// is being captured — the safe view can be on in production with raw capture off.
fn record_unmasked_connector_response<ResourceCommonData>(
resource_common_data: &mut ResourceCommonData,
body: &Response,
connector_name: &str,
config: &domain_types::connector_response_masking::ConnectorResponseMaskingConfig,
) where
ResourceCommonData: RawConnectorRequestResponse,
{
use std::str::FromStr;

// `connector_name` came from `ConnectorEnum::get_connector_name()`, and the enum derives
// `EnumString` with snake_case, so this always round-trips.
let Ok(connector) = domain_types::connector_types::ConnectorEnum::from_str(connector_name)
else {
return;
};

@AmitsinghTanwar007 AmitsinghTanwar007 Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add debug log here

@shuklatushar226 shuklatushar226 Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a log here would have papered over a live bug — the comment above this branch is wrong.

connector_name doesn't come from ConnectorEnum. It comes from ConnectorVariant::get_connector_name() (connector_types.rs:403), which stringifies five enums. Three names have no ConnectorEnum counterpart, so from_str fails and the function returns before populating anything:

Name Source enum
Interpayments SurchargeConnectorEnum
Deutschebank PayoutConnectorEnum
Plaid AuthenticatorConnectorEnum

The branch is reachable — set_unmasked_connector_response is implemented for surcharge_types.rs, payouts_types.rs and merchant_authentication_flow_data.rs. And the breakage was inconsistent rather than uniform: Loonio, Paypal, Itaubank, Worldpayxml, Cybersource (payout) and Kount (FRM) collide with ConnectorEnum names and worked fine, so behaviour differed per connector inside the same flow with no signal.

Same root cause hit config load: deserialize_connector_keys also parsed keys with ConnectorEnum::from_str, so deutschebank = "..." in a TOML aborted startup for a perfectly real connector.

Rather than logging the silent exit, I removed the re-parse. Ingress already validates the name per flow family (connector_variant_from_metadata, metadata.rs:113 — x-connector against ConnectorEnum, x-payout-connector against PayoutConnectorEnum, and so on), so the string arriving here is already canonical. It's now used directly as a lookup key, and config-load validation accepts a name that any of the five enums recognises. A lookup miss degrades to the empty allowlist — which is already the correct default for an unconfigured connector — so there's no failure branch left to log.

Verified: with interpayments/deutschebank/plaid as config keys the server boots; paysafee still aborts with unknown connector `paysafee` .


// By name: this HeaderMap is reqwest 0.11 (http 0.2), not the http 1.x in scope.
let content_type = body
.headers
.as_ref()
.and_then(|headers| headers.get("content-type"))
.and_then(|value| value.to_str().ok());

let masked = domain_types::connector_response_masking::mask_connector_response(
&body.response,
content_type,
&connector,
config,
);

if let Some(masked) = masked.as_deref() {
tracing::Span::current().record("response.unmasked_body", tracing::field::display(masked));
}

resource_common_data.set_unmasked_connector_response(masked);
}

/// Handles the connector response, processing both successful and error responses
#[allow(clippy::too_many_arguments)]
pub fn handle_connector_response<F, ResourceCommonData, Req, Resp>(
Expand Down Expand Up @@ -367,6 +409,18 @@ where
.set_connector_response_headers(body.headers.clone());
}

// Independent of `return_raw_connector_data`: this view is already sanitized.
if let Some(params) =
event_params.filter(|p| p.connector_response_masking.enabled)
{
record_unmasked_connector_response(
&mut updated_router_data.resource_common_data,
&body,
params.connector_name,
params.connector_response_masking,
);
}

let handle_response_result = connector.handle_response_v2(
&updated_router_data,
event.as_deref_mut(),
Expand Down Expand Up @@ -424,6 +478,18 @@ where
.set_connector_response_headers(body.headers.clone());
}

// A 4xx/5xx body is exactly when the masked view is most useful.
if let Some(params) =
event_params.filter(|p| p.connector_response_masking.enabled)
{
record_unmasked_connector_response(
&mut updated_router_data.resource_common_data,
&body,
params.connector_name,
params.connector_response_masking,
);
}

let error_response = match body.status_code {
500..=511 => connector.get_5xx_error_response(
body.clone(),
Expand Down Expand Up @@ -533,6 +599,10 @@ pub struct EventProcessingParams<'a> {
pub tenant_id: &'a str,
pub merchant_id: &'a str,
pub return_raw_connector_data: bool,
/// Per-connector key lists driving `unmasked_connector_response`. Gated by its own
/// `enabled` flag, deliberately independent of `return_raw_connector_data`.
pub connector_response_masking:
&'a domain_types::connector_response_masking::ConnectorResponseMaskingConfig,
pub connector_latency: ConnectorLatencyTracker,
}

Expand All @@ -546,6 +616,7 @@ pub struct EventProcessingParams<'a> {
request.url = Empty,
request.method = Empty,
response.body = Empty,
response.unmasked_body = Empty,
response.headers = Empty,
response.error_message = Empty,
response.status_code = Empty,
Expand Down
7 changes: 7 additions & 0 deletions crates/common/ucs_env/src/configs.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ use common_utils::{
SuperpositionConfig,
};
use domain_types::{
connector_response_masking::{
ConnectorResponseMaskingConfig, ConnectorResponseMaskingConfigPatch,
},
connector_types::ConnectorEnum,
types::{Connectors, ConnectorsPatch, ProxyConfig, ProxyConfigPatch},
};
Expand All @@ -34,6 +37,10 @@ pub struct Config {
pub lineage: LineageConfig,
#[serde(default)]
pub unmasked_headers: HeaderMaskingConfig,
/// Per-connector key lists controlling which response values stay visible in
/// `unmasked_connector_response`.
#[serde(default)]
pub connector_response_masking: ConnectorResponseMaskingConfig,
#[serde(default)]
pub test: TestConfig,
#[serde(default)]
Expand Down
2 changes: 2 additions & 0 deletions crates/grpc-server/grpc-server/src/server/disputes.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ impl DisputeService for Disputes {
tenant_id: &tenant_id,
merchant_id: merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency,
};

Expand Down Expand Up @@ -424,6 +425,7 @@ impl DisputeService for Disputes {
tenant_id: &tenant_id,
merchant_id: merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency,
};

Expand Down
7 changes: 7 additions & 0 deletions crates/grpc-server/grpc-server/src/server/events.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -504,6 +504,7 @@ impl EventServiceImpl {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -613,6 +614,7 @@ impl EventServiceImpl {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -722,6 +724,7 @@ impl EventServiceImpl {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -828,6 +831,7 @@ impl EventServiceImpl {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -934,6 +938,7 @@ impl EventServiceImpl {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -975,6 +980,7 @@ async fn verify_webhook_source_external(
connectors: config.connectors.clone(),
connector_request_reference_id: format!("webhook_verify_{}", metadata_payload.request_id),
raw_connector_response: None,
unmasked_connector_response: None,
raw_connector_request: None,
connector_response_headers: None,
};
Expand Down Expand Up @@ -1029,6 +1035,7 @@ async fn verify_webhook_source_external(
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down
13 changes: 13 additions & 0 deletions crates/grpc-server/grpc-server/src/server/payments.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,7 @@ impl Payments {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -715,6 +716,7 @@ impl Payments {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -1129,6 +1131,7 @@ impl PaymentService for Payments {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -2607,6 +2610,7 @@ impl PaymentMethod {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -2743,6 +2747,7 @@ impl PaymentMethod {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
runtime_metadata: &config.runtime_metadata,
};
Expand Down Expand Up @@ -2860,6 +2865,7 @@ impl MerchantAuthentication {
tenant_id: event_params.tenant_id,
merchant_id: event_params.merchant_id,
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: event_params.connector_latency.clone(),
};

Expand Down Expand Up @@ -3001,6 +3007,7 @@ impl MerchantAuthentication {
tenant_id: event_params.tenant_id,
merchant_id: event_params.merchant_id,
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: event_params.connector_latency.clone(),
};

Expand Down Expand Up @@ -3580,6 +3587,7 @@ impl RecurringPaymentService for RecurringPayments {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -3852,6 +3860,9 @@ pub fn generate_mandate_revoke_response(
let raw_connector_response = router_data_v2
.resource_common_data
.get_raw_connector_response();
let unmasked_connector_response = router_data_v2
.resource_common_data
.get_unmasked_connector_response();
let raw_connector_request = router_data_v2
.resource_common_data
.get_raw_connector_request();
Expand Down Expand Up @@ -3881,6 +3892,7 @@ pub fn generate_mandate_revoke_response(
network_transaction_id: None,
merchant_revoke_id: None,
raw_connector_response,
unmasked_connector_response,
raw_connector_request,
}),
Err(e) => Ok(RecurringPaymentServiceRevokeResponse {
Expand All @@ -3901,6 +3913,7 @@ pub fn generate_mandate_revoke_response(
network_transaction_id: None,
merchant_revoke_id: e.connector_transaction_id,
raw_connector_response,
unmasked_connector_response,
raw_connector_request,
}),
}
Expand Down
2 changes: 2 additions & 0 deletions crates/grpc-server/grpc-server/src/utils.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,7 @@ macro_rules! implement_connector_operation {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};

Expand Down Expand Up @@ -1070,6 +1071,7 @@ macro_rules! implement_connector_operation {
tenant_id: &metadata_payload.tenant_id,
merchant_id: metadata_payload.merchant_id.as_str(),
return_raw_connector_data: config.common.return_raw_connector_data,
connector_response_masking: &config.connector_response_masking,
connector_latency: metadata_payload.connector_latency.clone(),
};
let call_connector_action = connector_integration.get_call_connector_action();
Expand Down
Loading
Loading