feat(observability): add typed connector request/response fields by AmitsinghTanwar007 · Pull Request #2036 · juspay/hyperswitch-prism

AmitsinghTanwar007 · 2026-08-03T09:29:48Z

Summary

Adds typed_connector_request and typed_connector_response fields across the entire connector service stack to provide structured, domain-typed observability data separate from raw wire-format payloads.

Problem

Currently, only raw_connector_request / raw_connector_response exist — these carry the literal HTTP bytes sent/received. For debugging and analytics, we also need the typed representation (the Rust domain struct, masked and serialized) so consumers can reason about connector data in a connector-agnostic way without parsing raw JSON/XML from different providers.

What changed

Proto definitions (payment.proto, frm.proto):

Added typed_connector_request and typed_connector_response optional fields to all response messages (Authorize, Get, Void, Capture, Refund, CreateOrder, SetupRecurring, MandateRevoke, FRM pre/post risk check, etc.)

Domain types (connector_types.rs, frm_types.rs, surcharge_types.rs, payouts_types.rs, merchant_authentication_flow_data.rs):

Added typed_connector_request / typed_connector_response fields to all flow data structs (PaymentFlowData, RefundFlowData, DisputeFlowData, FrmFlowData, SurchargeFlowData, VerifyWebhookSourceFlowData, RefreshPaymentMethodFlowData, PayoutFlowData)
Extended the RawConnectorRequestResponse trait with get/set_typed_connector_request and get/set_typed_connector_response methods
Implemented the new trait methods for all flow data structs

Connector integration trait (connector_integration_v2.rs):

Added get_typed_connector_request() default method returning None
Wired it into build_request_v2() via set_typed_connector_request() on the request builder

Request infrastructure (request.rs):

Added typed_connector_request field to Request and RequestBuilder
Added set_typed_connector_request() builder method

Connector macros (macros.rs):

Added serialize_typed_connector_payload() utility — masked serialization of typed structs
Added expand_fn_get_typed_connector_request! macro — generates get_typed_connector_request() for each flow
Modified expand_fn_handle_response! to capture and set typed_connector_response on the flow data
Wired both macros into all 4 macro_connector_implementation! variants

Connector-specific (bamboraapac/transformers.rs):

Added Serialize derive to request structs that were missing it (required for typed serialization)

Service layer (service.rs):

Set typed_connector_request on flow data from the request object after connector call

gRPC response generators (types.rs, frm/types.rs, payments.rs):

All generate_*_response() functions now extract and pass typed fields to proto responses

Server layer (events.rs):

Added typed_connector_request/response: None to VerifyWebhookSourceFlowData construction

Key design principle: raw ≠ typed

raw: the literal bytes off the wire (HTTP body as-is)
typed: the domain Rust struct, masked-serialized to JSON — these MUST be different values

Test plan

cargo check passes across all crates
Existing connector tests (razorpay, calida, bamboraapac) compile with new fields
Verify typed fields are None by default for connectors not using the macro system
Verify typed fields are populated for macro-based connectors (e.g., bamboraapac)
Confirm raw and typed values differ when both are populated

gopikrishna000 · 2026-08-04T09:19:17Z

    }
 }

 pub fn generate_mandate_revoke_response(


why was these functions added in this file instead of type.rs
not related to this pr though, but how did we missed

Don't know but i think in future we need to write clippy rules so that the structure is maintained

JeevaRamu0104 · 2026-08-06T06:40:12Z

 [server]
 host = "0.0.0.0"
-port = 8000
+port = 8003


revert these changes.

JeevaRamu0104 · 2026-08-06T20:14:24Z

+            fn get_typed_connector_request(
+                &self,
+                req: &RouterDataV2<$flow, $resource_common_data, $request, $response>,
+            ) -> Option<String>
+            {
+                let bridge = self.[< $flow:snake >];
+                let input_data = [< $connector RouterData >] {
+                    connector: self.to_owned(),
+                    router_data: req.clone()
+                };
+                bridge
+                    .request_body(input_data)
+                    .ok()
+                    .and_then(|request| crate::connectors::macros::serialize_typed_connector_payload(
+                        &request,
+                        "typed_connector_request",
+                    ))
+            }


[S1] The typed request is reconstructed, not captured — it can differ from what was actually sent.

This re-invokes bridge.request_body(req.clone()) a second time, independently of get_request_body. For connectors with per-call non-determinism the two diverge:

airwallex/transformers.rs:1264 mints a fresh uuid::Uuid::new_v4() per call, with a comment stating reuse is rejected as duplicate_request.

nuvei/transformers.rs:82-86 derives a checksum from date_time::now().

The observability record will show a request_id/timestamp/signature that was never on the wire — misleading for exactly the debugging use case this PR exists for.

It also doubles request-transformation cost, including a full RouterDataV2 clone, on the payment hot path.

Capture the typed value once inside get_request_body and thread it through instead.

gopikrishna000 · 2026-08-07T10:29:05Z

+            .masked_serialize_inner()
+            .map(|(v, _)| v)
+            .unwrap_or_else(|| match request {
+                RequestContent::FormData(_) => json!({"request_type": "FORM_DATA"}),


for FormData and RawByte, can we still pass the another parameter of typed struct similar to other types

I will handle this change in later PR as this may require change in L3 layer which i need to go through,
already the pr has many changes.

gopikrishna000 · 2026-08-07T15:52:37Z

                    }

-                    let error_response = match body.status_code {
+                    let (error_response, typed_response) = match body.status_code {


for 4xx or 5xx, since we added the new field inside ConnectorError proto, so should adding the typed_response field fiedl inside ErrorResponse struct be sufficient?
Here we setting inside resource_common_data, and taking back to set ErrorResponse

gopikrishna000 · 2026-08-07T16:02:43Z


            let response_body = bridge.response(response_bytes, res.status_code)?;
-            event_builder.map(|i| i.set_connector_response(&response_body));
+            // Serialize once: masked Value for event logging, String for typed_connector_response


bro, actually for event also we need string type only., we are doing stringy at downstream and that dont handle all cases like array, float etc
So we can actually send string only to events as well, so no need of this divergence
(Both request and response)

gopikrishna000 · 2026-08-07T17:08:47Z

+/// logging via `set_connector_response`) and the stringified form (for
+/// `typed_connector_response`).  This avoids calling `masked_serialize` twice.
+pub(crate) fn masked_serialize_connector_response<T: serde::Serialize>(
+    payload: &T,


this fucntion is similar to from_masked_optional that we have as impl for MaskedSerdeValue type?

gopikrishna000 · 2026-08-07T17:08:52Z

+                crate::connectors::macros::masked_serialize_connector_response(&response_body);
+            if let Some(evt) = event_builder {
+                if let Some((ref value, _)) = serialized {
+                    evt.set_connector_response(value);


also this set_connector_response, already does MaskedSerdeValue::from_masked_optional

gopikrishna000 · 2026-08-07T17:14:34Z

+///
+/// Replaces the 12-line boilerplate block in every manual `handle_response_v2`.
+#[macro_export]
+macro_rules! set_typed_response {


name is set_typed_response
But we do more than that
we can name better

gopikrishna000 · 2026-08-07T17:17:10Z

    Ok(())
 }

+pub(crate) fn serialize_typed_connector_payload<T: serde::Serialize>(


it is similar to from_masked_optional that we have.
It comes out of box if we use typed MaskedSerdeValue instead of serde::json, so mistaken unmasked value also not possible

gopikrishna000 · 2026-08-07T17:25:25Z


        match self {
-            Self::ConnectorErrorResponse(error_response) => match error_response.status_code {
+            Self::ConnectorErrorResponse { error_response, .. } => match error_response.status_code


having raw_connector_response field inside error_response struct would be cleaner?

…serialization - Add typed_connector_response field directly to ErrorResponse struct - Serialize typed connector payload inside build_error_response, eliminating double-parsing on the error path - Remove get_typed_connector_error_response from ConnectorCommon trait - Remove resource_common_data round-trip for typed error responses in service.rs - Add typed_connector_request/response fields to proto ConnectorError message - Update all ~90 connector build_error_response impls and ~77 transformer ErrorResponse initializers

…dSerdeValue for typed payload serialization

…uest into ErrorResponse Simplify ConnectorErrorResponse from struct variant with 5 fields to a single-field tuple variant ConnectorErrorResponse(Box<ErrorResponse>). All raw/typed connector fields now live inside ErrorResponse, matching the typed_connector_response pattern from the previous commit.

…th request content get_request_body now returns ConnectorRequestData which pairs RequestContent with an Option<MaskedSerdeValue> typed payload. This ensures the typed connector request is serialized before conversion to FormData/RawBytes (which would otherwise lose the typed information), and propagates it through all build_request_v2 paths including manual overrides.

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

No longer needed — ConnectorRequestData guarantees typed request serialization at the macro level, and response/error typed serialization is handled by the macro-generated code.

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

gopikrishna000 · 2026-08-10T21:22:13Z

+        })
+        .map_err(|e| {
+            e.change_context($crate::ConnectorError::ResponseHandlingFailed {
+                context: Default::default(),


can we update default context

also why map_Err? already RouterDataV2::try_from has Report ConnectorError?

AmitsinghTanwar007 requested review from a team as code owners August 3, 2026 09:29

gopikrishna000 reviewed Aug 4, 2026

View reviewed changes

JeevaRamu0104 reviewed Aug 6, 2026

View reviewed changes

AmitsinghTanwar007 closed this Aug 7, 2026

AmitsinghTanwar007 reopened this Aug 7, 2026

gopikrishna000 reviewed Aug 7, 2026

View reviewed changes

AmitsinghTanwar007 force-pushed the typed-message branch from f7bcea5 to f929177 Compare August 7, 2026 20:57

AmitsinghTanwar007 and others added 8 commits August 8, 2026 10:00

fix: resolve clippy large_enum_variant, missing fields, and use Maske…

cf48fe7

…dSerdeValue for typed payload serialization

fix: box large enum variant in field-probe ProbeAttemptResult

1202fe9

fix: revert unintended development.toml change

cc1e29d

chore: auto-fix formatting and generated code

e81cc7b

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

chore: remove typed_observability_lint test module

278b310

No longer needed — ConnectorRequestData guarantees typed request serialization at the macro level, and response/error typed serialization is handled by the macro-generated code.

chore: auto-fix formatting and generated code

76f7169

Auto-applied by CI: - cargo +nightly fmt --all - make -C sdk generate (if applicable) - make docs (if applicable) This commit was automatically generated by GitHub Actions.

AmitsinghTanwar007 force-pushed the typed-message branch from 28ba1d5 to 76f7169 Compare August 10, 2026 07:21

Merge branch 'main' into typed-message

5aa8ed9

refactor: simplify typed connector serialization helpers

8b965f8

gopikrishna000 reviewed Aug 10, 2026

View reviewed changes

fix: preserve response handling context

ad12261

Uh oh!

Conversation

AmitsinghTanwar007 commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

What changed

Key design principle: raw ≠ typed

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AmitsinghTanwar007 commented Aug 3, 2026 •

edited

Loading