Problem propagating trace id to grpc service

[RobWalker]

I have an aspnet core app with a front end HTTP server and a second gRPC server with an AWS ALB between them.

I have XRay tracing working on both servers, but can not get the calls to the gRPC server to be linked to the originating call on the frontend server.

When making the gRPC call I am grabbing the current trace id and adding it to a X-Amzn-Trace-Id header. This is set as something like:

Root=1-6200946f-1a21642c074cf3445e4e537a;Parent=;Sampled=1

I have custom middleware in the gRPC server which is checking to see if this header is present, which it is, though as:

Self=1-6200946f-241125367dc7e44531ef72f8;Root=1-6200946f-1a21642c074cf3445e4e537a;Parent=;Sampled=1

So I suspect the ALB is modifying it?

However, within my custom middleware there is already an XRay recorder instance set up, and it has an unrelated trace id which results in two traces in the console.

I am calling: app.UseXRay("v4-rpc"); as part of the gRPC server startup

What is the recommended way of propagating the trace id over to the gRPC server?

[NathanielRN]

Hey @RobWalker, thanks for raising this issue!

I have an aspnet core app with a front end HTTP server and a second gRPC server with an AWS ALB between them.

Just to confirm, does your flow look like this?

User -> aspnet Core App Front-end HTTP Server (Instrumented with X-Ray) -> Application Load Balancer -> gRPC server (Instrumented with X-Ray)

And you want there to be 1 single trace throughout, correct?

When making the gRPC call I am grabbing the current trace id and adding it to a X-Amzn-Trace-Id header.

Can you explain in more detail how you are doing this? As long as you have your gRPC server instrumented with X-Ray .NET SDK, you shouldn't have to do anything special!

This is set as something like:

Root=1-6200946f-1a21642c074cf3445e4e537a;Parent=;Sampled=1

I believe this is the source of the issue. The Parent= parameter is empty at this point and this is crucial for linking traces (since traces across different services need the same parent ID), the X-Ray docs say this:

The tracing header can also contain a parent segment ID if the request originated from an instrumented application. For example, if your application calls a downstream HTTP web API with an instrumented HTTP client, the X-Ray SDK adds the segment ID for the original request to the tracing header of the downstream request. An instrumented application that serves the downstream request can record the parent segment ID to connect the two requests.

Maybe you can share how you are instrumenting the aspnet front-end app? As long as you are instrumenting the HTTP client that makes the request to the second gRPC server, the X-Ray instrumentation should know that we are leaving the context of the current app and add this Parent=; key so that the traces are linked. If you have X-Ray tracing on this app without instrumenting the specific outgoing call then this important key won't be added to the tracing header.

So I suspect the ALB is modifying it?

Correct. According to the ELB - ALB docs on request tracing, ALB will always add a trace ID if the X-Amzn-Trace-Id is missing, or add the Self=; component if it is already present.

However, if we want the traces to be linked we are more interested in the Parent=; key being present.

Please let me know if you have any questions! 🙂