Sends wrong HTTP/2 response when RpcException happens and custom HTTP/2 trailer appended in middleware

[kolonist]

What version of gRPC and what language are you using?

Should work on every version, but I tested on 2.67.0

What operating system (Linux, Windows,...) and version?

It doesn't matters, but I tested on Tested on Windows, Linux, MacOS M1

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

It doesn't matters, but I tested on:
SDK: 9.0.300
Runtime: 9.0.5

Reproduction

Create simplest gRPC service with Unary handler and throw RpcException from the inside like this:

public override async Task<Empty> Unary(Empty _, ServerCallContext context)
{
    // this is not important
    await Task.Yield(); 

    // RpcException with custom status code and some details
    throw new RpcException(new Status(StatusCode.InvalidArgument, "My Exception"));
}

Now if you call this method with dotnet client or external tool, written with another languages, then it works just fine.
I use grpc_cli (but also tried Postman):

$ grpc_cli call --noremotedb --proto_path /Users/username/dev/sandbox/Sandbox.Grpc/Protos --protofiles echo.proto --timeout 2 localhost:6001 GrpcService/Unary "" 

connecting to localhost:6001
Received trailing metadata from server:
content-length : 0
date : Wed, 25 Jun 2025 21:28:19 GMT
server : Kestrel
Rpc failed with status code 3, error message: My Exception

Both Status code and error message works fine.

Then I add middleware which write custom HTTP/2 trailer:

public class Startup
{
    ...

    public void Configure(IApplicationBuilder app)
    {
        ...

        app.Use(static async (context, next) =>
        {
            await next(context);

            if (context.Response.SupportsTrailers())
            {
                // append custom trailer
                context.Response.AppendTrailer("x-my-custom-trailer", "42");
            }
        });

        ...
    }

    ...
}

I can't write trailer from grpc-interceptor, because I need to do it after another middlewares which calculate some values for this trailer, so middleware in defined position is the only option I have.

Then if I call method I don't get correct error code:

$ grpc_cli call --noremotedb --proto_path /Users/username/dev/sandbox/Sandbox.Grpc/Protos --protofiles echo.proto --timeout 2 localhost:6001 GrpcService/Unary "" 

connecting to localhost:6001
Received trailing metadata from server:
x-my-custom-trailer : 42
Rpc failed with status code 2, error message: No status received

Note that I don't get correct error code or status message.

I also don't get other metadata like content-length, date and server.

In Wireshark it looks like this:

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 121, 200 OK
        ...
        Type: HEADERS (1)
        Flags: 0x04, End Headers
        ...
        [Header Count: 8]
        Header: :status: 200 OK
        Header: content-type: application/grpc
        Header: date: Wed, 25 Jun 2025 21:38:30 GMT
        Header: server: Kestrel
        Header: content-length: 0
        Header: grpc-status: 3
        Header: grpc-message: My Exception
        ...
HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 24
        ...
        Type: HEADERS (1)
        Flags: 0x05, End Headers, End Stream
        ...
        [Header Length: 29]
        [Header Count: 1]
        Header: x-my-custom-trailer: 42

Note that there is 2 HTTP/2 HEADERS sections: Flags: 0x04, End Headers and Flags: 0x05, End Headers, End Stream. And all metadata from gRPC library placed in the first section but my custom trailer placed in the second section.

Interesting that if I try to read it with another grpc-dotnet then I get correct RpcException but I don't get my custom trailer:

using var channel = GrpcChannel.ForAddress(url);
var client = new FixtureService.FixtureServiceClient(channel);

var request = new Empty();
using var call = client.UnaryCallAsync(request, cancellationToken: CancellationToken.None);

// it's empty here
var responseHeaders = await call.ResponseHeadersAsync;

try
{
    await call.ResponseAsync;
}
catch(Exception ex)
{
    // true
    if (ex is RpcException rpcException)
    {
        // this 3 values are correct
        rpcException.StatusCode;  // InvalidArgument
        rpcException.Status.StatusCode; // InvalidArgument
        rpcException.Status.Detail; // My Exception

        // but this is empty
        var errorTrailers = rpcException.Trailers;
    }
}

// and this is empty
var responseTrailers = call.GetTrailers();

The essence of the error

If I append HTTP/2 thailer to HTTPContext then grpc-dotnet library does not add its own metadata to trailers but it writes it to headers instead. Then we get HTTP/2 answer with two HEADERs sections (one with grpc-dotnet metadata and one with custom trailers) instead of one with all trailers.

Cause of error

I believe that all this magic came from this method: https://github.com/grpc/grpc-dotnet/blob/master/src/Grpc.AspNetCore.Server/Internal/GrpcProtocolHelpers.cs#L157

We check that response.HasStarted is false, believe that response is Trailers-only and write metadata to headers. But in our case someone already started another header with custom trailers.

So it seems that response.HasStarted == false is not enough to write metadata to headers. Maybe we need to check that there is no trailers already added like this:

public static IHeaderDictionary GetTrailersDestination(HttpResponse response)
{
    var feature = response.HttpContext.Features.Get<IHttpResponseTrailersFeature>();

    if (response.HasStarted)
    {
        // The response has content so write trailers to a trailing HEADERS frame
        if (feature?.Trailers == null || feature.Trailers.IsReadOnly)
        {
            throw new InvalidOperationException("Trailers are not supported for this response. The server may not support gRPC.");
        }

        return feature.Trailers;
    }
    else
    {
        if (feature?.Trailers.Count > 0)
        {
            return feature.Trailers;
        }
        else
        {
            // The response is "Trailers-Only". There are no gRPC messages in the response so the status
            // and other trailers can be placed in the header HEADERS frame
            return response.Headers;
        }
    }
}

Workaround

Now I use workaround - I copied method GetTrailersDestination and add HTTP/2 trailer to headers if response has not started but it seems that it's not clean solution.

I'll be happy if you could look at this issue in not too distant future.
Thank you for your attention