Leverage MEAI to build an AI Gateway

[verdie-g]

In my company, we would like to leverage MEAI to build a AI gateway. That would make all our LLM usages enjoy the MEAI features:

• provider-agnostic
• metrics
• tracing
• logs
• caching
• conversations

The OpenAI request/response model is quite complex. Since MEAI.OpenAI already has helpers to do these conversions:

• MEAI input -> OpenAI input
• OpenAI output -> MEIA output

It would be great if we could also have:

• OpenAI input -> MEIA input
• MEIA output -> OpenAI output

so at the end we could do this

[Route("openai")]
[ApiController]
[Consumes(MediaTypeNames.Application.Json)]
[Produces(MediaTypeNames.Application.Json)]
public class OpenAIController(IChatClient chatClient) : Controller
{
    private readonly IChatClient _chatClient = chatClient;

    /// <summary>
    /// Creates a model response for the given chat conversation (https://platform.openai.com/docs/api-reference/chat/create).
    /// </summary>
    [HttpPost("v1/chat/completions")]
    public async Task<OpenAIChatCompletion> CreateChatCompletion(
        [FromBody] OpenAIChatCompletionRequest request,
        CancellationToken cancellation)
    {
        var chatResponse = await _chatClient.GetResponseAsync(
            request.Messages.ToChatMessages(),
            request.ToChatOptions(),
            cancellation);
        return chatResponse.ToOpenAIChatCompletion();
    }
}

I'm playing with MicrosoftExtensionsAIChatExtensions but the extension methods there don't seem to match the OpenAI request/response. For example, serializing the result of AsOpenAIChatCompletion returns something that is very different from the expected response

{
  "createdAt": "2025-09-02T11:32:35+00:00",
  "finishReason": "Stop",
  "contentTokenLogProbabilities": [],
  "refusalTokenLogProbabilities": [],
  "role": "Assistant",
  "content": [
    {
      "kind": "Text",
      "text": "Hello! How can I assist you today?",
      "imageUri": null,
      "imageBytes": null,
      "imageBytesMediaType": null,
      "inputAudioBytes": null,
      "inputAudioFormat": null,
      "fileId": null,
      "fileBytes": null,
      "fileBytesMediaType": null,
      "filename": null,
      "imageDetailLevel": null,
      "refusal": null
    }
  ],
  "toolCalls": [],
  "refusal": null,
  "functionCall": null,
  "outputAudio": null,
  "annotations": [],
  "id": "chatcmpl-CBJXXMYPeRzKjNTMZ1QB6B2JI",
  "model": "gpt-4.1-mini-2025-04-14",
  "systemFingerprint": "fp_4f3e32cd4e",
  "usage": {
    "outputTokenCount": 10,
    "inputTokenCount": 19,
    "totalTokenCount": 29,
    "outputTokenDetails": {
      "reasoningTokenCount": 0,
      "audioTokenCount": 0,
      "acceptedPredictionTokenCount": 0,
      "rejectedPredictionTokenCount": 0
    },
    "inputTokenDetails": {
      "audioTokenCount": 0,
      "cachedTokenCount": 0
    }
  }
}

also deserializing an OpenAI.Chat.ChatMessage (so we can just do AsChatMessages to get a MEAI ChatMessage) throws

System.NotSupportedException: Deserialization of types without a parameterless constructor, a singular
parameterized constructor, or a parameterized constructor annotated with 'JsonConstructorAttribute' is
not supported. Type 'OpenAI.Chat.ChatMessage'. Path: $.messages[0] | LineNumber: 3 | BytePositionInLine: 7.

[stephentoub]

serializing the result of AsOpenAIChatCompletion returns something that is very different from the expected response

How are you serializing them? The OpenAI library relies on its own serialization mechanism rather than STJ. You need to cast to its IJsonModel and then use the methods there to read/write, eg https://github.com/openai/openai-dotnet/blob/cc157bd8f16898c0d8754037a8cccf2ea9d4198f/examples/Chat/Example08_ChatSerialization.cs#L27

[verdie-g]

Nice!!

I was able to find System.ClientModel.Primitives.JsonModelConverter that creates the bridge with STJ.

I had to define my own OpenAIChatCompletionRequest though and create a method to convert the options to MEAI's ChatOptions. Is there a better way to do that?

public sealed record OpenAIChatCompletionRequest
{
    [JsonPropertyName("messages")]
    public required IReadOnlyList<OpenAI.Chat.ChatMessage> Messages { get; init; }

    [JsonPropertyName("model")]
    public required string Model { get; init; }

    // Other options.
}

[verdie-g]

Ok the OpenAIChatCompletionRequest is actually OpenAI.Chat.ChatCompletionOptions but the Messages property is internal 🤔

[verdie-g]

Proposed #6764.