Skip to content

feat: surface tool_calls on ChatResponse#19

Merged
avivsinai merged 1 commit into
avivsinai:mainfrom
freelabz:feat/tool-calls
Jul 2, 2026
Merged

feat: surface tool_calls on ChatResponse#19
avivsinai merged 1 commit into
avivsinai:mainfrom
freelabz:feat/tool-calls

Conversation

@ocervell

Copy link
Copy Markdown
Contributor

Summary

  • Adds a tool_calls: Option<Value> field to ChatResponse carrying model-requested function calls in OpenAI shape — an array of { id, type: \"function\", function: { name, arguments } } where arguments is the raw JSON string the model produced.
  • OpenAI-compatible providers pass tool_calls through unchanged from the assistant message. Empty / missing arrays normalize to None so callers can use if let Some(_) reliably.
  • Anthropic tool_use content blocks are mapped to the OpenAI shape (the block id / name / input become the call id / function name / arguments).
  • Gemini functionCall parts on the first candidate are mapped the same way, synthesizing call_<idx> ids since Gemini doesn't supply them.

Why

Building an agent loop on top of litellm-rust currently isn't possible — you can SEND req.tools = Some(...) but the model's tool_calls reply is dropped by the OpenAI parser (OpenAIMessage only deserializes content + reasoning_*), so the agent never sees what the model wants to invoke.

This patch closes that gap with a single new field. The shape is OpenAI's (the existing target for ChatRequest.tools) so consumers get one dispatch path; Anthropic and Gemini are normalized through the same shape so swapping providers doesn't change call-site code.

Test plan

  • cargo test — 69 passing (was 65, +4 new):
    • chat_completion_surfaces_tool_calls — OpenAI tool_calls round-trip
    • chat_completion_omits_tool_calls_when_absent — missing field stays None
    • chat_completion_treats_empty_tool_calls_as_none — empty array stays None
    • chat_completion_maps_anthropic_tool_use_to_openai_tool_calls — Anthropic mapping
  • No breaking changes: ChatResponse gains a field with #[serde(default, skip_serializing_if = \"Option::is_none\")]; existing serialized payloads still deserialize, payloads with no tool calls serialize identically.

CHANGELOG entry added under `[Unreleased]`.

Adds a `tool_calls: Option<Value>` field to `ChatResponse` carrying
model-requested function calls in OpenAI shape — an array of
`{ id, type: "function", function: { name, arguments } }` where
`arguments` is the raw JSON string the model produced.

- OpenAI-compatible providers pass `tool_calls` through unchanged from the
  assistant message. Empty / missing arrays normalize to `None` so callers
  can use `if let Some(_)` reliably.
- Anthropic `tool_use` content blocks are mapped to the OpenAI shape
  (the block `id`/`name`/`input` become the call id/function name/arguments).
- Gemini `functionCall` parts on the first candidate are mapped the same
  way, synthesizing `call_<idx>` ids since Gemini doesn't supply them.

Tests cover all three providers plus the empty-array and missing-field
cases.
@avivsinai

Copy link
Copy Markdown
Owner

Sorry for the slow turnaround, this one slipped past us. Took a proper look now: clean, useful change, and it builds, tests, and clippy-passes locally. Two small things before CI goes green:

  1. cargo fmt. The format check fails on a few lines in anthropic.rs, gemini.rs, and the new tests. Running cargo fmt fixes it.
  2. A Gemini test. extract_gemini_tool_calls is currently uncovered (the new tests hit OpenAI and Anthropic). This repo WireMock-covers provider response behavior, so a functionCall case keeps the third path honest. Drop-in below if it's useful.

Once those land I'll approve the workflow run and we can merge. Thanks for the contribution.

Gemini test — add to tests/integration_providers.rs
mod gemini_chat_tests {
    use super::*;

    /// Gemini `functionCall` parts map to OpenAI-shape tool_calls so consumers
    /// can use a single dispatch path across providers. Gemini supplies no call
    /// ids, so they are synthesized as `call_<part_idx>` — note the text part at
    /// index 0 means the first (and only) call here is `call_1`, not `call_0`.
    #[tokio::test]
    async fn chat_completion_maps_gemini_function_call_to_openai_tool_calls() {
        let mock_server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/models/gemini-2.5-flash:generateContent"))
            .and(header("x-goog-api-key", "test-key"))
            .respond_with(ResponseTemplate::new(200).set_body_json(json!({
                "candidates": [{
                    "content": {
                        "parts": [
                            { "text": "I'll scan that for you." },
                            {
                                "functionCall": {
                                    "name": "run_task",
                                    "args": { "name": "nmap", "targets": ["example.com"] }
                                }
                            }
                        ]
                    }
                }],
                "usageMetadata": {
                    "promptTokenCount": 5,
                    "candidatesTokenCount": 5,
                    "totalTokenCount": 10
                }
            })))
            .mount(&mock_server)
            .await;
        let cfg = ProviderConfig {
            base_url: Some(mock_server.uri()),
            api_key: Some("test-key".to_string()),
            ..Default::default()
        };
        let resp = gemini::chat(&make_client(), &cfg, simple_chat_request("gemini-2.5-flash"))
            .await
            .unwrap();
        let calls = resp.tool_calls.expect("tool_calls populated");
        let arr = calls.as_array().expect("array");
        assert_eq!(arr.len(), 1);
        assert_eq!(arr[0]["type"], "function");
        // Part 0 is text, part 1 is the functionCall -> synthesized id `call_1`.
        assert_eq!(arr[0]["id"], "call_1");
        assert_eq!(arr[0]["function"]["name"], "run_task");
        // arguments is a JSON string (OpenAI convention) — parse it.
        let args: serde_json::Value =
            serde_json::from_str(arr[0]["function"]["arguments"].as_str().unwrap()).unwrap();
        assert_eq!(args["name"], "nmap");
        assert_eq!(args["targets"][0], "example.com");
    }

    /// A Gemini response with no `functionCall` parts leaves `tool_calls` as None.
    #[tokio::test]
    async fn chat_completion_omits_gemini_tool_calls_when_absent() {
        let mock_server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/models/gemini-2.5-flash:generateContent"))
            .and(header("x-goog-api-key", "test-key"))
            .respond_with(ResponseTemplate::new(200).set_body_json(json!({
                "candidates": [{
                    "content": { "parts": [ { "text": "just text, no calls" } ] }
                }],
                "usageMetadata": {
                    "promptTokenCount": 3,
                    "candidatesTokenCount": 3,
                    "totalTokenCount": 6
                }
            })))
            .mount(&mock_server)
            .await;
        let cfg = ProviderConfig {
            base_url: Some(mock_server.uri()),
            api_key: Some("test-key".to_string()),
            ..Default::default()
        };
        let resp = gemini::chat(&make_client(), &cfg, simple_chat_request("gemini-2.5-flash"))
            .await
            .unwrap();
        assert!(resp.tool_calls.is_none());
    }
}

@avivsinai avivsinai merged commit d495201 into avivsinai:main Jul 2, 2026
1 check passed
@avivsinai

Copy link
Copy Markdown
Owner

Merged, thanks @ocervell — this is a clean, genuinely useful addition and the cross-provider normalization is exactly the right shape. I applied the two small follow-ups from the review directly on main as maintainer polish so this didn't sit waiting on you: cargo fmt and a Gemini functionCall WireMock test. The feature commit is all yours. Appreciate the contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants