Add Vertex AI Batch Passthrough Support with Cost Tracking #15562

Sameerlite · 2025-10-15T14:34:02Z

Title

Add Vertex AI Batch Passthrough Support with Cost Tracking

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
I have added a screenshot of my new test passing locally
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature

Changes

Configuration Example:

model_list:
  - model_name: gemini-1.5-flash
    litellm_params:
      model: vertex_ai/gemini-1.5-flash
      vertex_project: your-project-id
      vertex_location: us-central1
      vertex_credentials: path/to/service-account.json

This fixes Claude's models via the Converse API, which should also fix Claude Code.

* fix(router): update model_name_to_deployment_indices on deployment removal When a deployment is deleted, the model_name_to_deployment_indices map was not being updated, causing stale index references. This could lead to incorrect routing behavior when deployments with the same model_name were dynamically removed. Changes: - Update _update_deployment_indices_after_removal to maintain model_name_to_deployment_indices mapping - Remove deleted indices and decrement indices greater than removed index - Clean up empty entries when no deployments remain for a model name - Update test to verify proper index shifting and cleanup behavior * fix(router): remove redundant index building during initialization Remove duplicate index building operations that were causing unnecessary work during router initialization: 1. Removed redundant `_build_model_id_to_deployment_index_map` call in __init__ - `set_model_list` already builds all indices from scratch 2. Removed redundant `_build_model_name_index` call at end of `set_model_list` - the index is already built incrementally via `_create_deployment` -> `_add_model_to_list_and_index_map` Both indices (model_id_to_deployment_index_map and model_name_to_deployment_indices) are properly maintained as lookup indexes through existing helper methods. This change eliminates O(N) duplicate work during initialization without any behavioral changes. The indices continue to be correctly synchronized with model_list on all operations (add/remove/upsert).

…environment (#14929) Co-authored-by: sotazhang <[email protected]>

merge main

* fix(router): update model_name_to_deployment_indices on deployment removal When a deployment is deleted, the model_name_to_deployment_indices map was not being updated, causing stale index references. This could lead to incorrect routing behavior when deployments with the same model_name were dynamically removed. Changes: - Update _update_deployment_indices_after_removal to maintain model_name_to_deployment_indices mapping - Remove deleted indices and decrement indices greater than removed index - Clean up empty entries when no deployments remain for a model name - Update test to verify proper index shifting and cleanup behavior * fix(router): remove redundant index building during initialization Remove duplicate index building operations that were causing unnecessary work during router initialization: 1. Removed redundant `_build_model_id_to_deployment_index_map` call in __init__ - `set_model_list` already builds all indices from scratch 2. Removed redundant `_build_model_name_index` call at end of `set_model_list` - the index is already built incrementally via `_create_deployment` -> `_add_model_to_list_and_index_map` Both indices (model_id_to_deployment_index_map and model_name_to_deployment_indices) are properly maintained as lookup indexes through existing helper methods. This change eliminates O(N) duplicate work during initialization without any behavioral changes. The indices continue to be correctly synchronized with model_list on all operations (add/remove/upsert).

…environment (#14929) Co-authored-by: sotazhang <[email protected]>

Add tiered pricing and cost calculation for xai

…15503)

…_block_repair Add support for thinking blocks and redacted thinking blocks in Anthropic v1/messages API

(feat) Add voyage model integration in sagemaker

Add support for extended thinking in Anthropic's models via Bedrock's Converse API

* docs: fix doc * docs(index.md): bump rc * [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500) * fix: add google_routes to llm_api_routes * test: test_virtual_key_llm_api_routes_allows_google_routes * build: bump version * bump: version 1.78.0 → 1.78.1 * add application level encryption in SQS * add application level encryption in SQS --------- Co-authored-by: Krrish Dholakia <[email protected]> Co-authored-by: Ishaan Jaff <[email protected]> Co-authored-by: deepanshu <[email protected]>

Litellm fix mypy ruff errors1

Litellm staging oct

Add Vertex AI Discovery Engine Rerank Support

vercel · 2025-10-15T14:34:11Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
litellm	Ready	Preview	Comment	Oct 17, 2025 3:23am

…ugh_support

ishaan-jaff

missing tests man

ishaan-jaff · 2025-10-15T15:51:15Z

enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py

        - if not, return False
        - if so, return True
        """
-        from litellm_enterprise.proxy.hooks.managed_files import (


don't change this please

reverted the changed

ishaan-jaff

missing tests man -

ishaan-jaff · 2025-10-15T16:00:19Z

litellm/batches/batch_utils.py

+            model_response = ModelResponse()
+
+            # Create a minimal logging object with required attributes
+            class MockLoggingObj:


please create a real LiteLLM_Logging_object()

Updated the code

Sameerlite · 2025-10-15T18:49:35Z

missing tests man -

added

krrishdholakia · 2025-10-16T22:49:02Z

docs/my-website/docs/providers/vertex.md


 s/o @[Darien Kindlund](https://www.linkedin.com/in/kindlund/) for this tutorial

+## Batch Passthrough


refactor into separate page and add a table indicating cost tracking is supported - https://docs.litellm.ai/docs/generateContent

krrishdholakia · 2025-10-17T20:22:36Z

@Sameerlite your pr fails mapped tests. can you look into this?

CLAassistant · 2025-10-18T19:09:56Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 6 committers have signed the CLA.

✅ lcfyi
✅ ishaan-jaff
✅ AlexsanderHamir
✅ deepanshululla
✅ Sameerlite
❌ LoadingZhang
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Sameerlite · 2025-10-20T18:28:26Z

Closing this due to many conflicts - #15744

lcfyi and others added 30 commits October 5, 2025 03:36

Implement fix for thinking_blocks and converse API calls

dd7d12e

This fixes Claude's models via the Converse API, which should also fix Claude Code.

Add thinking literal

1c3ec18

Fix mypy issues

310e3b3

Type fix for redacted thinking

d07455b

Add voyage model integration in sagemaker

25769b5

Add tiered pricing and cost calculation for xai

a33c348

Add config file logic

6002880

Use already exiting voyage transformation

8c7d7a3

Use generic cost calculator

d0e26e2

UI new build

cb9b65d

fix(prometheus): Fix Prometheus metric collection in a multi-workers …

6060537

…environment (#14929) Co-authored-by: sotazhang <[email protected]>

Merge pull request #15477 from BerriAI/main

75ef5df

merge main

Merge pull request #15478 from BerriAI/main

c496097

merge main

fix conversion of thinking block

a39d263

Resolve conflicts in generated HTML files

ef0d6f0

fix(prometheus): Fix Prometheus metric collection in a multi-workers …

b49ab39

…environment (#14929) Co-authored-by: sotazhang <[email protected]>

Merge pull request #15368 from BerriAI/litellm_add_above_128k_pricing

7a68c57

Add tiered pricing and cost calculation for xai

Remove penalty params as supported params for gemini preview model (#…

2afae44

…15503)

Merge pull request #15501 from BerriAI/litellm_fix_anthropic_thinking…

939d499

…_block_repair Add support for thinking blocks and redacted thinking blocks in Anthropic v1/messages API

refactor code as per comments

6fadfd3

fix merge error

c748321

refactor code as per comments

0906b99

refactor code as per comments

a6ed454

Merge branch 'litellm_staging_oct' into litellm_voyage_sagemaker

2653d6a

Merge pull request #15367 from BerriAI/litellm_voyage_sagemaker

2bd7c88

(feat) Add voyage model integration in sagemaker

Merge branch 'litellm_staging_oct' into lcfyi/add-support-for-thinking

cee4b58

Merge pull request #15220 from lcfyi/lcfyi/add-support-for-thinking

0175c13

Add support for extended thinking in Anthropic's models via Bedrock's Converse API

Sameerlite added 4 commits October 14, 2025 17:48

Merge pull request #15533 from BerriAI/litellm_fix_mypy_ruff_errors1

298fa0c

Litellm fix mypy ruff errors1

Merge pull request #15534 from BerriAI/litellm_staging_oct

43d37a6

Litellm staging oct

Merge pull request #15532 from BerriAI/litellm_google_rerank_endpoint

9af054b

Add Vertex AI Discovery Engine Rerank Support

Add full passthrough support

efe7cb9

Merge branch 'litellm_staging_oct' into litellm_batch_gemini_passthro…

045ded3

…ugh_support

vercel bot deployed to Preview October 15, 2025 14:35 View deployment

Add passthrough handler

570b32a

vercel bot deployed to Preview October 15, 2025 15:06 View deployment

ishaan-jaff requested changes Oct 15, 2025

View reviewed changes

ishaan-jaff reviewed Oct 15, 2025

View reviewed changes

refactor code according to the comments

0cb2aef

vercel bot deployed to Preview October 15, 2025 18:49 View deployment

Sameerlite requested a review from ishaan-jaff October 15, 2025 18:50

remove invalid params

1d0e83d

vercel bot deployed to Preview October 15, 2025 19:06 View deployment

krrishdholakia reviewed Oct 16, 2025

View reviewed changes

Sameerlite added 2 commits October 17, 2025 08:49

Updated documentation

bb77a25

Updated documentation

3693877

vercel bot deployed to Preview October 17, 2025 03:20 View deployment

Updated documentation

dd7e163

Sameerlite requested a review from krrishdholakia October 17, 2025 03:22

vercel bot deployed to Preview October 17, 2025 03:23 View deployment

Sameerlite force-pushed the litellm_staging_oct branch from 789eec0 to 5a637cc Compare October 17, 2025 18:44

Sameerlite changed the base branch from litellm_staging_oct to litellm_sameer_oct_staging October 20, 2025 18:18

Sameerlite closed this Oct 20, 2025


		s/o @[Darien Kindlund](https://www.linkedin.com/in/kindlund/) for this tutorial

		## Batch Passthrough

Uh oh!

Add Vertex AI Batch Passthrough Support with Cost Tracking #15562

Add Vertex AI Batch Passthrough Support with Cost Tracking #15562

Uh oh!

Conversation

Sameerlite commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Title

Relevant issues

Pre-Submission checklist

Type

Changes

Configuration Example:

Uh oh!

vercel bot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ishaan-jaff left a comment

Choose a reason for hiding this comment

Uh oh!

ishaan-jaff Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Sameerlite Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

ishaan-jaff left a comment

Choose a reason for hiding this comment

Uh oh!

ishaan-jaff Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Sameerlite Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Sameerlite commented Oct 15, 2025

Uh oh!

krrishdholakia Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Sameerlite Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

krrishdholakia commented Oct 17, 2025

Uh oh!

CLAassistant commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sameerlite commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Sameerlite commented Oct 15, 2025 •

edited

Loading

vercel bot commented Oct 15, 2025 •

edited

Loading

CLAassistant commented Oct 18, 2025 •

edited

Loading