[Bug]: enable_litellm_models_monitoring is documented for LiteLLMChatModel but also governs AnthropicChatModel

[feiiiiii5]

What component(s) are affected?

• Opik Python SDK
• Opik Typescript SDK
• Opik Agent Optimizer SDK
• Opik UI
• Opik Server
• Documentation

Opik version

• Opik version: 2.2.71 (origin/main at f06298d93)

Describe the problem

enable_litellm_models_monitoring is documented as governing one class, but the Anthropic adapter also reads it.

sdks/python/src/opik/config.py:217-222:

enable_litellm_models_monitoring: bool = True
"""
If set to True - Opik will create llm spans for LiteLLMChatModel calls.
It is mainly to be used in tests since litellm uses external Opik callback
which makes HTTP requests not via the opik package.
"""

sdks/python/src/opik/evaluation/models/anthropic/anthropic_chat_model.py:65:

enable_tracking = track and config.enable_litellm_models_monitoring

So an AnthropicChatModel built with track=True silently produces no spans when a user or test harness sets enable_litellm_models_monitoring=False — a flag whose own documentation limits it to LiteLLMChatModel and whose stated reason for existing (litellm's external callback plumbing) does not apply to the Anthropic client, which traces through opik.integrations.anthropic.track_anthropic in-process.

This surfaced while writing #8408/#8409: the repo triage bot suggested the LangChain adapter should honour the same config flag "for consistency". I did not do that, precisely because the flag is documented as LiteLLM-scoped — but checking that led here, and the Anthropic use looks like the same copy-paste viewed from the other side.

Which way is correct?

Either reading is defensible and they are not the same change, so I would rather not guess:

1. The docstring is accurate and the usage is wrong → drop the config term from anthropic_chat_model.py:65, leaving track to govern that adapter. Behaviour change for anyone currently setting the flag to False.
2. The flag is meant as a general "SDK-attached LLM span" switch → then config.py:217 and its name should be generalised (and the LangChain adapter should read it too, which is what the triage bot assumed). Also a behaviour change, in the opposite direction.

Option 3 is adding an Anthropic-specific option, which is a new configuration surface rather than a fix.

Reproduction steps and code snippets

Not executed as a live trace test — this is a code-and-documentation contradiction rather than a runtime mystery, and the two lines above are the whole of it. rg enable_[a-z_]*monitoring src/opik/config.py returns exactly this one option, so there is no general switch being read by mistake.

Error logs or stack trace

None — the failure mode is silent absence of spans.

[github-actions]

I now have all the information needed. The issue is clear and well-described. Let me write the response.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing solution or workaround found. The reporter's analysis is accurate: AnthropicChatModel silently produces no spans when enable_litellm_models_monitoring=False, despite the flag's documentation saying it only governs LiteLLMChatModel.

As a temporary workaround, users who have set OPIK_ENABLE_LITELLM_MODELS_MONITORING=false (or the ini-file equivalent) and want Anthropic spans back can unset that flag, but that restores LiteLLM spans too — there is no current way to control them independently.

Code Investigation

Three files are involved:

sdks/python/src/opik/config.py (lines 217–222)

enable_litellm_models_monitoring: bool = True
"""
If set to True - Opik will create llm spans for LiteLLMChatModel calls.
It is mainly to be used in tests since litellm uses external Opik callback
which makes HTTP requests not via the opik package.
"""

The docstring is explicit: this flag exists because of LiteLLM's external callback plumbing and is described as test-oriented. Nothing in the docstring mentions the Anthropic adapter.

sdks/python/src/opik/evaluation/models/litellm/litellm_chat_model.py

config = opik_config.OpikConfig()
enable_tracking = track and config.enable_litellm_models_monitoring

Correct and intended use of the flag.

sdks/python/src/opik/evaluation/models/anthropic/anthropic_chat_model.py (line 65)

config = opik_config.OpikConfig()
enable_tracking = track and config.enable_litellm_models_monitoring

AnthropicChatModel uses track_anthropic (an in-process integration), which has completely different tracking mechanics from LiteLLM's external callback. Reading the same flag here is a copy-paste from the LiteLLM adapter and is incorrect per the flag's documented semantics.

sdks/python/src/opik/evaluation/models/langchain/langchain_chat_model.py
LangChain does not read enable_litellm_models_monitoring at all — it handles its track parameter via a separate opik_monitoring module. This is the correct pattern for a non-LiteLLM adapter.

Test coverage gap: sdks/python/tests/unit/evaluation/models/test_anthropic_chat_model.py has no test asserting that track=False prevents track_anthropic from being called (or that enable_litellm_models_monitoring=False has any effect). The tests cover response parsing and message adaptation extensively, but not the tracking initialization path.

This is also closely related to #8408, which found the same class of problem in LangchainChatModel (track=False being silently ignored there).

Next Steps

Option 1 is almost certainly correct (the docstring is accurate; the usage is wrong). The fix is minimal:

In sdks/python/src/opik/evaluation/models/anthropic/anthropic_chat_model.py, change line 65 from:

config = opik_config.OpikConfig()
enable_tracking = track and config.enable_litellm_models_monitoring

to simply:

enable_tracking = track

and remove the now-unused import opik.config as opik_config at the top of the file if it's not referenced elsewhere. This makes AnthropicChatModel behave consistently with how LangchainChatModel handles the track flag, and matches what the docstring says the config flag does.

A test should be added in test_anthropic_chat_model.py to assert that when track=False, track_anthropic is not called during __init__.

If the team instead decides on Option 2 (generalising the flag), that's a broader naming/API decision: the flag name enable_litellm_models_monitoring would need to be deprecated or aliased, all three adapters would need to read it, and the docstring + env-var name (OPIK_ENABLE_LITELLM_MODELS_MONITORING) would all need updating — that scope warrants a separate design discussion.

Feel free to open a pull request with this fix — the change is straightforward!

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik