Skip to content

Conversation

drewby
Copy link
Member

@drewby drewby commented May 4, 2025

Description

This PR adds instrumentation for OpenAI's embeddings API in the GenAI instrumentation suite. The implementation follows the OpenTelemetry semantic conventions for generative AI systems and provides automatic instrumentation for the OpenAI Python client when using embeddings functionality.

The implementation captures important metadata about embedding operations including model, dimensions, and relevant timing information while respecting sensitive data handling practices.

  • Added instrumentation for both synchronous and asynchronous OpenAI embedding API calls
  • Implemented span and metrics using existing attributes, with two new custom:
    • ai.embedding.dimensions - Number of dimensions in the embedding vectors
    • ai.embedding.encoding_format - The encoding format of the embedding vectors response (base64 or float)
  • Capturing input text content (disabled by default for privacy)
  • Added a usage example called embeddings

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

  • Unit tests using mock responses to verify proper span creation and attribute population
  • Integration tests with the OpenAI client against a mock server
  • Manual testing using examples/embeddings with real OpenAI service

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

Copy link
Member

@lmolkova lmolkova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!!!

@drewby
Copy link
Member Author

drewby commented Jun 9, 2025

@lmolkova @xrmx, I think I've removed all of the Event related code, so now this is just recording the Span and Metrics.

@anuraaga
Copy link
Contributor

Hey - just wondering what's the status of this PR as interested in having embeddings instrumented here. If there's anything I can do to help on it, let me know.

@xrmx xrmx self-requested a review September 10, 2025 16:19
@drewby
Copy link
Member Author

drewby commented Sep 12, 2025

@pmcollins @xrmx @lmolkova Thank You! I'm working on it and wil have updates soon.

@drewby
Copy link
Member Author

drewby commented Sep 22, 2025

@xrmx @lmolkova I've made several updates based on the feedback. I hope I addressed most concerns. There is one check not passing but seems unrelated to openai library.

@aabmass
Copy link
Member

aabmass commented Oct 7, 2025

@xrmx can you take another pass?

@lmolkova lmolkova requested review from Copilot and removed request for alizenhom and gyliu513 October 14, 2025 16:16
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds comprehensive instrumentation for OpenAI's embeddings API to the existing OpenTelemetry instrumentation suite. The implementation provides automatic tracing and metrics collection for both synchronous and asynchronous embedding operations following OpenTelemetry semantic conventions.

  • Adds support for tracing OpenAI embeddings API calls with proper span attributes and metrics
  • Introduces new custom attributes for embedding-specific metadata (dimensions and encoding format)
  • Includes comprehensive test coverage with both unit and integration tests

Reviewed Changes

Copilot reviewed 26 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/opentelemetry/instrumentation/openai_v2/__init__.py Adds instrumentation hooks for embeddings endpoints
src/opentelemetry/instrumentation/openai_v2/patch.py Implements tracing logic for sync/async embeddings operations
src/opentelemetry/instrumentation/openai_v2/utils.py Adds embeddings-specific request attribute handling
tests/test_embeddings.py Comprehensive test suite for synchronous embeddings API
tests/test_async_embeddings.py Test coverage for asynchronous embeddings functionality
tests/test_utils.py Shared test utilities for consistent assertion patterns

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

opentelemetry-sdk~=1.36.0
opentelemetry-exporter-otlp-proto-grpc~=1.36.0
opentelemetry-distro~=0.57b0
opentelemetry-instrumentation-openai-v2~=2.1b0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
opentelemetry-instrumentation-openai-v2~=2.1b0
opentelemetry-instrumentation-openai-v2~=2.2b0

Copy link
Contributor

@xrmx xrmx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of comments but LGTM.

attributes["gen_ai.embeddings.dimension.count"] = dimensions

# Add encoding format if specified
if "encoding_format" in kwargs:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to guard against instances of openai.NotGiven (available since 1.13.4) and so add one test for sync and one for async checking that for the encoding_format parameter . We have another PR adding guards for chat completions #3760

] = response_format

service_tier = kwargs.get("service_tier")
attributes[GenAIAttributes.GEN_AI_OPENAI_RESPONSE_SERVICE_TIER] = (
Copy link
Contributor

@xrmx xrmx Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this should be GenAIAttributes.GEN_AI_OPENAI_REQUEST_SERVICE_TIER, care to fix while you are at it? Don't want to create conflicts on another PR :) With a separate entry in the changelog though. If you don't want to do that in this PR we can do it later, no worries.

@xrmx
Copy link
Contributor

xrmx commented Oct 16, 2025

Also I would ignore copilot comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.