Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(chore)LLMObs: instrument bedrock runtime invoke model #5116

Merged
merged 56 commits into from
Jan 24, 2025

Conversation

yahya-mouman
Copy link
Contributor

@yahya-mouman yahya-mouman commented Jan 15, 2025

What does this PR do?

Adds LLM Obs support for Bedrock InvokeModel method (only llm calls and not the embed ones)

Supports invocation for the following model providers :

  • AI21
  • AMAZON
  • ANTHROPIC
  • COHERE
  • META
  • STABILITY
  • MISTRAL

Disclaimer : This PR doesn't cover the token counts for the bedrock model calls

app datadoghq com_llm_traces_query=%40ml_app%3Abedrock-example-js%20%40event_type%3Aspan%20%40parent_id%3Aundefined agg_m=count agg_m_source=base agg_t=count fromUser=false llmPanels=%5B%7B%22t%22%3A%22sampleDetailPanel%22%2C%22rEID%22%3A%2

Motivation

Plugin Checklist

Additional Notes

Copy link

github-actions bot commented Jan 15, 2025

Overall package size

Self size: 8.53 MB
Deduped: 94.89 MB
No deduping: 95.41 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.2.0 | 13.9 MB | 13.91 MB | | @datadog/pprof | 5.4.1 | 9.76 MB | 10.13 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.6.1 | 2.59 MB | 2.73 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 826.22 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | semver | 7.6.3 | 95.82 kB | 95.82 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.1 | 51.46 kB | 51.46 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@pr-commenter
Copy link

pr-commenter bot commented Jan 15, 2025

Benchmarks

Benchmark execution time: 2025-01-24 14:24:50

Comparing candidate commit 9107d4d in PR branch yahya/llmobs-add-bedrock-runtime-support with baseline commit f41f5f7 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 912 metrics, 21 unstable metrics.

@yahya-mouman yahya-mouman changed the base branch from master to yahya/refactor-apm-bedrock-runtime January 20, 2025 17:54
Copy link

codecov bot commented Jan 20, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.56%. Comparing base (c28765a) to head (9107d4d).
Report is 3 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #5116       +/-   ##
===========================================
- Coverage   95.02%   72.56%   -22.47%     
===========================================
  Files         108      150       +42     
  Lines        3475     5277     +1802     
===========================================
+ Hits         3302     3829      +527     
- Misses        173     1448     +1275     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yahya-mouman yahya-mouman marked this pull request as ready for review January 20, 2025 18:06
@yahya-mouman yahya-mouman requested a review from a team as a code owner January 20, 2025 18:06
@yahya-mouman yahya-mouman changed the title yahya/llmobs-add-bedrock-runtime-support (chore)[LLMOBS] : instrument bedrock runtime invoke model Jan 20, 2025
@yahya-mouman yahya-mouman changed the title (chore)[LLMOBS] : instrument bedrock runtime invoke model (chore)LLMObs: instrument bedrock runtime invoke model Jan 20, 2025
Base automatically changed from yahya/refactor-apm-bedrock-runtime to master January 24, 2025 13:02
@yahya-mouman yahya-mouman enabled auto-merge (squash) January 24, 2025 15:01
@yahya-mouman yahya-mouman merged commit 29c26b0 into master Jan 24, 2025
332 checks passed
@yahya-mouman yahya-mouman deleted the yahya/llmobs-add-bedrock-runtime-support branch January 24, 2025 16:13
szegedi pushed a commit that referenced this pull request Jan 28, 2025
* Update file structure

* join spaces

* add top_k

* add index for plugins

* add llmobs index

* refactor apm tracing

* add llmobs instrumentation for bedrock

* add llmobs instrumentation for bedrock

* rename

* Update file structure

* conflict

* conflict

* add index for plugins

* add llmobs index

* add llmobs instrumentation for bedrock

* lint change

* Update packages/datadog-plugin-aws-sdk/src/services/bedrockruntime/index.js

Co-authored-by: Sam Brenner <[email protected]>

* Update packages/datadog-plugin-aws-sdk/src/services/bedrockruntime/tracing.js

Co-authored-by: Sam Brenner <[email protected]>

* Update packages/datadog-plugin-aws-sdk/src/services/bedrockruntime/tracing.js

Co-authored-by: Sam Brenner <[email protected]>

* Update packages/datadog-plugin-aws-sdk/src/services/bedrockruntime/utils.js

Co-authored-by: Sam Brenner <[email protected]>

* CODEOWNERS

* CODEOWNERS

* remove shouldSetChoiceId override

* remove shouldSetChoiceId override

* remove shouldSetChoiceId override

* remove shouldSetChoiceId override

* lint

* Update packages/dd-trace/src/llmobs/plugins/bedrockruntime.js

Co-authored-by: Sam Brenner <[email protected]>

* Update packages/dd-trace/test/llmobs/plugins/bedrockruntime/bedrockruntime.spec.js

Co-authored-by: Sam Brenner <[email protected]>

* resolving review comments

* rename span

* Add test fixtures and reduce redundant code

* Add test to ci workflow

* Update file structure

* conflict

* conflict

* add index for plugins

* add llmobs index

* add llmobs instrumentation for bedrock

* lint change

* Update packages/dd-trace/src/llmobs/plugins/bedrockruntime.js

Co-authored-by: Sam Brenner <[email protected]>

* Update packages/dd-trace/test/llmobs/plugins/bedrockruntime/bedrockruntime.spec.js

Co-authored-by: Sam Brenner <[email protected]>

* resolving review comments

* rename span

* Add test fixtures and reduce redundant code

* Add test to ci workflow

* rename plugin name to aws sdk

* update ci yml

* add refactor for split

---------

Co-authored-by: Sam Brenner <[email protected]>
@szegedi szegedi mentioned this pull request Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants