Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add W&B Weave Tracing Integration #14262

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

parambharat
Copy link

@parambharat parambharat commented Feb 24, 2025

Summary

This PR adds W&B Weave as a new tracing provider in Dify, resolving the feature request in #14463. The integration follows similar patterns to the existing tracers, allowing users to track LLM calls, workflows, and tool usage in W&B Weave.

Motivation and Context

Many Dify users leverage Weights & Biases for their ML workflow tracking and would benefit from native W&B Weave integration. This addition gives users more flexibility in choosing tracing providers that align with their existing MLOps stack.

Weave offers specialized capabilities for LLM applications:

  • Detailed call tracing with input/output capture
  • Support for complex application flows
  • Rich visualization and analytics
  • Integration with evaluation frameworks

Implementation Details

The implementation:

  • Adds Weave as a new tracing provider option
  • Creates a WeaveConfig model for storing API key, entity, project, and endpoint configuration
  • Implements a client wrapper for W&B Weave client
  • Develops a tracer class implementing the BaseTraceInstance interface
  • Integrates with Dify's existing trace queue management system

Dependencies

  • Added dependency: weave package for Python SDK integration

The implementation follows Dify's existing patterns to ensure consistency while providing the new functionality needed for W&B Weave integration.

Resolves #14463

Checklist

  • This change requires a documentation update, included: Dify Document :Docs PR

  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)

  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.

  • I've updated the documentation accordingly.

  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

@parambharat parambharat changed the title feat: Add W&B weave tracing [WIP] feat: Add W&B weave tracing Feb 24, 2025
@parambharat parambharat changed the title [WIP] feat: Add W&B weave tracing feat: Add W&B Weave Tracing Integration Feb 27, 2025
@parambharat parambharat marked this pull request as ready for review February 27, 2025 09:55
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. dependencies Pull requests that update a dependency file 💪 enhancement New feature or request labels Feb 27, 2025
@parambharat parambharat changed the title feat: Add W&B Weave Tracing Integration [WIP] feat: Add W&B Weave Tracing Integration Feb 27, 2025
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Feb 28, 2025
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Feb 28, 2025
@parambharat parambharat changed the title [WIP] feat: Add W&B Weave Tracing Integration feat: Add W&B Weave Tracing Integration Feb 28, 2025
Comment on lines +66 to +70
if field_name == "inputs":
data = {
"messages": v,
}
elif field_name == "outputs":

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When inputs is a list of dictionaries, the usage_metadata and file_list are not added to each message in the list, potentially losing token usage tracking.

📝 Committable Code Suggestion

‼️ Ensure you review the code suggestion before committing it to the branch. Make sure it replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
if field_name == "inputs":
data = {
"messages": v,
}
elif field_name == "outputs":
if field_name == "inputs":
data = {
"messages": [dict(msg, **{"usage_metadata": usage_metadata, "file_list": file_list}) for msg in v] if isinstance(v, list) else v,
}
elif field_name == "outputs":

Comment on lines +282 to +283
attributes["start_time"] = (trace_info.start_time or trace_info.message_data.created_at,)
attributes["end_time"] = (trace_info.end_time or trace_info.message_data.updated_at,)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect tuple creation in start_time and end_time attributes due to trailing comma, causing them to be tuples instead of datetime values

📝 Committable Code Suggestion

‼️ Ensure you review the code suggestion before committing it to the branch. Make sure it replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
attributes["start_time"] = (trace_info.start_time or trace_info.message_data.created_at,)
attributes["end_time"] = (trace_info.end_time or trace_info.message_data.updated_at,)
attributes["start_time"] = trace_info.start_time or trace_info.message_data.created_at
attributes["end_time"] = trace_info.end_time or trace_info.message_data.updated_at

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file 💪 enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: Implement WandB Weave Tracer Integration
2 participants