Skip to content

Conversation

@SajidAlamQB
Copy link
Contributor

Description

Related to: #1203

This PR adds LangfuseTraceDataset as an experimental dataset to enable Langfuse integration for tracing in Kedro pipelines. The implementation for this dataset can be found here: kedro-org/kedro-academy#77.

The dataset provides three modes of integration:

langchain: Returns CallbackHandler for LangChain tracing
openai: Returns wrapped OpenAI client with automatic tracing
sdk: Returns raw Langfuse client for manual instrumentation

Development notes

  • Added LangfuseTraceDataset to kedro_datasets_experimental.langfuse module
  • The dataset manages Langfuse credentials and sets required environment variables during initialisation
  • Implements read-only pattern as tracing clients are not data to be persisted
  • Added documentation
  • Updated RELEASE.md

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Updated jsonschema/kedro-catalog-X.XX.json if necessary
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

Signed-off-by: Sajid Alam <[email protected]>
Signed-off-by: Sajid Alam <[email protected]>
Signed-off-by: Sajid Alam <[email protected]>
Copy link
Contributor

@ElenaKhaustova ElenaKhaustova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks great!

Left two suggestions to double check.

@SajidAlamQB SajidAlamQB marked this pull request as ready for review October 10, 2025 12:17
Copy link
Contributor

@ElenaKhaustova ElenaKhaustova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Signed-off-by: Sajid Alam <[email protected]>
Signed-off-by: Sajid Alam <[email protected]>
@ravi-kumar-pilla ravi-kumar-pilla self-requested a review October 14, 2025 17:25
Copy link
Contributor

@ravi-kumar-pilla ravi-kumar-pilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @SajidAlamQB , Overall it looks good. I left few minor comments. Other than that it LGTM. Thank you

Signed-off-by: Sajid Alam <[email protected]>
@SajidAlamQB SajidAlamQB enabled auto-merge (squash) October 15, 2025 06:43
@SajidAlamQB SajidAlamQB merged commit bcaa38b into main Oct 15, 2025
17 checks passed
@SajidAlamQB SajidAlamQB deleted the feat/langfuse_trace_dataset branch October 15, 2025 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants