Langfuse-Not recording llamaindex response. Please help

[mraguth]

I am using local langfuse. It is recording sessions and tockens for this example fine.

# Example context, thx ChatGPT
from llama_index.core import VectorStoreIndex
from llama_index.core import Document

doc1 = Document(text="""
Maxwell "Max" Silverstein, a lauded movie director, screenwriter, and producer, was born on October 25, 1978, in Boston, Massachusetts. A film enthusiast from a young age, his journey began with home movies shot on a Super 8 camera. His passion led him to the University of Southern California (USC), majoring in Film Production. Eventually, he started his career as an assistant director at Paramount Pictures. Silverstein's directorial debut, “Doors Unseen,” a psychological thriller, earned him recognition at the Sundance Film Festival and marked the beginning of a successful directing career.
""")
doc2 = Document(text="""
Throughout his career, Silverstein has been celebrated for his diverse range of filmography and unique narrative technique. He masterfully blends suspense, human emotion, and subtle humor in his storylines. Among his notable works are "Fleeting Echoes," "Halcyon Dusk," and the Academy Award-winning sci-fi epic, "Event Horizon's Brink." His contribution to cinema revolves around examining human nature, the complexity of relationships, and probing reality and perception. Off-camera, he is a dedicated philanthropist living in Los Angeles with his wife and two children.
""")

# Example index construction + LLM query

index = VectorStoreIndex.from_documents([doc1, doc2])

# Chat
response = index.as_chat_engine().chat("What did he do growing up?")
print(response)

However, if I use my own llamaindex, like below, I can not see any traces. Do I need to configure different? please help.

response = query_engine.query(
    "Sitename 'xxxx' month over month progress in Total Cases\n"
    "Please write your insights in 3 major points and add all needed datapoints to the sitename.\n"

)

[dosubot[bot]]

Yes, you need to configure your environment variables to see traces in Langfuse when using your own llamaindex. Specifically, you need to set the LANGFUSE_SECRET_KEY, LANGFUSE_PUBLIC_KEY, and LANGFUSE_HOST environment variables as shown in your Langfuse project settings.

Here is the relevant code snippet:

from llama_index.core import set_global_handler

# Make sure you've installed the 'llama-index-callbacks-langfuse' integration package.

# NOTE: Set your environment variables 'LANGFUSE_SECRET_KEY', 'LANGFUSE_PUBLIC_KEY' and 'LANGFUSE_HOST'
# as shown in your langfuse.com project settings.

set_global_handler("langfuse")

Additionally, observability in LlamaIndex is now handled via the instrumentation module (available in v0.10.20 and later). This allows you to seamlessly integrate the LlamaIndex library with powerful observability/evaluation tools offered by partners like Langfuse.

For a more detailed setup, including installing necessary packages and configuring the environment, you can refer to the Langfuse Callback Handler documentation ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu. Not working... I am not sure this might be due to my query_engine configuration:

query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    pandas_prompt= pandas_prompt,
    pandas_output_parser= pandas_output_parser,
    llm=llm,
    qa_prompt=qa_prompt,
    synthesize_response=True,
    response_synthesis_prompt=response_synthesis_prompt,
)
response = query_engine.query("Explain the Sitenames in the dataset with 7 lines of insights")
print(response)

I am seeing only embeddings like shown in the screenshot:

[dosubot[bot]]

To ensure your RAGStringQueryEngine setup records traces properly in Langfuse, follow these steps:

1. Install the required packages:

   %pip install llama-index llama-index-callbacks-langfuse

2. Set the required environment variables: LANGFUSE_SECRET_KEY, LANGFUSE_PUBLIC_KEY, and LANGFUSE_HOST as shown in your Langfuse project settings.

   import os
   
   os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
   os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
   os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"  # 🇪🇺 EU region, 🇺🇸 US region: "https://us.cloud.langfuse.com"
   os.environ["OPENAI_API_KEY"] = "sk-..."

3. Register the Langfuse callback handler:

   from llama_index.core import Settings
   from llama_index.core.callbacks import CallbackManager
   from langfuse.llama_index import LlamaIndexCallbackHandler
   
   langfuse_callback_handler = LlamaIndexCallbackHandler()
   Settings.callback_manager = CallbackManager([langfuse_callback_handler])

4. Ensure all events are flushed to Langfuse before exiting your application:

   # ... your LlamaIndex calls here ...
   
   langfuse_callback_handler.flush()

Here is an example of how to use the setup with RAGStringQueryEngine:

from llama_index.core import Settings, VectorStoreIndex
from llama_index.core.callbacks import CallbackManager
from langfuse.llama_index import LlamaIndexCallbackHandler

langfuse_callback_handler = LlamaIndexCallbackHandler()
Settings.callback_manager = CallbackManager([langfuse_callback_handler])

query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    pandas_prompt=pandas_prompt,
    pandas_output_parser=pandas_output_parser,
    llm=llm,
    qa_prompt=qa_prompt,
    synthesize_response=True,
    response_synthesis_prompt=response_synthesis_prompt,
)

response = query_engine.query("Explain the Sitenames in the dataset with 7 lines of insights")
print(response)

# Flush the callback handler to see results in Langfuse
langfuse_callback_handler.flush()

Ensure that all these steps are correctly followed. If traces are still not being recorded, double-check the environment variables and confirm that they are correctly set and accessible in your runtime environment ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu Langfuse is working with regular index quries. However, it is not producing any trace for RAGStringQueryEngine like example below:

langfuse_callback_handler.flush()
query_engine = RAGStringQueryEngine(retriever=recursive_retriever, response_synthesizer=response_synthesizer, llm=llm,qa_prompt=qa_prompt,)

query_response = query_engine.query("Describe the dataset")
print(query_response)
langfuse_callback_handler.flush()

[dosubot[bot]]

To ensure that RAGStringQueryEngine records traces in Langfuse, you need to explicitly connect it to the CallbackManager. Here is the updated code snippet:

import os

os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
os.environ["OPENAI_API_KEY"] = "sk-..."

from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager
from langfuse.llama_index import LlamaIndexCallbackHandler

langfuse_callback_handler = LlamaIndexCallbackHandler()
Settings.callback_manager = CallbackManager([langfuse_callback_handler])

# Ensure the query engine uses the callback manager
query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    llm=llm,
    qa_prompt=qa_prompt,
    callback_manager=Settings.callback_manager  # Add this line
)

langfuse_callback_handler.flush()
query_response = query_engine.query("Describe the dataset")
print(query_response)
langfuse_callback_handler.flush()

By adding the callback_manager=Settings.callback_manager parameter to the RAGStringQueryEngine initialization, you ensure that the query engine is properly connected to the CallbackManager and thus to the LangfuseCallbackHandler. This should enable the recording of traces for RAGStringQueryEngine queries ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu Perfect. It worked

[dosubot[bot]]

Great to hear that it worked! If you have any more questions or need further assistance, feel free to ask. Happy coding!

To continue talking to Dosu, mention @dosu.