agent short & long term memory with langgraph.

[lkk12014402]

Description

integrate llama stack implementations of the agent memory, refer https://github.com/meta-llama/llama-stack/blob/main/llama_stack/providers/impls/meta_reference/agents/agents.py#L27

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
comps/cores/proto/agents/agents.py	0.00%	8 Missing ⚠️
comps/cores/proto/agents/__init__.py	0.00%	1 Missing ⚠️

Files with missing lines	Coverage Δ
comps/cores/proto/agents/__init__.py	0.00% <0.00%> (ø)
comps/cores/proto/agents/agents.py	0.00% <0.00%> (ø)

[minmin-intel]

A few thoughts from my side:

1. Langgraph has "checkpoint" and "store" for short-term and long-term memory, both have in-memory and also SQL db based implementations. We can utilize those langgraph APIs. https://langchain-ai.github.io/langgraph/concepts/persistence/, https://langchain-ai.github.io/langgraph/reference/checkpoints/, https://langchain-ai.github.io/langgraph/reference/store/
2. What and how to save/retrieve/use memories: depend on different agent designs, but we should provide common functions, for example, save agent_id, thread_id, messages - these will be part of our assistants APIs.
3. Ultimately (may be not in v1.1 release), the memories will be microservices, and agents send requests to memory DBs. Your diagram captured that, which I agree. Right now, langgraph implemented postgres sql db based checkpointers and stores. Langgraph also has an example showing vector db as memory store. We need to decide which DBs to support and the interface.
4. We should expose a unified interface to send requests to memory microservices and process the returns from memory microservices, so that from agent developer point of view, they just need to use this unified interface with the memory endpoint url, agent_id, thread_id, and content.

[lkk12014402]

hi, minmin @minmin-intel As our discussion with chendi, I would like to integrate langgraph memory implementations.

langgraph implements the short-term & long-term memory:

see this reference: https://langchain-ai.github.io/langgraph/concepts/persistence/

1. short-term memory
   with the langgraph checkpointer, the state can be written by a checkpointer to a thread at each graph step, enabling state persistence

2. long-term memory
   with checkpointers alone, we cannot share information across threads. This motivates the need for the Store interface

And,
for the checkpointer, langgraph has 4 implementations:

1. libs/checkpoint/langgraph/checkpoint
2. libs/checkpoint-postgres/langgraph/checkpoint/postgres
3. libs/checkpoint-sqlite/langgraph/checkpoint/sqlite
4. libs/checkpoint-duckdb/langgraph/checkpoint/duckdb

for the Store, langgraph has 3 implementations:

1. libs/checkpoint/langgraph/store
2. libs/checkpoint-postgres/langgraph/store/postgres
3. libs/checkpoint-duckdb/langgraph/store/duckdb

I will integrate the libs/checkpoint/langgraph/checkpoint and libs/checkpoint/langgraph/store first for the version 1.1

[lkk12014402]

test short-term memory

1. build docker image

docker build -t opea/agent-langchain:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/agent/langchain/Dockerfile .

2. setup agent with -e with_memory=true

docker run --runtime=runc --name="comps-langchain-agent-endpoint" -v $WORKPATH/comps/agent/langchain/tools:/home/user/comps/agent/langchain/tools -p 9090:9090 --ipc=host -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} -e model=${LLM_MODEL_ID} -e ip_address=${ip_address} -e strategy=react_llama -e llm_endpoint_url=${llm_endpoint_url} -e llm_engine=tgi -e recursion_limit=5 -e require_human_feedback=false -e tools=/home/user/comps/agent/langchain/tools/custom_tools.yaml -e streaming=false -e with_memory=true opea/agent-langchain:latest

3. test the agent like the langgraph example

curl http://localhost:9090/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
     "query": "hi! I am bob"
    }'

# with enabling the memory, the agent knows the user name `bob`

# then ask the agent

curl http://localhost:9090/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
     "query": "what is my name?"
    }'

[lkk12014402]

add timeout handling for LLM response, which can return Request timed out.

we can also add node-retry, like this: https://github.com/langchain-ai/langgraph/blob/main/docs/docs/how-tos/node-retries.ipynb

https://github.com/langchain-ai/langgraph/blob/main/docs/docs/how-tos/tool-calling-errors.ipynb

[lkk12014402]

need discuss the Improve Assistant api with ‘tool’ keyword

[lkk12014402]

if not set -e HABANA_VISIBLE_DEVICES=all, the ut will fail when start opea/vllm:hpu with RuntimeError: synStatus=8 [Device not found] Device acquire failed. No devices found.

[xuechendi]

if not set -e HABANA_VISIBLE_DEVICES=all, the ut will fail when start opea/vllm:hpu with RuntimeError: synStatus=8 [Device not found] Device acquire failed. No devices found.

This is interesting, is this only happened on specific CI node? or to certain docker runtime version?

[minmin-intel]

So for v1.1 timeout only applies to waiting for LLM response. Can we add timeout for tools in later release?

[lkk12014402]

will comfirm

[minmin-intel]

What is the relationship between user_id and assistant_id?

[lkk12014402]

will comfirm

[minmin-intel]

I think instantiate_agent when microservice starts makes sense when it is chat_completion, but does not quite make sense when it is assistants api. Shall we initiate agent only when user send a create_assistant request? And even then, we are not materializing the agent, but instead only record the configs (like llama-stack create_agent), the agent is then materialized later when user send request to the thread api (like llama-stack get_agent).

The benefits of such an approach: one microservice can support multiple configs, which means multiple different types of agents, instead of just one config. So this is more scalable.