Langchain::Assistant support for Google Gemini LLM

[andreibondarev]

Langchain::Assistant works with Langchain::LLM::GoogleGemini now. This PR includes a number of breaking changes:

• Changing the format of JSON tool annotations
• Splitting Langchain::Message class into Langchain::Messages::OpenAIMessage and Langchain::Messages::GoogleGeminiMessage

The Langchain::Assistant can be instantiate to use Google Gemini the following way, e.g.:

llm = Langchain::LLM::GoogleGemini.new(api_key: ENV["GOOGLE_GEMINI_API_KEY"])
thread = Langchain::Thread.new
assistant = Langchain::Assistant.new(
  llm: llm,
  thread: thread,
  instructions: "You are a News Assistant.",
  tools: [Langchain::Tool::NewsRetriever.new(api_key: ENV["NEWS_API_KEY"])]
)

assistant.add_message_and_run content:"What are the latest news from China?", auto_tool_execution: true

To add a message:

assistant.add_message content: "..."

To run the assistant

assistant.run

To run the assistant and auto execute tools

assistant.run auto_tool_execution: true

[mattlindsey]

Different Message types look good to me. Then I think you can put extract_openai_tool_call and extract_gemini_tool_call into them as extract_tool_call to avoid the conditionals in assistant.rb. (Later have extract_tool_calls for mutiple/parallel tools.)
And tool_role to avoid that conditional too, but looks like you're in the middle of doing that.

P.S. Seems you could put it all in OpenAIResponse and GeminiResponse, unless you want to limit it to Assistants.

[mattlindsey]

@andreibondarev Totally up to you, but if you merge my weather tool improvement (#602) and database tool improvement (#591) into google-gemini-assistant-support you can more easily confirm that they work and that handling of errors from tools is implemented how you want.

[palladius]

Just pinging the thread on the status. Seems like only tests are missing?

[andreibondarev]

Note to self: OpenAI support more JSON Schema options (like "examples" and "default" keys) compared to Gemini. I'm sure it'll change in the future.

[mattlindsey]

Missing "en" for English, and the table I found says "eg" is Unassigned.

[mattlindsey]

I think you mistakenly put the parameters for get_top_headlines country here. They should go below under news_retriever__get_top_headlines, and the language values listed in the docs should go here (https://newsapi.org/docs/endpoints/everything).

[andreibondarev]

@mattlindsey Nice find!

[mattlindsey]

You probably know, but missing some parameters like 'language', 'from', 'to',...
So currently if you ask, for example, for "Top headlines from Germany in English" the tool calls fails with unknown keyword: :language (ArgumentError) and leaves the Assistant in a bad state (missing role=tool messages) and every subsequent attempt fails like below (with OpenAI):

3.2.1 :014 > assistant.add_message_and_run content:"Now give the news in Germany?", auto_tool_execution: true
I, [2024-05-06T20:49:45.668612 #8222]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::OpenAI
OpenAI HTTP Error (spotted in ruby-openai 6.4.0): {"error"=>{"message"=>"An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_u4qPhNeJInxgjIgAQQ32AuE2", "type"=>"invalid_request_error", "param"=>"messages.[3].role", "code"=>nil}}

[mattlindsey]

I added language and en to parameter enum, but unfortunately it doesn't seem to like translating headlines to English:
"I couldn't find any latest news from Germany in English. Would you like me to search for news from a different country or on a different topic?"
Details:
"name"=>"news_retriever__get_top_headlines", "arguments"=>"{\"country\":\"de\",\"language\":\"en\",\"page_size\":5}" causes news api to return {\"status\":\"ok\",\"totalResults\":0,\"articles\":[]})

I also saw OpenAI sometimes not pass the ':language' parameter to the tool and try multiple times since the headlines returned weren't English. Wierd.

[mattlindsey]

Actually your code looks fine. Sorry. I think you have all of the correct parameters.

Although it does seem the llm tries to pass language to top_headlines even though it's not listed in allowed params. Try it: "Top headlines from Germany in English".

[mattlindsey]

Maybe you need to say if response.code != 200 || response.body.empty? (or maybe if response.body == "{}\n") since we're still getting that occasionally (for GoogleGemini), which causes exception later when you try to parse it.

[andreibondarev]

@palladius Could you please test it with GoogleVertexAI LLM?

llm = Langchain::LLM::GoogleVertexAI.new(project_id: ENV["GOOGLE_GEMINI_API_KEY"], region: "us-central-1")
thread = Langchain::Thread.new
assistant = Langchain::Assistant.new(
  llm: llm,
  thread: thread,
  instructions: "You are a News Assistant.",
  tools: [Langchain::Tool::NewsRetriever.new(api_key: ENV["NEWS_API_KEY"])]
)

assistant.add_message_and_run content:"What are the latest news from Italy?", auto_tool_execution: true

Don't worry about the specs failing, I'll fix them.

[palladius]

Hi, I got it to work with my personal account.

Occasionally I get an empty/nil return:

usr/local/google/home/ricc/git/langchainrb-pr513-v2/lib/langchain/llm/response/google_gemini_response.rb:18:in `tool_calls': undefined method `has_key?' for nil:NilClass (NoMethodError)

      if raw_response.dig("candidates", 0, "content", "parts", 0).has_key?("functionCall")
                                                                 ^^^^^^^^^

[andreibondarev]

Hi, I got it to work with my personal account.

Occasionally I get an empty/nil return:

usr/local/google/home/ricc/git/langchainrb-pr513-v2/lib/langchain/llm/response/google_gemini_response.rb:18:in `tool_calls': undefined method `has_key?' for nil:NilClass (NoMethodError)

      if raw_response.dig("candidates", 0, "content", "parts", 0).has_key?("functionCall")
                                                                 ^^^^^^^^^

I'm pretty confident that this happens when the quota is exceeded. I get this occasionally as well.

[palladius]

Awesome progress so far! Thanks for the hard work Andrei!

[palladius]

works liek a charm for me!