Agent gets confused when declining use of tool

[marcelklehr]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

---

Steps to reproduce

Expected behaviour

Tool should not be called but agent should still be functional

Actual behaviour

Agent gets confused:

Server configuration

Web server: php -S
Database: SQLite
PHP version: 8.3
Nextcloud version: master / 32

[janepie]

What was the generated prompt for this?

[marcelklehr]

Should mention that this was using llm2 and the first response failed, so may be hard to reproduce :/

[marcelklehr]

{"system_prompt":"\nYou are a helpful AI assistant with access to tools, please respond to the user's query to the best of your ability, using the provided tools if necessary. If no tool is needed to provide a correct answer, do not use one. If you used a tool, you still need to convey its output to the user.\nUse the same language for your answers as the user used in their message.\nToday is 2025-07-31.\nIntuit the language the user is using (there is no tool for this, you will need to guess). Reply in the language intuited. Do not output the language you intuited.\nOnly use tools if you cannot answer the user without them.\nOnly use the duckduckgo_results_json tool if the user explicitly asks for a web search.\nYou can check which conversations exist using the list_talk_conversations tool, if a conversation cannot be found.\nYou can check which calendars exist using the list_calendars tool, if a calendar can not be found.\nyou can find out a user's email address and location by using the find_person_in_contacts tool.\nyou can find out the current user's location by using the find_details_of_current_user tool.\nIf an item should be added to a list, check list_calendars for a fitting calendar and add the item as a task there.\nIf you get a link as a tool output, always add the link to your response.\n","input":"","tool_message":"[{\"name\": null, \"content\": \"API call denied by user. Reasoning: 'Can you use the ask_question tool instead?'. Continue assisting, accounting for the user's input.\",.....

it seems to return the user's new input as a tool message

[janepie]

Yes, because with not accepting you "deny" the tool use. We can maybe update the system prompt with what to do in case of user-denied tool calls

[marcelklehr]

What if we put the user's new input into the input field and only mention in the tool message that the tool call was denied?

[janepie]

Yeah, why not, then we can get rid of the reasoning field in the tool message I guess?