-
-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: [v0.6.5] Streaming tool call responses with the hermes template is inconsistent with the non-stream version. #11392
Comments
Could be related to this: {
"type": "function",
"function": {
"name": "getWorkspaceDirectoryStructure",
"description": "Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\n such as node_modules and hidden files, ensuring paths are within workspace boundaries."
}
}, it's known that the model sometimes has problems generating tool calls when parameters are not specified, and that the stream parser has problems when the arguments are empty or not present. Can you try adding a parameter to this function (even if just a dummy one) and see if that solves the problem? Obviously that's not a fix but will help us triage & ship a fix if that's the case |
Indeed, I can confirm that this issue is not related to that function. First, if we try using the completions API with the correct prompt directly as below: {
"model": "Qwen/Qwen2.5-7B-Instruct",
"prompt": "<|im_start|>system\n# Instructions\n\nYou are an AI assistant integrated into Theia IDE, designed to assist software developers with concise answers to programming-related questions. Your goal is to enhance\nproductivity with quick, relevant solutions, explanations, and best practices. Keep responses short, delivering valuable insights and direct solutions.\n\nUse the following functions to interact with the workspace files as needed:\n- **You can call function: getWorkspaceDirectoryStructure(): Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\n such as node_modules and hidden files, ensuring paths are within workspace boundaries.**: Returns the complete directory structure.\n- **You can call function: getWorkspaceFileList(path: string): List files and directories within a specified workspace directory. Paths are relative to the workspace root, and only workspace-contained paths are\n allowed. If no path is provided, the root contents are listed. Paths outside the workspace will result in an error.**: Lists files and directories in a specific directory.\n- **You can call function: getFileContent(file: string): The relative path to the target file within the workspace. This path is resolved from the workspace root, and only files within the workspace boundaries\n are accessible. Attempting to access paths outside the workspace will result in an error.**: Retrieves the content of a specific file.\n\n### Workspace Navigation Guidelines\n\n1. **Start at the Root**: For general questions (e.g., \"How to build the project\"), check root-level documentation files or setup files before browsing subdirectories.\n2. **Confirm Paths**: Always verify paths by listing directories or files as you navigate. Avoid assumptions based on user input alone.\n3. **Navigate Step-by-Step**: Move into subdirectories only as needed, confirming each directory level.\n\n### Response Guidelines\n\n1. **Contextual Focus**: Provide answers relevant to the workspace, avoiding general advice. Use provided functions without assuming file structure or content.\n2. **Clear Solutions**: Offer direct answers and concise explanations. Link to official documentation as needed.\n3. **Tool & Language Adaptability**: Adjust guidance based on the programming language, framework, or tool specified by the developer.\n4. **Supportive Tone**: Maintain a friendly, professional tone with clear, accurate technical language.\n5. **Stay Relevant**: Limit responses to software development, frameworks, Theia, terminal usage, and related technologies. Decline unrelated questions politely.\n\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"getWorkspaceDirectoryStructure\", \"description\": \"Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\\n such as node_modules and hidden files, ensuring paths are within workspace boundaries.\", \"parameters\": null}}\n{\"type\": \"function\", \"function\": {\"name\": \"getWorkspaceFileList\", \"description\": \"List files and directories within a specified workspace directory. Paths are relative to the workspace root, and only workspace-contained paths are\\n allowed. If no path is provided, the root contents are listed. Paths outside the workspace will result in an error.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"path\": {\"type\": \"string\", \"description\": \"Optional relative path to a directory within the workspace. If no path is specified, the function lists contents directly in the workspace\\n root. Paths are resolved within workspace boundaries only; paths outside the workspace or unvalidated paths will result in an error.\"}}}}}\n{\"type\": \"function\", \"function\": {\"name\": \"getFileContent\", \"description\": \"The relative path to the target file within the workspace. This path is resolved from the workspace root, and only files within the workspace boundaries\\n are accessible. Attempting to access paths outside the workspace will result in an error.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"file\": {\"type\": \"string\", \"description\": \"Return the content of a specified file within the workspace. The file path must be provided relative to the workspace root. Only files within\\n workspace boundaries are accessible; attempting to access files outside the workspace will return an error.\"}}}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nList all the files in my workspace <|im_end|>\n<|im_start|>assistant\n",
"temperature": 0.1,
"max_tokens": 1024
} We get {
"id": "cmpl-18d5ea583779406f955f6db94910b78f",
"object": "text_completion",
"created": 1734843753,
"model": "Qwen/Qwen2.5-7B-Instruct",
"choices": [
{
"index": 0,
"text": "<tool_call>\n{\"name\": \"getWorkspaceFileList\", \"arguments\": {}}\n</tool_call>",
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null,
"prompt_logprobs": null
}
],
"usage": {
"prompt_tokens": 942,
"total_tokens": 960,
"completion_tokens": 18,
"prompt_tokens_details": null
}
} Which has arguments correctly specified. Second, we could remove that function without parameters altogether from the request, and the issue persists. |
The tool parser works as expected. However, the stream response generator ignored the empty argument object. here is the tool parser log for your reference. @K-Mistele
|
Related #9874 |
Your current environment
Environment
Model Input Dumps
No response
🐛 Describe the bug
The streaming feature combined with tool calling is still quite bugged.
If we POST the following tool call generation request to vllm endpoint /v1/chat/completions:
We get
No arguments are returned. However if we use the none streaming version, we get:
The arguments disappeared from the streaming response.
Before submitting a new issue...
Probably related to #11279 , I provide a easy example here to check with the Qwen2.5-7B-Instruct model.
The text was updated successfully, but these errors were encountered: