Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: [v0.6.5] Streaming tool call responses with the hermes template is inconsistent with the non-stream version. #11392

Open
1 task done
elementary-particle opened this issue Dec 21, 2024 · 4 comments · May be fixed by #11554
Labels
bug Something isn't working

Comments

@elementary-particle
Copy link

elementary-particle commented Dec 21, 2024

Your current environment

Environment
PyTorch version: 2.5.1
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

Python version: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct  4 2024, 13:27:36) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28
Is CUDA available: True

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-ml-py==12.560.30
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.46.2
[pip3] triton==3.1.0
[conda] blas                      1.0                         mkl    defaults
[conda] cuda-cudart               12.1.105                      0    nvidia
[conda] cuda-cupti                12.1.105                      0    nvidia
[conda] cuda-libraries            12.1.0                        0    nvidia
[conda] cuda-nvrtc                12.1.105                      0    nvidia
[conda] cuda-nvtx                 12.1.105                      0    nvidia
[conda] cuda-opencl               12.6.77                       0    nvidia
[conda] cuda-runtime              12.1.0                        0    nvidia
[conda] cuda-version              12.6                          3    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libcublas                 12.1.0.26                     0    nvidia
[conda] libcufft                  11.0.2.4                      0    nvidia
[conda] libcufile                 1.11.1.6                      0    nvidia
[conda] libcurand                 10.3.7.77                     0    nvidia
[conda] libcusolver               11.4.4.55                     0    nvidia
[conda] libcusparse               12.0.2.55                     0    nvidia
[conda] libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
[conda] libnpp                    12.0.2.50                     0    nvidia
[conda] libnvjitlink              12.1.105                      0    nvidia
[conda] libnvjpeg                 12.1.1.14                     0    nvidia
[conda] mkl                       2023.1.0         h213fc3f_46344    defaults
[conda] mkl-service               2.4.0           py312h5eee18b_1    defaults
[conda] mkl_fft                   1.3.11          py312h5eee18b_0    defaults
[conda] mkl_random                1.2.8           py312h526ad5a_0    defaults
[conda] numpy                     1.26.4          py312hc5e2394_0    defaults
[conda] numpy-base                1.26.4          py312h0da6c21_0    defaults
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] pytorch                   2.5.1           py3.12_cuda12.1_cudnn9.1.0_0    pytorch
[conda] pytorch-cuda              12.1                 ha16c6d3_6    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torchaudio                2.5.1               py312_cu121    pytorch
[conda] torchtriton               3.1.0                     py312    pytorch
[conda] torchvision               0.20.1              py312_cu121    pytorch
[conda] transformers              4.46.2                   pypi_0    pypi

vLLM Version: 0.6.5
vLLM Build Flags:

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/home/rocky/miniconda3/envs/llm/lib/python3.12/site-packages/cv2/../../lib64:
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

The streaming feature combined with tool calling is still quite bugged.

If we POST the following tool call generation request to vllm endpoint /v1/chat/completions:

{
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "messages": [
    {
      "role": "system",
      "content": "# Instructions\n\nYou are an AI assistant integrated into Theia IDE, designed to assist software developers with concise answers to programming-related questions. Your goal is to enhance\nproductivity with quick, relevant solutions, explanations, and best practices. Keep responses short, delivering valuable insights and direct solutions.\n\nUse the following functions to interact with the workspace files as needed:\n- **You can call function: getWorkspaceDirectoryStructure(): Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\n            such as node_modules and hidden files, ensuring paths are within workspace boundaries.**: Returns the complete directory structure.\n- **You can call function: getWorkspaceFileList(path: string): List files and directories within a specified workspace directory. Paths are relative to the workspace root, and only workspace-contained paths are\n             allowed. If no path is provided, the root contents are listed. Paths outside the workspace will result in an error.**: Lists files and directories in a specific directory.\n- **You can call function: getFileContent(file: string): The relative path to the target file within the workspace. This path is resolved from the workspace root, and only files within the workspace boundaries\n             are accessible. Attempting to access paths outside the workspace will result in an error.**: Retrieves the content of a specific file.\n\n### Workspace Navigation Guidelines\n\n1. **Start at the Root**: For general questions (e.g., \"How to build the project\"), check root-level documentation files or setup files before browsing subdirectories.\n2. **Confirm Paths**: Always verify paths by listing directories or files as you navigate. Avoid assumptions based on user input alone.\n3. **Navigate Step-by-Step**: Move into subdirectories only as needed, confirming each directory level.\n\n### Response Guidelines\n\n1. **Contextual Focus**: Provide answers relevant to the workspace, avoiding general advice. Use provided functions without assuming file structure or content.\n2. **Clear Solutions**: Offer direct answers and concise explanations. Link to official documentation as needed.\n3. **Tool & Language Adaptability**: Adjust guidance based on the programming language, framework, or tool specified by the developer.\n4. **Supportive Tone**: Maintain a friendly, professional tone with clear, accurate technical language.\n5. **Stay Relevant**: Limit responses to software development, frameworks, Theia, terminal usage, and related technologies. Decline unrelated questions politely.\n"
    },
    {
      "role": "user",
      "content": "List all the files in my workspace "
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "getWorkspaceDirectoryStructure",
        "description": "Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\n            such as node_modules and hidden files, ensuring paths are within workspace boundaries."
      }
    },
    {
      "type": "function",
      "function": {
        "name": "getWorkspaceFileList",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {
              "type": "string",
              "description": "Optional relative path to a directory within the workspace. If no path is specified, the function lists contents directly in the workspace\n                         root. Paths are resolved within workspace boundaries only; paths outside the workspace or unvalidated paths will result in an error."
            }
          }
        },
        "description": "List files and directories within a specified workspace directory. Paths are relative to the workspace root, and only workspace-contained paths are\n             allowed. If no path is provided, the root contents are listed. Paths outside the workspace will result in an error."
      }
    },
    {
      "type": "function",
      "function": {
        "name": "getFileContent",
        "parameters": {
          "type": "object",
          "properties": {
            "file": {
              "type": "string",
              "description": "Return the content of a specified file within the workspace. The file path must be provided relative to the workspace root. Only files within\n                         workspace boundaries are accessible; attempting to access files outside the workspace will return an error."
            }
          }
        },
        "description": "The relative path to the target file within the workspace. This path is resolved from the workspace root, and only files within the workspace boundaries\n             are accessible. Attempting to access paths outside the workspace will result in an error."
      }
    }
  ],
  "tool_choice": "auto",
  "stream": true
}

We get

{"id":"chatcmpl-095ddb0721564d69984d67ed51f23e4d","object":"chat.completion.chunk","created":1734758880,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
{"id":"chatcmpl-095ddb0721564d69984d67ed51f23e4d","object":"chat.completion.chunk","created":1734758880,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-6b879898e4bf4d27bf01269db8c5222d","type":"function","index":0,"function":{"name":"getWorkspaceFileList"}}]},"logprobs":null,"finish_reason":null}]}
{"id":"chatcmpl-095ddb0721564d69984d67ed51f23e4d","object":"chat.completion.chunk","created":1734758880,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":"tool_calls","stop_reason":null}]}

No arguments are returned. However if we use the none streaming version, we get:

{
  "id": "chatcmpl-ede5dbc93dc44f859fea8fff3f10086b",
  "object": "chat.completion",
  "created": 1734762064,
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "chatcmpl-tool-dc87e34d9d5445a5bc3a3aa94efbabc0",
            "type": "function",
            "function": {
              "name": "getWorkspaceFileList",
              "arguments": "{}"
            }
          }
        ]
      },
      "logprobs": null,
      "finish_reason": "tool_calls",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 942,
    "total_tokens": 960,
    "completion_tokens": 18,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null
}

The arguments disappeared from the streaming response.

Before submitting a new issue...

Probably related to #11279 , I provide a easy example here to check with the Qwen2.5-7B-Instruct model.

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@elementary-particle elementary-particle added the bug Something isn't working label Dec 21, 2024
@elementary-particle elementary-particle changed the title [Bug]: Streaming tool call responses with the hermes template is STILL inconsistent with the non-stream version in v0.6.5. [Bug]: [v0.6.5] Streaming tool call responses with the hermes template is inconsistent with the non-stream version. Dec 21, 2024
@K-Mistele
Copy link
Contributor

K-Mistele commented Dec 22, 2024

Could be related to this:

    {
      "type": "function",
      "function": {
        "name": "getWorkspaceDirectoryStructure",
        "description": "Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\n            such as node_modules and hidden files, ensuring paths are within workspace boundaries."
      }
    },

it's known that the model sometimes has problems generating tool calls when parameters are not specified, and that the stream parser has problems when the arguments are empty or not present. Can you try adding a parameter to this function (even if just a dummy one) and see if that solves the problem?

Obviously that's not a fix but will help us triage & ship a fix if that's the case

@elementary-particle
Copy link
Author

Indeed, I can confirm that this issue is not related to that function.

First, if we try using the completions API with the correct prompt directly as below:

{
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "prompt": "<|im_start|>system\n# Instructions\n\nYou are an AI assistant integrated into Theia IDE, designed to assist software developers with concise answers to programming-related questions. Your goal is to enhance\nproductivity with quick, relevant solutions, explanations, and best practices. Keep responses short, delivering valuable insights and direct solutions.\n\nUse the following functions to interact with the workspace files as needed:\n- **You can call function: getWorkspaceDirectoryStructure(): Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\n            such as node_modules and hidden files, ensuring paths are within workspace boundaries.**: Returns the complete directory structure.\n- **You can call function: getWorkspaceFileList(path: string): List files and directories within a specified workspace directory. Paths are relative to the workspace root, and only workspace-contained paths are\n             allowed. If no path is provided, the root contents are listed. Paths outside the workspace will result in an error.**: Lists files and directories in a specific directory.\n- **You can call function: getFileContent(file: string): The relative path to the target file within the workspace. This path is resolved from the workspace root, and only files within the workspace boundaries\n             are accessible. Attempting to access paths outside the workspace will result in an error.**: Retrieves the content of a specific file.\n\n### Workspace Navigation Guidelines\n\n1. **Start at the Root**: For general questions (e.g., \"How to build the project\"), check root-level documentation files or setup files before browsing subdirectories.\n2. **Confirm Paths**: Always verify paths by listing directories or files as you navigate. Avoid assumptions based on user input alone.\n3. **Navigate Step-by-Step**: Move into subdirectories only as needed, confirming each directory level.\n\n### Response Guidelines\n\n1. **Contextual Focus**: Provide answers relevant to the workspace, avoiding general advice. Use provided functions without assuming file structure or content.\n2. **Clear Solutions**: Offer direct answers and concise explanations. Link to official documentation as needed.\n3. **Tool & Language Adaptability**: Adjust guidance based on the programming language, framework, or tool specified by the developer.\n4. **Supportive Tone**: Maintain a friendly, professional tone with clear, accurate technical language.\n5. **Stay Relevant**: Limit responses to software development, frameworks, Theia, terminal usage, and related technologies. Decline unrelated questions politely.\n\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"getWorkspaceDirectoryStructure\", \"description\": \"Retrieve the complete directory structure of the workspace, listing only directories (no file contents). This structure excludes specific directories,\\n            such as node_modules and hidden files, ensuring paths are within workspace boundaries.\", \"parameters\": null}}\n{\"type\": \"function\", \"function\": {\"name\": \"getWorkspaceFileList\", \"description\": \"List files and directories within a specified workspace directory. Paths are relative to the workspace root, and only workspace-contained paths are\\n             allowed. If no path is provided, the root contents are listed. Paths outside the workspace will result in an error.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"path\": {\"type\": \"string\", \"description\": \"Optional relative path to a directory within the workspace. If no path is specified, the function lists contents directly in the workspace\\n                         root. Paths are resolved within workspace boundaries only; paths outside the workspace or unvalidated paths will result in an error.\"}}}}}\n{\"type\": \"function\", \"function\": {\"name\": \"getFileContent\", \"description\": \"The relative path to the target file within the workspace. This path is resolved from the workspace root, and only files within the workspace boundaries\\n             are accessible. Attempting to access paths outside the workspace will result in an error.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"file\": {\"type\": \"string\", \"description\": \"Return the content of a specified file within the workspace. The file path must be provided relative to the workspace root. Only files within\\n                         workspace boundaries are accessible; attempting to access files outside the workspace will return an error.\"}}}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nList all the files in my workspace <|im_end|>\n<|im_start|>assistant\n",
  "temperature": 0.1,
  "max_tokens": 1024
}

We get

{
  "id": "cmpl-18d5ea583779406f955f6db94910b78f",
  "object": "text_completion",
  "created": 1734843753,
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "text": "<tool_call>\n{\"name\": \"getWorkspaceFileList\", \"arguments\": {}}\n</tool_call>",
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "prompt_logprobs": null
    }
  ],
  "usage": {
    "prompt_tokens": 942,
    "total_tokens": 960,
    "completion_tokens": 18,
    "prompt_tokens_details": null
  }
}

Which has arguments correctly specified.

Second, we could remove that function without parameters altogether from the request, and the issue persists.

@ATAKing1023
Copy link

The tool parser works as expected. However, the stream response generator ignored the empty argument object.
the stream response is {'name': 'sql_db_list_tables'} rather than {'name': 'sql_db_list_tables', 'arguments': {}}

here is the tool parser log for your reference. @K-Mistele

DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: <tool_call>
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [151657]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:188] Starting on a new tool 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call None
DEBUG 12-25 13:14:26 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text:
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [198]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:242] not enough tokens to parse into JSON yet
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: {"
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [4913]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: name
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [606]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: ":
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [788]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text:  "
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [330]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: sql
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [3544]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: _db
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [8685]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: _list
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [2019]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: _tables
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [35632]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {}
DEBUG 12-25 13:14:26 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: ",
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [497]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables'}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text:  "
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [330]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables'}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:279] Trying to parse current tool call with ID 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:293] diffing old arguments: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:294] against new ones: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298] Skipping text  " - no arguments
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: arguments
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [16370]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables'}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:279] Trying to parse current tool call with ID 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:293] diffing old arguments: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:294] against new ones: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298] Skipping text arguments - no arguments
DEBUG 12-25 13:14:26 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: ":
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [788]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables'}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:279] Trying to parse current tool call with ID 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:293] diffing old arguments: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:294] against new ones: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298] Skipping text ": - no arguments
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text:  {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [4687]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables', 'arguments': {}}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:279] Trying to parse current tool call with ID 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:293] diffing old arguments: None
DEBUG 12-25 13:14:26 hermes_tool_parser.py:294] against new ones: {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298] Skipping text  {} - no arguments
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: }
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [532]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables', 'arguments': {}}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:279] Trying to parse current tool call with ID 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:293] diffing old arguments: {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:294] against new ones: {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298] Skipping text }
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298]  - no arguments
DEBUG 12-25 13:14:26 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text: </tool_call>
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [151658]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:154] tool_call_end_token in delta_text
DEBUG 12-25 13:14:26 hermes_tool_parser.py:240] Parsed tool call {'name': 'sql_db_list_tables', 'arguments': {}}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:279] Trying to parse current tool call with ID 0
DEBUG 12-25 13:14:26 hermes_tool_parser.py:293] diffing old arguments: {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:294] against new ones: {}
DEBUG 12-25 13:14:26 hermes_tool_parser.py:298] Skipping text  - no arguments
DEBUG 12-25 13:14:26 hermes_tool_parser.py:124] delta_text:
DEBUG 12-25 13:14:26 hermes_tool_parser.py:125] delta_token_ids: [151645]
DEBUG 12-25 13:14:26 hermes_tool_parser.py:150] Generating text content! skipping tool parsing.

@K-Mistele
Copy link
Contributor

Related #9874

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
3 participants