evalscope perf 测试sglang 部署的openai api server 无法输出结果

[hetian127]

版本：evalscope 0.5.3
sglang 0.3.0

在本地起了一个sglang的openai api server，命令如下：
CUDA_VISIBLE_DEVICES=4,5,6,7 python -m sglang.launch_server --model-path /local/models/Qwen2-72B-Instruct --tp 4 --host 0.0.0.0
能够正常访问，用下面的语句进行benchmark测试：
evalscope perf --url 'http://localhost:30000/v1/chat/completions' --parallel 128 --model '/local/models/Qwen2-72B-Instruct' --log-every-n-query 10 --read-timeout=120 --dataset-path './dataset/open_qa.jsonl' -n 50 --max-prompt-length 128000 --api openai --stream --stop '<|im_end|>' --dataset openqa --debug

server端有输出：

测试脚本到这个位置就不动了

同样的命令在测试vllm时可以正常输出结果，但是测试sglang时无法生成结果。
麻烦帮忙看看问题出在哪里，多谢

[Yunnglin]

我们后面复现一下

[Yunnglin]

您可以拉取main分支代码，来使用最新的perf模块尝试，参考使用指南。

[xinyang920]

测试mindie时遇到同样的问题

[Yunnglin]

测试mindie时遇到同样的问题

麻烦提供一下执行命令和--debug模式的控制台输出

[tghfly]

我在使用evalscope 最新版本 0.8.1压测sglang部署的大模型报如下错误：
我执行的命令及报错如下：

# evalscope perf --url "http://192.168.171.166:21840/v1/chat/completions" --parallel 1 --model Qwen2.5-14B-Instruct --number 30 --max-tokens 2048 --temperature 0.7 --api openai --dataset openqa --stream 

报错信息如下：
2024-12-25 09:57:13,439 - evalscope - http_client.py - _handle_stream - 52 - DEBUG - Response recevied: data: [DONE]
Processing: 0it [00:01, ?it/s]
2024-12-25 09:57:13,440 - evalscope - handler.py - async_wrapper - 19 - ERROR - Exception in async function 'statistic_benchmark_metric_worker': 'object'
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/benchmark.py", line 167, in statistic_benchmark_metric_worker
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/utils/benchmark_util.py", line 86, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/utils/benchmark_util.py", line 43, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/home/user/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/perf/plugin/api/openai_api.py", line 116, in parse_responses
    if js['object'] == 'chat.completion':
KeyError: 'object'

问题应该是使用了stream流式输出导致的问题，不启用流式输出压测是正常的。

[Yunnglin]

我们复现并修复一下

[tghfly]

这是因为sglang的api返回有这种没带object的数据 data: {"id":"89467d81d1ea4435b74a34651faeedb4","model":"Qwen2.5-14B-Instruct","choices":[],"usage":{"prompt_tokens":44,"total_tokens":328,"completion_tokens":284}} 这种没有带object需要处理一下，我改了一下perf/plugin/api/openai_api.py文件中的parse_responses方法，判断接收的数据是否包含object属性，没有的话补一下就可以了。

    def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
        """Parser responses and return number of request and response tokens.
           sample of the output delta:
           {"id":"4","object":"chat.completion.chunk","created":1714030870,"model":"llama3","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}


        Args:
            responses (List[bytes]): List of http response body, for stream output,
                there are multiple responses, for general only one.
            kwargs: (Any): The command line --parameter content.
        Returns:
            Tuple: Return number of prompt token and number of completion tokens.
        """
        full_response_content = ''
        delta_contents = {}
        input_tokens = None
        output_tokens = None
        for response in responses:
            js = json.loads(response)
           # 判断object是否存在
            if 'object' not in js:
                logger.warning(f"Response does not contain 'object' field: {js}")
                js['object'] = 'chat.completion.chunk'
            #if 'created' not in js:
            #    logger.warning(f"Response does not contain 'created' field: {js}")
            #    js['created'] = int(time.time())
            if js['object'] == 'chat.completion':
                for choice in js['choices']:
                    delta_contents[choice['index']] = [choice['message']['content']]
                input_tokens = js['usage']['prompt_tokens']
                output_tokens = js['usage']['completion_tokens']
            elif js['object'] == 'text_completion':
                for choice in js['choices']:
                    delta_contents[choice['index']] = [choice['text']]
                input_tokens = js['usage']['prompt_tokens']
                output_tokens = js['usage']['completion_tokens']
            elif js['object'] == 'chat.completion.chunk':
                if 'choices' in js:
                    for choice in js['choices']:
                        if 'delta' in choice and 'index' in choice:
                            delta = choice['delta']
                            idx = choice['index']
                            if 'content' in delta:
                                delta_content = delta['content']
                                if idx in delta_contents:
                                    delta_contents[idx].append(delta_content)
                                else:
                                    delta_contents[idx] = [delta_content]
                # usage in chunk: {"id":"","object":"chat.completion.chunk","created":1718269986,"model":"llama3",
                # "choices":[],"usage":{"prompt_tokens":32,"total_tokens":384,"completion_tokens":352}}
                if 'usage' in js and js['usage']:
                    input_tokens = js['usage']['prompt_tokens']
                    output_tokens = js['usage']['completion_tokens']
        if (input_tokens is None and output_tokens is None and self.tokenizer is not None):
            input_tokens = 0
            output_tokens = 0
            for idx, choice_contents in delta_contents.items():
                full_response_content = ''.join([m for m in choice_contents])
                input_tokens += len(self.tokenizer.encode(request['messages'][0]['content']))
                output_tokens += len(self.tokenizer.encode(full_response_content))
        elif input_tokens is None and output_tokens is None:  # no usage info get.
            input_tokens = 0
            output_tokens = 0
            logger.warning('No usage information found. Please specify `--tokenizer-path` to generate usage details.')

        return input_tokens, output_tokens

[yingdachen]

这是因为sglang的api返回有这种没带object的数据 data: {"id":"89467d81d1ea4435b74a34651faeedb4","model":"Qwen2.5-14B-Instruct","choices":[],"usage":{"prompt_tokens":44,"total_tokens":328,"completion_tokens":284}} 这种没有带object需要处理一下，我改了一下perf/plugin/api/openai_api.py文件中的parse_responses方法，判断接收的数据是否包含object属性，没有的话补一下就可以了。

    def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
        """Parser responses and return number of request and response tokens.
           sample of the output delta:
           {"id":"4","object":"chat.completion.chunk","created":1714030870,"model":"llama3","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}


        Args:
            responses (List[bytes]): List of http response body, for stream output,
                there are multiple responses, for general only one.
            kwargs: (Any): The command line --parameter content.
        Returns:
            Tuple: Return number of prompt token and number of completion tokens.
        """
        full_response_content = ''
        delta_contents = {}
        input_tokens = None
        output_tokens = None
        for response in responses:
            js = json.loads(response)
           # 判断object是否存在
            if 'object' not in js:
                logger.warning(f"Response does not contain 'object' field: {js}")
                js['object'] = 'chat.completion.chunk'
            #if 'created' not in js:
            #    logger.warning(f"Response does not contain 'created' field: {js}")
            #    js['created'] = int(time.time())
            if js['object'] == 'chat.completion':
                for choice in js['choices']:
                    delta_contents[choice['index']] = [choice['message']['content']]
                input_tokens = js['usage']['prompt_tokens']
                output_tokens = js['usage']['completion_tokens']
            elif js['object'] == 'text_completion':
                for choice in js['choices']:
                    delta_contents[choice['index']] = [choice['text']]
                input_tokens = js['usage']['prompt_tokens']
                output_tokens = js['usage']['completion_tokens']
            elif js['object'] == 'chat.completion.chunk':
                if 'choices' in js:
                    for choice in js['choices']:
                        if 'delta' in choice and 'index' in choice:
                            delta = choice['delta']
                            idx = choice['index']
                            if 'content' in delta:
                                delta_content = delta['content']
                                if idx in delta_contents:
                                    delta_contents[idx].append(delta_content)
                                else:
                                    delta_contents[idx] = [delta_content]
                # usage in chunk: {"id":"","object":"chat.completion.chunk","created":1718269986,"model":"llama3",
                # "choices":[],"usage":{"prompt_tokens":32,"total_tokens":384,"completion_tokens":352}}
                if 'usage' in js and js['usage']:
                    input_tokens = js['usage']['prompt_tokens']
                    output_tokens = js['usage']['completion_tokens']
        if (input_tokens is None and output_tokens is None and self.tokenizer is not None):
            input_tokens = 0
            output_tokens = 0
            for idx, choice_contents in delta_contents.items():
                full_response_content = ''.join([m for m in choice_contents])
                input_tokens += len(self.tokenizer.encode(request['messages'][0]['content']))
                output_tokens += len(self.tokenizer.encode(full_response_content))
        elif input_tokens is None and output_tokens is None:  # no usage info get.
            input_tokens = 0
            output_tokens = 0
            logger.warning('No usage information found. Please specify `--tokenizer-path` to generate usage details.')

        return input_tokens, output_tokens

it would be great it you just submit a pull-request :)

[tghfly]

哈哈，我尝试提了一个pull-request #260 @yingdachen

[wangxingjun778]

哈哈，我尝试提了一个pull-request #260 @yingdachen

merged ~ you can pull the main branch and perform regression testing :)