[Bug]A Chat only can answer one question when using chatqna-conversation-ui:1.0 #795
Description
Priority
P3-Medium
OS type
Ubuntu
Hardware type
GPU-Nvidia
Installation method
- Pull docker images from hub.docker.com
- Build docker images from source
- Other
Deploy method
- Kubernetes Helm Charts
- Kubernetes GMC
- Other
Running nodes
Single Node
What's the version?
opea 1.0
Description
A Chat only can answer one question. If I answer second question, it still response the first question.
Reproduce steps
values-deepseek.yaml
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
# Default values for chatqna.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: opea/chatqna
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: "1.0"
port: 8888
service:
type: ClusterIP
port: 8888
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
nodeSelector: {}
tolerations: []
affinity: {}
# This is just to avoid Helm errors when HPA is NOT used
# (use hpa-values.yaml files to actually enable HPA).
horizontalPodAutoscaler:
enabled: false
# Override values in specific subcharts
tgi:
LLM_MODEL_ID: deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
accelDevice: "nvidia"
image:
repository: ghcr.io/huggingface/text-generation-inference
tag: "2.2.0"
resources:
limits:
nvidia.com/gpu: 4
livenessProbe:
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 1
readinessProbe:
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 1
startupProbe:
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 12000
# disable guardrails-usvc by default
# See guardrails-values.yaml for guardrail related options
guardrails-usvc:
enabled: false
global:
http_proxy: ""
https_proxy: "http://proxy.ims.intel.com:911"
no_proxy: "chatqna-chatqna-ui,chatqna-ui,chatqna-data-prep,data-prep,chatqna-embedding-usvc,embedding-usvc,embedding-svc,chatqna-llm-uservice,llm-uservice,llm-svc,chatqna-redis-vector-db,redis-vector-db,chatqna-reranking-usvc,reranking-usvc,reranking-svc,chatqna-retriever-usvc,retriever-usvc,retriever-svc,chatqna-tei,tei,chatqna-teirerank,teirerank,chatqna-tgi,tgi,chatqna-nginx,chatqna,chatqna-chatqna-ui,chatqna-ui,192.168.0.0/24,127.0.0.1,localhost,.intel.com,.default.svc.cluster.local,10.96.0.0/12,10.244.0.0/16"
HUGGINGFACEHUB_API_TOKEN: "insert you token"
huggingfacehub_api_token: "insert you token"
# HF_ENDPOINT: "https://hf-mirror.com"
# set modelUseHostPath or modelUsePVC to use model cache.
# modelUseHostPath: ""
modelUseHostPath: /mnt/s3-mount
# Prometheus Helm installation info for subchart serviceMonitors
prometheusRelease: prometheus-stack
Run chatqna:
helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=/mnt/s3-mount/ -f chatqna/values-deepseek.yaml
Raw log
2025-02-11T05:49:02.754064Z INFO text_generation_launcher: Args {
model_id: "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
revision: None,
validation_workers: 2,
sharded: None,
num_shard: None,
quantize: None,
speculate: None,
dtype: None,
trust_remote_code: false,
max_concurrent_requests: 128,
max_best_of: 2,
max_stop_sequences: 4,
max_top_n_tokens: 5,
max_input_tokens: None,
max_input_length: None,
max_total_tokens: None,
waiting_served_ratio: 0.3,
max_batch_prefill_tokens: None,
max_batch_total_tokens: None,
max_waiting_tokens: 20,
max_batch_size: None,
cuda_graphs: Some(
[
0,
],
),
hostname: "chatqna-tgi-76c78f4c54-r4scd",
port: 2080,
shard_uds_path: "/tmp/text-generation-server",
master_addr: "localhost",
master_port: 29500,
huggingface_hub_cache: Some(
"/data",
),
weights_cache_override: None,
disable_custom_kernels: false,
cuda_memory_fraction: 1.0,
rope_scaling: None,
rope_factor: None,
json_output: false,
otlp_endpoint: None,
otlp_service_name: "text-generation-inference.router",
cors_allow_origin: [],
watermark_gamma: None,
watermark_delta: None,
ngrok: false,
ngrok_authtoken: None,
ngrok_edge: None,
tokenizer_config_path: None,
disable_grammar_support: false,
env: false,
max_client_batch_size: 4,
lora_adapters: None,
disable_usage_stats: false,
disable_crash_reports: false,
}
2025-02-11T05:49:02.754174Z INFO hf_hub: Token file not found "/tmp/.cache/huggingface/token"
2025-02-11T05:49:07.509615Z INFO text_generation_launcher: Model supports up to 131072 but tgi will now set its default to 4096 instead. This is to save VRAM by refusing large prompts in order to allow more users on the same hardware. You can increase that size using `--max-batch-prefill-tokens=131122 --max-total-tokens=131072 --max-input-tokens=131071`.
2025-02-11T05:49:07.509648Z INFO text_generation_launcher: Default `max_input_tokens` to 4095
2025-02-11T05:49:07.509655Z INFO text_generation_launcher: Default `max_total_tokens` to 4096
2025-02-11T05:49:07.509659Z INFO text_generation_launcher: Default `max_batch_prefill_tokens` to 4145
2025-02-11T05:49:07.509690Z INFO text_generation_launcher: Sharding model on 4 processes
2025-02-11T05:49:07.509994Z INFO download: text_generation_launcher: Starting check and download process for deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
2025-02-11T05:49:13.286145Z INFO text_generation_launcher: Files are already present on the host. Skipping download.
2025-02-11T05:49:14.016397Z INFO download: text_generation_launcher: Successfully downloaded weights for deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
2025-02-11T05:49:14.016906Z INFO shard-manager: text_generation_launcher: Starting shard rank=0
2025-02-11T05:49:14.016983Z INFO shard-manager: text_generation_launcher: Starting shard rank=1
2025-02-11T05:49:14.017140Z INFO shard-manager: text_generation_launcher: Starting shard rank=2
2025-02-11T05:49:14.017250Z INFO shard-manager: text_generation_launcher: Starting shard rank=3
2025-02-11T05:49:24.026857Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:49:24.027027Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:24.027355Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:24.027630Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:34.033869Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:34.033977Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:49:34.034037Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:34.034155Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:44.040766Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:44.040850Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:49:44.040953Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:44.040966Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:54.047538Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:49:54.047889Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:49:54.047897Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:49:54.047919Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:04.054401Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:04.054548Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:04.054909Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:04.054958Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:14.061171Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:14.061177Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:14.061498Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:14.061723Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:24.068200Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:24.068205Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:24.068332Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:24.068383Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:34.075132Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:34.075158Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:34.075395Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:34.075418Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:44.082019Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:44.082018Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:50:44.082117Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:44.082167Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:54.088794Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:50:54.088847Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:50:54.089019Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:50:54.089022Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:04.095735Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:04.095776Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:04.096233Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:04.098523Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:14.102841Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:14.103346Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:14.103982Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:14.109104Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:24.109849Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:24.109873Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:24.110851Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:24.115700Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:34.116727Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:34.116823Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:34.117833Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:34.122757Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:44.123718Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:44.123914Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:44.124249Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:44.129360Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:51:54.130816Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:51:54.130863Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:51:54.130905Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:51:54.141376Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:04.137669Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:04.137673Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:04.137673Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:04.148239Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:14.144571Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:14.144582Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:14.144615Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:14.155153Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:24.151693Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:24.151697Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:24.151768Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:24.162123Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:34.158579Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:34.158597Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:34.158594Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:34.169015Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:44.165590Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:44.165597Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:44.165588Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:44.176320Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:52:54.172531Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:52:54.172595Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:52:54.172673Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:52:54.183247Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:04.179369Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:04.179443Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:04.179519Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:04.190149Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:14.186236Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:14.186278Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:14.186365Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:14.197106Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:24.193077Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:24.193127Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:24.193156Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:24.204008Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:34.199926Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:34.199971Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:34.200042Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:34.210785Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:44.208146Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:44.208955Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:44.208959Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:44.217787Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:53:54.215026Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:53:54.215744Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:53:54.215801Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:53:54.224443Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:04.222008Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:04.222203Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:04.222564Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:04.231264Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:14.228902Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:14.228891Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:14.229379Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:14.237779Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:24.235708Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:24.235840Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:24.236172Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:24.244499Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:34.242573Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:34.242677Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:34.242920Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:34.251412Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:44.249302Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:44.249442Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:44.249742Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:44.258099Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:54:54.256299Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:54:54.256324Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:54:54.256620Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:54:54.264803Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:04.263293Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:04.263385Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:04.263601Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:04.271516Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:14.270308Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:14.270316Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:14.270876Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:14.278237Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:24.277208Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:24.277305Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:24.277760Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:24.285046Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:34.284250Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:34.284269Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:34.284716Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:34.293369Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:44.291195Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:44.291428Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:44.291778Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:44.300106Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:55:54.298032Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:55:54.298093Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:55:54.298580Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:55:54.307091Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:04.305016Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:04.305224Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:04.305505Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:04.313885Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:14.312071Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:14.312118Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:14.312307Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:14.320593Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:24.319164Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:24.319192Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:24.319303Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:24.327472Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:34.326131Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:34.326166Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:34.326218Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:34.334405Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:44.333060Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:44.333060Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:44.333113Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:44.341175Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:56:54.339970Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:56:54.339970Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:56:54.340054Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:56:54.348120Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:04.346984Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:04.347012Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:04.347022Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:04.355152Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:14.353976Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:14.354024Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:14.354047Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:14.361882Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:24.360885Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:24.360888Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:24.360948Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:24.368766Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:34.367645Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:34.367676Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:34.367651Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:34.375281Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:44.374476Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:44.374575Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:44.374682Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:44.382112Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:57:54.381574Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:57:54.381651Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:57:54.382005Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:57:54.389292Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:04.388513Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:04.388531Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:04.388852Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:04.396262Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:14.395839Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:14.395849Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:14.395849Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:14.405708Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:24.402939Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:24.402936Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:24.402993Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:24.412681Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:34.409699Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:34.409953Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:34.410033Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:34.419615Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:44.416694Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:44.416867Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:44.416957Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:44.426414Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:58:54.423747Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:58:54.423892Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:58:54.424043Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:58:54.433169Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:04.430729Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:04.430914Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:04.431060Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:04.440080Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:14.437713Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:14.438082Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:14.438815Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:14.446831Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:24.444769Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:24.444794Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:24.445716Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:24.453586Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:34.451772Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:34.451788Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:34.452669Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:34.460941Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:44.458765Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:44.459001Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:44.459531Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:44.467846Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T05:59:54.465755Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T05:59:54.466185Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T05:59:54.466608Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T05:59:54.474791Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:04.472814Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:04.473111Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:04.473398Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:04.481512Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:14.479759Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:14.479792Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:14.480177Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:14.488542Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:24.486670Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:24.486754Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:24.486976Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:24.495513Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:34.493631Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:34.493644Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:34.493930Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:34.502467Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:44.500444Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:44.500477Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:44.500814Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:44.509281Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:00:54.507466Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:00:54.507614Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:00:54.507866Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:00:54.516409Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:04.514251Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:04.514577Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:04.514601Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:04.523495Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:14.521457Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:14.521624Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:14.521628Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:14.530334Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:24.528305Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:24.528480Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:24.528582Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:24.537176Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:34.535630Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:34.535625Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:34.535651Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:34.544026Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:44.542635Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:44.542668Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:44.542637Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:44.550866Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:01:54.549717Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:01:54.549767Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:01:54.549817Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:01:54.557846Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:04.556633Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:04.556679Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:04.556683Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:04.564676Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:14.563653Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:14.563700Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:14.563737Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:14.571837Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:24.570651Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:24.570673Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:24.570735Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:24.579002Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:34.577383Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:34.577570Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:34.577763Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:34.585797Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:44.584306Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:44.584314Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:44.584970Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:44.592416Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:02:54.591370Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:02:54.591447Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:02:54.592128Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:02:54.601378Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:03:04.598287Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:03:04.598395Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:03:04.598909Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:03:04.608307Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:03:14.605140Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:03:14.605228Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:03:14.605638Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:03:14.615124Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2025-02-11T06:03:23.863102Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0
2025-02-11T06:03:23.922983Z INFO shard-manager: text_generation_launcher: Shard ready in 849.903057579s rank=0
2025-02-11T06:03:24.612319Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=3
2025-02-11T06:03:24.612449Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=2
2025-02-11T06:03:24.612487Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2025-02-11T06:03:25.859857Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-2
2025-02-11T06:03:25.860944Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-3
2025-02-11T06:03:25.865287Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-1
2025-02-11T06:03:25.913263Z INFO shard-manager: text_generation_launcher: Shard ready in 851.892492758s rank=3
2025-02-11T06:03:25.913384Z INFO shard-manager: text_generation_launcher: Shard ready in 851.893030923s rank=2
2025-02-11T06:03:25.913417Z INFO shard-manager: text_generation_launcher: Shard ready in 851.893317076s rank=1
2025-02-11T06:03:25.980482Z INFO text_generation_launcher: Starting Webserver
2025-02-11T06:03:26.687648Z INFO text_generation_router: router/src/main.rs:228: Using the Hugging Face API
2025-02-11T06:03:26.687697Z INFO hf_hub: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hf-hub-0.3.2/src/lib.rs:55: Token file not found "/tmp/.cache/huggingface/token"
2025-02-11T06:03:36.170416Z INFO text_generation_router: router/src/main.rs:577: Serving revision 3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 of model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
2025-02-11T06:03:36.484596Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜end▁of▁sentence｜>' was expected to have ID '151643' but was given ID 'None'
2025-02-11T06:03:36.484611Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜User｜>' was expected to have ID '151644' but was given ID 'None'
2025-02-11T06:03:36.484614Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜Assistant｜>' was expected to have ID '151645' but was given ID 'None'
2025-02-11T06:03:36.484617Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<｜begin▁of▁sentence｜>' was expected to have ID '151646' but was given ID 'None'
2025-02-11T06:03:36.484618Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|EOT|>' was expected to have ID '151647' but was given ID 'None'
2025-02-11T06:03:36.484620Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<think>' was expected to have ID '151648' but was given ID 'None'
2025-02-11T06:03:36.484635Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '</think>' was expected to have ID '151649' but was given ID 'None'
2025-02-11T06:03:36.484637Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|quad_start|>' was expected to have ID '151650' but was given ID 'None'
2025-02-11T06:03:36.484638Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|quad_end|>' was expected to have ID '151651' but was given ID 'None'
2025-02-11T06:03:36.484640Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|vision_start|>' was expected to have ID '151652' but was given ID 'None'
2025-02-11T06:03:36.484641Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|vision_end|>' was expected to have ID '151653' but was given ID 'None'
2025-02-11T06:03:36.484643Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|vision_pad|>' was expected to have ID '151654' but was given ID 'None'
2025-02-11T06:03:36.484644Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|image_pad|>' was expected to have ID '151655' but was given ID 'None'
2025-02-11T06:03:36.484646Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|video_pad|>' was expected to have ID '151656' but was given ID 'None'
2025-02-11T06:03:36.484647Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<tool_call>' was expected to have ID '151657' but was given ID 'None'
2025-02-11T06:03:36.484649Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '</tool_call>' was expected to have ID '151658' but was given ID 'None'
2025-02-11T06:03:36.484650Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_prefix|>' was expected to have ID '151659' but was given ID 'None'
2025-02-11T06:03:36.484652Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_middle|>' was expected to have ID '151660' but was given ID 'None'
2025-02-11T06:03:36.484654Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_suffix|>' was expected to have ID '151661' but was given ID 'None'
2025-02-11T06:03:36.484658Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|fim_pad|>' was expected to have ID '151662' but was given ID 'None'
2025-02-11T06:03:36.484660Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|repo_name|>' was expected to have ID '151663' but was given ID 'None'
2025-02-11T06:03:36.484662Z WARN tokenizers::tokenizer::serialization: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.19.1/src/tokenizer/serialization.rs:159: Warning: Token '<|file_sep|>' was expected to have ID '151664' but was given ID 'None'
2025-02-11T06:03:36.486228Z INFO text_generation_router: router/src/main.rs:342: Overriding LlamaTokenizer with TemplateProcessing to follow python override defined in https://github.com/huggingface/transformers/blob/4aa17d00690b7f82c95bb2949ea57e22c35b4336/src/transformers/models/llama/tokenization_llama_fast.py#L203-L205
2025-02-11T06:03:36.486235Z INFO text_generation_router: router/src/main.rs:357: Using config Some(Qwen2)
2025-02-11T06:03:36.486239Z WARN text_generation_router: router/src/main.rs:384: Invalid hostname, defaulting to 0.0.0.0
2025-02-11T06:03:36.547050Z INFO text_generation_router::server: router/src/server.rs:1572: Warming up model
2025-02-11T06:03:40.722557Z INFO text_generation_launcher: Cuda Graphs are enabled for sizes [0]
2025-02-11T06:03:40.823710Z INFO text_generation_router::server: router/src/server.rs:1599: Using scheduler V3
2025-02-11T06:03:40.823736Z INFO text_generation_router::server: router/src/server.rs:1651: Setting max batch total tokens to 373920
2025-02-11T06:03:40.920081Z INFO text_generation_router::server: router/src/server.rs:1889: Connected
2025-02-11T06:04:26.630119Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.285263251s" validation_time="1.036747ms" queue_time="119.465µs" inference_time="1.284107407s" time_per_token="33.7923ms" seed="Some(17207679548585787341)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:05:33.038265Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.404232248s" validation_time="560.26µs" queue_time="52.421µs" inference_time="1.403619944s" time_per_token="31.900453ms" seed="Some(16275190732384590354)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:05:55.181522Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.216876593s" validation_time="948.563µs" queue_time="51.145µs" inference_time="1.215877199s" time_per_token="31.996768ms" seed="Some(14699494694487423509)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:00.596233Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.391676975s" validation_time="349.937µs" queue_time="58.445µs" inference_time="1.391268961s" time_per_token="31.619749ms" seed="Some(15828993800647044877)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:14.351810Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.210583081s" validation_time="469.041µs" queue_time="113.129µs" inference_time="1.210001384s" time_per_token="31.842141ms" seed="Some(3690229400508701128)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:32.156409Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.210452984s" validation_time="376.402µs" queue_time="42.773µs" inference_time="1.210034249s" time_per_token="31.843006ms" seed="Some(6777010132799794204)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:06:43.358804Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.394022016s" validation_time="361.85µs" queue_time="37.92µs" inference_time="1.39362255s" time_per_token="31.673239ms" seed="Some(725893851467889474)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:08:20.017441Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="1.211224469s" validation_time="379.444µs" queue_time="44.521µs" inference_time="1.210800824s" time_per_token="31.863179ms" seed="Some(5207544670684677685)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:08:55.509801Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="21.24950047s" validation_time="764.322µs" queue_time="89.915µs" inference_time="21.248646533s" time_per_token="31.526181ms" seed="Some(14816016702781754406)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:09:48.768802Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="27.378218083s" validation_time="452.25µs" queue_time="84.92µs" inference_time="27.377681266s" time_per_token="31.432469ms" seed="Some(896637297712218630)"}: text_generation_router::server: router/src/server.rs:511: Success
2025-02-11T06:10:27.694657Z INFO compat_generate{default_return_full_text=true compute_type=Extension(ComputeType("4-nvidia-l40s"))}:generate_stream{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: Some(1.0), frequency_penalty: None, top_k: Some(10), top_p: Some(0.95), typical_p: None, do_sample: false, max_new_tokens: Some(1024), return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None } total_time="28.452501692s" validation_time="3.629787ms" queue_time="48.31µs" inference_time="28.448824118s" time_per_token="32.699797ms" seed="Some(16454489381862762319)"}: text_generation_router::server: router/src/server.rs:511: SuccessAttachments
No response