Slightly reduce sleep time when batching queries

This can give a small speedup for free, since usually batched queries all come in within <0.5s Signed-off-by: Haifeng Qian <[email protected]>
NVIDIA · Dec 3, 2024 · 3409659 · 3409659
1 parent 2f2b0c8
commit 3409659
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/nemo/collections/nlp/modules/common/text_generation_server.py b/nemo/collections/nlp/modules/common/text_generation_server.py
@@ -193,7 +193,7 @@ def chat_completion(self, data):
 
                 MegatronGenerate.inputs.append(conversation)
                 MegatronGenerate.tasks.put(None)  # The tasks queue is only used as a "counter"
-            time.sleep(1)
+            time.sleep(0.5)  # process one batch every 0.5s
         else:
             queryid = 0
         end_strings = ['<|endoftext|>', special_tokens['turn_start'], special_tokens['label_start']]