What's the difference between num_workers from WhisperModel and batch_size from BatchedInferencePipeline?

[ZhengYuan-Public]

I'm tyring to build a simple app for my company, and the GPU server has 4 GPUs, I wonder how to config the model so it can fully utilize all GPUs.

I'm confused about num_workers from the WhisperModel class notation and the batch_size from the example

from faster_whisper import WhisperModel, BatchedInferencePipeline

model = WhisperModel("turbo", device="cuda", compute_type="float16")
batched_model = BatchedInferencePipeline(model=model)
segments, info = batched_model.transcribe("audio.mp3", batch_size=16)

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

[MahmoudAshraf97]

batch size affects the number of segment in one generation pass for a single worker, if you have multiple workers, each will generate batch_size segments

[ZhengYuan-Public]

Thanks for your answer.
Does this mean I should set num_workers=num_GPUs then test what's the optimal batch_size that maximize GPU utilization? The GPU memory utilization is relatively small (only about 10%), is it possible for me to use more workers?