Making a batch job save output as it runs

[amazingvince]

I am generating synthetic data using sglang. I would like to be saving the outputs of each query as it runs batch preferably in such a way that if something breaks I could resume where it where it left off. I was wondering as anyone implemented something like this or have a strategy to do this? Like async operations or using joblib?

[merrymercy]

You do not need to send all of them in a single batch.
You can launch your own threads and manage them at per-request level, so you can store any partial results.
The performance will be the same because the low-level engine does auto-batching for you.