Is there a way to have the LLM timeout or to cancel a request before the LLM starts to respond?

[nicholusi2021]

Most of my requests take less than a minute for a response to come back. However, some requests take over 10 minutes before I stop and restart the service. I played around with the cancelation token, but that seems to only cancels the request once the LLM starts to respond so it didn't speed things up.

Is there a way to have a request timeout after a certain amount of time, or a better way to cancel a request while it's still processing? I would rather respond back with a generic response then have the user wait over 10 minutes.

Thanks!

[martindevans]

Have you debugged it at all to see where it's getting stuck? 10 minutes is extreme, even for a large prompt!

[nicholusi2021]

Have you debugged it at all to see where it's getting stuck? 10 minutes is extreme, even for a large prompt!

Yes, it's right at the await foreach:

            var response = session.ChatAsync(history, GetInferenceParams(), cancellationToken.Token);

            await foreach (var text in response)
            {
                cancellationToken.Token.ThrowIfCancellationRequested();

                result += text;
            }

Looking at the Task Manager I can see it's very busy, but takes too long. Is there a way to set a timeout or a different way to cancel the request?

[n-ate]

You need to use your cancellationToken variable which is really a token source to cancel the request, but the LlamaSharp library will need to get at least one token first which is probably the slowest when a model runs slow. It doesn't look like the cancellation gets very deep into the library. The pipeline sample call won't cancel with the cancellation token so still worth calling it, but there is some room for improvement in the library around this. I think I'll make some changes to the cancellation and see how it goes.

[martindevans]

I haven't dug into it much, but I think llama.cpp actually supports cancelling mid-inference with a callback you can attach to the context. Wiring cancellation in that deep might be (very) tricky though!