Exception when shutting down an asp.net core website

[cadilhac]

I use this package with an injected store thanks to UseRavenDbWithInjectedDocumentStore in an asp.net core 6 project.
When shutting down, I get these traces:

info: Microsoft.Hosting.Lifetime[0]
Application is shutting down...
info: Quartz.Core.QuartzScheduler[0]
Scheduler Scheduler_$NON_CLUSTERED shutting down.
info: Quartz.Core.QuartzScheduler[0]
Scheduler Scheduler$_NON_CLUSTERED paused.

It hangs there, then this is displayed:

Unhandled exception. System.AggregateException: One or more hosted services failed to stop. (The operation was canceled.)
---> System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.CancellationToken.ThrowIfCancellationRequested()
at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in C:\Builds\RavenDB-Stable-5.3\53029\src\Raven.Client\Http\RequestExecutor.cs:line 823
at Raven.Client.Documents.Session.AsyncDocumentSession.LoadAsync[T](String id, CancellationToken token) in C:\Builds\RavenDB-Stable-5.3\53029\src\Raven.Client\Documents\Session\AsyncDocumentSession.Load.cs:line 29
at Quartz.Impl.RavenDB.RavenJobStore.SetSchedulerState(SchedulerState state, CancellationToken cancellationToken)
at Quartz.Impl.RavenDB.RavenJobStore.Shutdown(CancellationToken cancellationToken)
at Quartz.Core.QuartzScheduler.Shutdown(Boolean waitForJobsToComplete, CancellationToken cancellationToken)
at Quartz.QuartzHostedService.StopAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)

Any idea @ayende ? Could it be that the access to the RavenDb database has closed first and that the RavenDb Quartz store can not communicate anymore? If yes, how to ensure that the db closes after quartz has done its things?

[cadilhac]

Oh, it seems that this is simply because the cancellationToken requests a cancellation that RavenJobStore.SetSchedulerState throws an exception when LoadAsync is called with the token. Maybe a solution would be to avoid using this token in some terminal states like Paused and Shutdown?

[ayende]

The problem is that we need to cancel ongoing requests.
Assume that you want to stop, and then you use another cancellation token, but the requests hangs, what do you do then?

[cadilhac]

The problem is that we need to cancel ongoing requests.

In the case of my server shutdown, my intent is not to stop instantly. The intent is to let the ongoing jobs finish their tasks or interrupt them gracefully (when Quartz is configured that way). So Quartz is waiting for the jobs to finish but the RavenDb store on its side is more in a hurry to interrupt any databse operation.

Assume that you want to stop, and then you use another cancellation token, but the requests hangs, what do you do then?

Maybe use another token with a reasonable timeout?
Otherwise, isn't it a problem also to leave the quartz entities in the database in an inconsistent state? (state of the scheduler for instance).

[ayende]

Would this be something that you can provide a PR for?