lasthearbeat null and tasks with old schedulers remain on db

[philMoe1]

Hi,

We noticed that last_heartbeat field on the scheduler_tasks table isn't being updated on tasks with dead schedulers is this expected? We are using v14.0.3 of db-scheduler with postgres db.

We've configured the scheduler via java not application properties. As we aren't using the spring boot starter due to being outside our requirements because it creates schedulers on every node due to the @ConditionalOnMissingBean annotation on the scheduler bean defined in DbSchedulerAutoConfiguration.

final Scheduler scheduler = Scheduler.create(dataSource, aTask, bTask) .startTasks(cTask, dTask, eTask) .schedulerName(schedulerName) .threads(10) .pollingInterval(Duration.ofSeconds(10)) .pollUsingLockAndFetch(1.0, 4.0) .heartbeatInterval(ofMinutes(1)) .missedHeartbeatsLimit(4) .enableImmediateExecution() .deleteUnresolvedAfter(ofMinutes(5)) .registerShutdownHook() .failureLogging(LogLevel.ERROR, true) .build();

We noticed that the last_heartbeat field isn't being updated for tasks with dead schedulers. Am I misunderstanding the how the scheduler should be updating the scheduled_tasks table every minute including tasks that are picked by a dead scheduler (node death or some other reason) that has previously picked a task in the scheduled_tasks table.

Currently, the tasks with a dead scheduler remains on the table indefinitely. Is db-scheduler library expected to clean up any of these tasks or is it a case of manual removal somehow i.e. making a cleanup task to remove said tasks with picked by dead schedulers.

[kagkarlsson]

We noticed that last_heartbeat field on the scheduler_tasks table isn't being updated on tasks with dead schedulers is this expected?

Yeah, that is exactly what is expected. Then they will eventually be considered dead and rescheduled (according to DeadExecutionHandler). They will be cleaned up after 8* I think.

If you have lots of dead executions, are you possibly not shutting down scheduler gracefully?

[philMoe1]

"Then they will eventually be considered dead and rescheduled (according to DeadExecutionHandler). They will be cleaned up after 8* I think."

Thanks for the quick response, Is there way to instead remove the dead task from the task set rather than revive? We have the following configured task and set the DeadExecutionHandler to cancel the dead execution instead. Additionally is possible to change the DeadExecutionHandler time to be instantly after it's considered a dead execution?

task = Tasks.recurring(new TaskWithDataDescriptor<>(getName(), Long.class), getSchedule()) .onFailure(new FailureHandler.MaxRetriesFailureHandler<>(0, (ExecutionComplete executionComplete, ExecutionOperations<Long> executionOperations) -> { getLogger().error("Failed {}. Instance: {}, cause: {}", getName(), executionComplete.getExecution().taskInstance, executionComplete.getCause()); executionOperations.remove(); })) .onDeadExecution(new DeadExecutionHandler.CancelDeadExecution<>()) .execute(wrapperExecute(doExecute(springComponents)));

"If you have lots of dead executions, are you possibly not shutting down scheduler gracefully?"

This is due to possible node deaths on our AWS cluster hence we need to be able to clear tasks locked by dead schedulers in less than an hour.