[BUG] job_pipeline task does not set job's final status on exception jobs get stuck in RUNNING forever

[Abhishek9639]

What happened

While reading through the job_pipeline task in intel_owl/tasks.py (lines 247-266), I noticed that when job.execute() throws an exception, the except block marks individual plugin reports as FAILED but it never actually updates the Job object itself.

Here's the relevant code:

@shared_task(base=FailureLoggedTask, name="job_pipeline", soft_time_limit=100)
def job_pipeline(job_id: int):
    from api_app.models import Job

    job = Job.objects.get(pk=job_id)
    try:
        job.execute()
    except Exception as e:
        logger.exception(e)
        for report in (
            list(job.analyzerreports.all())
            + list(job.connectorreports.all())
            + list(job.pivotreports.all())
            + list(job.visualizerreports.all())
        ):
            report.status = report.STATUSES.FAILED.value
            report.save()

The issue is inside Job.execute() (line 714 in models.py), the very first thing it does is set self.status = self.STATUSES.RUNNING and save it. So by the time an exception is raised (say during _get_pipeline() or _get_signatures() or if the broker is temporarily down), the job is already marked as RUNNING in the DB.

But the except block above doesn't set it back to FAILED. It also doesn't:

• Set job.finished_analysis_time
• Send a WebSocket notification via JobConsumer.serialize_and_send_job(job)
• Update the parent Investigation status

So the job just stays stuck in RUNNING forever. The job_set_final_status task (line 224) is supposed to handle this cleanup, but it's chained as the last step of the Celery pipeline if the pipeline never starts, that task never runs.

I noticed there's already a check_stuck_analysis periodic task (line 124) that catches these after ~25 minutes and marks them as failed, but that's more of a safety net. It also doesn't send any WebSocket notifications, so the user still sees a stuck spinner until they manually refresh.

Worth noting: the run_plugin task (line 268) does call JobConsumer.serialize_and_send_job(job) after exception handling, so there's already a pattern for this in the codebase job_pipeline just doesn't follow it.

Environment

1. OS: Ubuntu (Docker)
2. IntelOwl version: develop (v6.6.0)

What did you expect to happen

If job.execute() fails, the except block should also clean up the job state properly. Something like:

1. Set job.status to FAILED
2. Append the error to job.errors
3. Set job.finished_analysis_time = now()
4. Call JobConsumer.serialize_and_send_job(job) so the frontend knows right away
5. Update the Investigation status if the job belongs to one

Basically the same stuff job_set_final_status would have done if the pipeline had completed normally.

How to reproduce your issue

1. Cause an exception during job.execute() for example a DB error in _get_signatures(), a broken PythonConfig import, or even just the broker being momentarily unreachable when runner() is called
2. The job will be stuck as RUNNING in the database indefinitely
3. Frontend shows an infinite spinner with no error feedback
4. Eventually check_stuck_analysis cleans it up after ~25 min, but without notifying the frontend

You can also just read the code to confirm:

• intel_owl/tasks.py line 247-266 → the incomplete except block
• intel_owl/tasks.py line 224-232 → job_set_final_status that does the proper cleanup
• api_app/models.py line 714-715 → where execute() sets status to RUNNING before building the pipeline

Error messages and logs

No error on the frontend that's the whole problem. The user just sees a job stuck in "Running" forever. Server-side, the exception gets logged via logger.exception(e) but from the user's perspective nothing happens.

DB state of a stuck job:

• job.status = running (never reaches a terminal state)
• job.finished_analysis_time = NULL
• Because finished_analysis_time is NULL, remove_old_jobs won't clean these up either they just accumulate

[Abhishek9639]

Hi @mlodic,
I'd like to work on this. Can you please assign this to me?
Thanks

[mlodic]

to validate this is a real issue I would need to check this closely because it is most probably intended. Also, I think this bug was already raised and closed

[Abhishek9639]

Thanks for the clarification
I’ll investigate this further, verify if it’s intended behavior, and look for any previously closed issues related to this.
I’ll update here once I have more clarity.

[Abhishek9639]

I looked into this more carefully.

The closest issues I found are #3555 (worker killed via SIGKILL/OOM) and #3193 (S3 race condition), but this is a different case the worker is alive here and the except block does run, it just doesn't update the job itself. Only the individual plugin reports get marked FAILED, but job.status, finished_analysis_time, and the WebSocket notification are never touched.

I know check_stuck_analysis handles this eventually, but there's a ~25 min gap where the frontend just shows "Running" with no feedback. The run_plugin task already calls JobConsumer.serialize_and_send_job(job) after exceptions, so the pattern is there job_pipeline just skips it.

Anyway, if you decide it's intentional I'm fine with that, just wanted to flag the difference.
Thanks

[gks281263]

Hi @mlodic,

I’ve tested into the job lifecycle, and @Abhishek9639 is correct. This is a Real edge case that differs from #3555 (worker OOM) and #3193 (S3 race conditions). Those issues dealt with the Celery worker dying unexpectedly; this issue is about the worker actively catching an exception within job_pipeline but handling it incompletely.

While check_stuck_analysis exists as a safety net, relying on it in this specific scenario creates a few silent failure cascades:

Frontend UI Lockup: The UI spinner hangs indefinitely (up to 25 mins) because no terminal WebSocket notification is sent.
Investigation Deadlock: The parent Investigation status never resolves.
DB Retention Leak: Because finished_analysis_time remains NULL until the safety net fires, these jobs temporarily evade the remove_old_jobs cron.
The root cause is that job.execute() saves the job status as RUNNING before invoking _get_pipeline() or runner(). If pipeline construction fails (e.g., a broken PythonConfig import or a transient broker drop), the except block in job_pipeline (line 255 of tasks.py) marks the child reports as FAILED but orphans the parent Job.

Proposed Fix We just need to align the job_pipeline except block with the contracts already established in run_plugin() and set_final_status().

In api_app/tasks.py -> job_pipeline (around line 255)

except Exception as e:
    logger.exception(e)
    for report in (
        list(job.analyzerreports.all())
        + list(job.connectorreports.all())
        + list(job.pivotreports.all())
        + list(job.visualizerreports.all())
    ):
        report.status = report.STATUSES.FAILED.value
        report.save()
        
    # --- NEW CODE ---
    job.status = job.STATUSES.FAILED.value
    job.errors.append(str(e))
    job.finished_analysis_time = now()
    job.save(update_fields=["status", "errors", "finished_analysis_time"])
    
    if root_investigation := job.get_root().investigation:
        root_investigation.set_correct_status(save=True)
        
    from api_app.websocket import JobConsumer
    JobConsumer.serialize_and_send_job(job)

This ensures the job terminates gracefully, the UI updates immediately, and the DB cleans up as expected. If you agree with this approach, I can happily open a PR for this!

[Abhishek9639]

Hi @mlodic,
I had opened this issue and I’m already working on it. I noticed another contributor has also started working here just wanted to confirm if I should continue or if you’d prefer assigning it to someone else.
Thanks

[Abhishek9639]

Thanks for assigning this issue. I’ll start working on it and share a PR soon.