bug: <Vulnerability Scanning gets stuck while scanning multiple domains>

[RBoorsma-Onv]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

While scanning multiple domains, the reNgine celery queue gets stuck with a pool full of vulnerability_scan. Because of that, there is no room in the queue for the nuclei_scan task and the nuclei_individual_severity_module.

This is because in tasks.py both the nuclei_scan and vulnerability_scan methodes have the following code:

while not job.ready(): # wait for all jobs to complete time.sleep(5)

The only reason this task is awaiting in the queue is because it wants to complete the scan succesfully after that.
This is an inefficient way to manage the celery queue and can get the scan engines stuck.

Expected Behavior

Both nuclei_scan and vulnerability_scan shouldn't wait for the tasks to complete. After they have send the tasks to the message broker they should exit the celery queue.

Then after all scans are done it can be marked as completed. This should be a seperate method or check that doesn't impact the celery queue.

Steps To Reproduce

1. Startup reNgine with a MAX_CONCURRENCY of 20.
2. Scan multiple domains at the same time (preferably more than 20) with a scan engine that at least uses vulnerability scanning with nuclei.
3. Wait for a while and look in the logs, you can notice that the scan will get stuck.
4. In the celery container, inspect the queue to see what is active and u will see multiple vulnerability_scan methodes. These methodes don't do any scanning so it is an infinite loop.

Environment

- reNgine: 2.2.0
- OS: Ubuntu 24.10
- Python: 3.12.7
- Docker Engine: 28.1.1
- Docker Compose: 2.38.1
- Browser: Google Chrome 137.0.7151.122

Anything else?

No response

[github-actions]

Hey @RBoorsma-Onv! 👋 Thanks for flagging this bug! 🐛🔍

You're our superhero bug hunter! 🦸‍♂️🦸‍♀️ Before we suit up to squash this bug, could you please:

📚 Double-check our documentation: https://rengine.wiki
🕵️ Make sure it's not a known issue
📝 Provide all the juicy details about this sneaky bug

Once again - thanks for your vigilance! 🛠️🚀

[dmchaledev]

Amazing context and info for reproduction, thank you RBoorsma-Onv, I'll test the proposed changes to tasks.py so that nuclei_scan functionality is building on top of vulnerability_scan instead of overlapping it, then I'll test and submit a PR.

[dmchaledev]

I'm working on a re-architecture for the celery workers/queue distribution where we are using while/sleep anti-pattern described above, we will instead try using chords and callbacks in celery and will update here when I've tested and it's ready for a PR.

[dmchaledev]

I also received a report that this in tandem with the network intensive scans like nuclei and fuzzing are flooding NICs on certain hardware. Thinking of separate queues instead of one big queue and setting separate limiting by network load intensity:

CELERY_ROUTES = {
    # Separate queues by network intensity
    'nuclei_scan': {'queue': 'high_network_queue'},
    'directory_fuzzing': {'queue': 'high_network_queue'},
    'port_scan': {'queue': 'medium_network_queue'},
    'screenshot': {'queue': 'low_network_queue'},
    'subdomain_discovery': {'queue': 'low_network_queue'},
    
    # Management queues
    'scan_state_manager': {'queue': 'management_queue'},
    'report_generation': {'queue': 'report_queue'},
    'notification': {'queue': 'notification_queue'},
}

# Scaling based on queue load
CELERY_WORKER_AUTOSCALER_CONFIG = {
    'high_network_queue': {'max': 5, 'min': 1},    # Limited for network-intensive tasks  
    'medium_network_queue': {'max': 10, 'min': 2},
    'low_network_queue': {'max': 20, 'min': 3},
    'management_queue': {'max': 50, 'min': 5},     # Fast management tasks
}