Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fallback on local execution if slurm fails #323

Merged
merged 4 commits into from
Sep 11, 2024
Merged

fallback on local execution if slurm fails #323

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2f8d2a2
fallback on local execution if slurm fails
tmichela Aug 30, 2024
2f970ec
log error information
tmichela Sep 2, 2024
7382ab2
limit the number of concurrent process running locally
tmichela Sep 4, 2024
c08cd9b
patch kafak consumer in test;
tmichela Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion damnit/backend/listener.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import platform
from pathlib import Path
from socket import gethostname
from threading import Thread

from kafka import KafkaConsumer

Expand Down Expand Up @@ -110,7 +111,11 @@
log.info(f"Added p%d r%d ({run_data.value} data) to database", proposal, run)

req = ExtractionRequest(run, proposal, run_data)
self.submitter.submit(req)
try:
self.submitter.submit(req)
except Exception:
log.warning("Slurm job submission failed, starting process locally")
tmichela marked this conversation as resolved.
Show resolved Hide resolved
Thread(target=self.submitter.execute_direct, args=(req, )).start()

Check warning on line 118 in damnit/backend/listener.py

View check run for this annotation

Codecov / codecov/patch

damnit/backend/listener.py#L114-L118 

Added lines #L114 - L118 were not covered by tests


def listen():
Expand Down