🚀 HPC Rocket

HPC Rocket is a tool to send slurm commands to a remote machine and monitor the job progress. It was primarily written to launch slurm jobs from a CI pipeline.

Installation

You can get the latest version of HPC Rocket on PyPI:

python3 -m pip install hpc-rocket

Authentication

HPC Rocket does support authentication via password and private key. Both can be set via environment variables.

Slurm configuration

Currently all sbatch configuration must happen in the job file. HPC Rocket does not offer any other way of configuring your batch jobs.

Configuration file

HPC Rocket uses a configuration file in YAML format containing credentials to connect to the remote machine. Additionally it allows copying files to the remote machine, copying results back to the local machine (collecting) and eventually cleaning up copied or produced files. Note that all paths in the configuration file must be relative paths. On the local machine paths are evaluated from the current working directory, on the remote machine from the user's home directory, unless absolute paths are specified. If you want to overwrite existing files on the remote machine, make sure to specify the overwrite instruction for each file you would like to overwrite. HPC Rocket will evaluate environment variables on the LOCAL machine in the form of ${VAR} and $VAR when parsing the config file.

host: cluster.example.com
user: myuser
private_keyfile: ~/.ssh/id_rsa

proxyjumps:
  - host: myproxy.example.com
    user: myproxy-user
    private_keyfile: ~/.ssh/proxy_keyfile

copy:
  - from: jobs/slurm.job
    to: slurm.job
    overwrite: true

  - from: bin/myexecutable
    to: myexecutable

collect:
  - from: remote_slurmresult.out
    to: local_slurmresult.out
    overwrite: true

clean:
  - slurm.job
  - myexecutable

sbatch: slurm.job

Usage

Launching a job on the remote machine

Use the launch command to launch a job on the remote machine. You must provide a configuration file. The optional --watch flag makes hpc-rocket wait until your job is finished (defaults to false). The collection and cleaning steps in the configuration file are only executed if --watch is set.

hpc-rocket launch --watch config.yml

Checking a job's status

If a job was launched without --watch you can still check its status using the status command. You will need to provide a configuration file with connection data and a job ID to check.

hpc-rocket status config.yml 12345

Monitoring a job until it finishes

Similar to the status command, hpc-rocket also provides the watch command to monitor a job's status continuously by entering a config file and a job id.

hpc-rocket watch config.yml 12345

Canceling a running job

Jobs may also be canceled using the cancel command. Like the previous commands it accepts a config file and the id of a running job.

hpc-rocket cancel config.yml 12345