HTCondor

[StephMcCallum]

Hello,
Is it possible to add integration for the HTCondor scheduler? This scheduler is used with OSPool, which is distributed high throughput computing. They currently develop a tool called DAGMan to manage workflows with pre/post conditions. I think this community would benefit from signac+row functionality.

[joaander]

I have never used HTCondor, nor do I have access to a HTCondor system for testing. Judging by the quick start guide, it appears to have submit and status commands similar to SLURM. Therefore, it should be possible to add an HTCondor submission engine to row following the implementation of scheduler/slurm.rs. I encourage you to submit a pull request that adds this functionality. I have no plans to implement this functionality myself.

Would you use this in an environment with a shared filesystem? Or in one where HTCondor needs to transfer files?

Row and signac are designed with the assumption of a shared filesystem. For example, row generates submission scripts that call the row executable during the job execution (via absolute path). It produces staged completion files (in the .row directory) that row on the submission host will later process to track action/directory status efficiently. signac assumes a flat directory structure on the filesystem and there are no restrictions in place that prevent actions on one directory accessing data from another. At the same time, row produces unique completion status files for each job and the overall workspace caches are not used during the job. I suppose you could configure HTCondor to transfer the entire workspace over for each job. Not only will that be expensive for large workspaces, but it may not work for multiple jobs (depending on how HTCondor handles merging the changes when the data is transferred back). If you are using Python environments, those can also be extremely expensive to transfer due to the large file count. We've had some experience accelerating Python load times on HPC systems using conda-pack to produce a single tar file that can be efficiently copied to NVME and unpacked. You could use a similar procedure with HTCondor on a distributed filesystem.

If you restricted your actions to both read and write only the directory or directories they run on, then you could in theory develop a more efficient system that transfers only those directories to the execution host. Such a complex system is beyond the scope of row. At most, row could generate the script that passes the appropriate options to a tool that handles the transfers and merges.

None of these extra considerations are needed on a shared filesystem. All you need to do is implement scheduler/ht_condor.rs to generate properly formatted submission scripts, submit them with condor_submit (you'll need to add the appropriate option to get the ids of the submitted job), and check which jobs are still queued/running with condor_q.

[joaander]

Do you have any questions about implementing HTCondor support in row?

[StephMcCallum]

Thank you for your thoughtful response, Joshua! This question came up from a workshop with HTCondor. I'm not familiar with rust and HTCondor does not have a shared file system, so I expect this may be too expensive to integrate. I will share this discussion with the HTCondor developers and hope they will pursue this PR if it makes sense with their system!

[joaander]

Thanks!

I'm happy to answer any questions anyone has on this topic. I would suggest that anyone interested investigate other workflow engines that are purpose built for distributed systems first. row and signac in many ways are built around the idea of a shared filesystem.