feat: handle singularity/apptainer sandbox images

[natthan-pigoux]

Running cwltool with dockerRequirements (with pull disabled) using a local Singularity/Apptainer sandbox image is not currently handle. cwltool is only searching for files (e.g. sif images) and not path as it should for sandbox images.

Here I provide:

• a function using singularity inspect to verify that a path is a singularity image
• searching for the sandbox image in the get_image method
• a test which build an alpine sandbox image and use it in dockerPull

Some questions I have:

• if the sandbox image is found I decided to skip the candidates search, I don't know if we wish that
• I'm not sure the singularity inspect is strong enough to ensure the presence of the image

It is the first time I contribute here so I might have miss some things :)

closes #2165

[codecov]

Codecov Report

❌ Patch coverage is 96.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85.17%. Comparing base (7be5a2f) to head (1ccfab8).

Files with missing lines	Patch %	Lines
cwltool/singularity.py	96.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2166      +/-   ##
==========================================
+ Coverage   85.13%   85.17%   +0.04%     
==========================================
  Files          46       46              
  Lines        8368     8392      +24     
  Branches     1956     1960       +4     
==========================================
+ Hits         7124     7148      +24     
  Misses        779      779              
  Partials      465      465              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mr-c]

Thank you @natthan-pigoux ; can you run make cleanup and push again?

[natthan-pigoux]

@mr-c , thank you to already have a look. I will be away next week, so I will probably not answer before 2nd November.

[natthan-pigoux]

@mr-c , sorry I should have put this PR as Draft before ... but now it should be ready for review. The tests should cover quiet well the changes.

[natthan-pigoux]

Hello @mr-c , I have change a bit the tests so that the coverage should be happy. Can you run them and tell me if its good for you?

[natthan-pigoux]

What about now ? ^^

[mr-c]

@natthan-pigoux Code coverage of the tests looks great, thank you! I'll investigate the mypy/mypyc issue

[mr-c]

So this checks with respect to the current working directory? What if the user wants to store their container sandbox image in another place? Where does apptainer/singularity put them? Should we re-use CWL_SINGULARITY_CACHE for this purpose?

[natthan-pigoux]

So this checks with respect to the current working directory? What if the user wants to store their container sandbox image in another place? Where does apptainer/singularity put them? Should we re-use CWL_SINGULARITY_CACHE for this purpose?

I started from the statement that the sandbox image is already deployed somewhere accessible and that the user only wants to use it.

Here I didn't implement any mechanism to "pull" a sandbox image. If that is of interest I can do it in another PR maybe? Is that what you were asking?

The image path can be specified within dockerPull and should not necessarily stored in the current working directory. But you are right that in tests I refer only to relative path, I will add another one with an absolute path.

[mr-c]

The image path can be specified within dockerPull and should not necessarily stored in the current working directory. But you are right that in tests I refer only to relative path, I will add another one with an absolute path.

No absolute paths and no relative paths that go "up"; we need to check for both and not allow them. They are bad for portability of the CommandLineTool descriptions; and possible security issues.

We need to decide on a default location for sandbox/unpacked singularity/apptainer images, and way for the user to specify an alternative root folder for those (command line option to cwltool and/or a environment variable).

[natthan-pigoux]

We need to decide on a default location for sandbox/unpacked singularity/apptainer images, and way for the user to specify an alternative root folder for those (command line option to cwltool and/or a environment variable).

Indeed that seems necessary.
Then, should the default be the current working directory?

What would be nice is that this variable corresponds to the base path from where to get the image e.g., for example CWL_SINGULARITY_IMAGE_BASE_PATH , then dockerPull would contain path/to/image . Then we build the image path from this base path: $CWL_SINGULARITY_IMAGE_BASE_PATH/path/to/image

If we also add a cwltool cli option name could follow the same semantic --image_base_path

I don't have a strong opinion on the naming of this env variable and cli option. Maybe CWL_SINGULARITY_IMAGE_BASE_PATH is too long.
What do you think?

[mr-c]

We need to decide on a default location for sandbox/unpacked singularity/apptainer images, and way for the user to specify an alternative root folder for those (command line option to cwltool and/or a environment variable).

Indeed that seems necessary. Then, should the default be the current working directory?

Sure, both the current working directory plus whatever other environment variable or command line option we come up with should be checked.

What would be nice is that this variable corresponds to the base path from where to get the image e.g., for example CWL_SINGULARITY_IMAGE_BASE_PATH , then dockerPull would contain path/to/image . Then we build the image path from this base path: $CWL_SINGULARITY_IMAGE_BASE_PATH/path/to/image

Yes, but I'd like to re-use CWL_SINGULARITY_CACHE for this purpose.

If we also add a cwltool cli option name could follow the same semantic --image_base_path

How about --singularity-sandbox-path ?