Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

84-add singularity guide #112

Merged
merged 34 commits into from
Nov 6, 2020
Merged

84-add singularity guide #112

merged 34 commits into from
Nov 6, 2020

Conversation

fdiblen
Copy link
Collaborator

@fdiblen fdiblen commented Jul 6, 2020

Refs: #84 #108 #17

@fdiblen fdiblen changed the title add singularity guide 84-add singularity guide Jul 6, 2020
@fdiblen fdiblen marked this pull request as ready for review July 7, 2020 17:53
@fdiblen fdiblen requested a review from jspaaks July 7, 2020 17:53
Copy link
Member

@jspaaks jspaaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super job man

.github/workflows/super-linter.yml Outdated Show resolved Hide resolved
Copy link
Collaborator

@sverhoeven sverhoeven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying it out with https://github.com/ci-for-research/example-python-1 repo with black check disabled in workflow. The jobs fails with incomplete job log and when I CTRL-C the singularity container, it errors with

---> Removing the runner.
Unhandled exception. System.IO.IOException: No space left on device
   at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
   at System.IO.FileStream.FlushWriteBuffer()
   at System.IO.FileStream.FlushInternalBuffer()
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Flush()
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.Write(String message)
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Trace(TraceEventType eventType, String message)
   at GitHub.Runner.Common.HostContext..ctor(String hostType, String logFile)
   at GitHub.Runner.Listener.Program.Main(String[] args)
./config.sh: line 79: 20881 Aborted                 (core dumped) ./bin/Runner.Listener "$@"

And keeps the runner (run.sh) running in the background.

Is this because my tmpfs has 1.6Gb available?

ubuntu-singularity/github-actions-runner-singularity.def Outdated Show resolved Hide resolved
Alternatively, you can start it as an instance (service).

```shell
singularity instance start github-actions-runner-singularity.sif github-actions-runner --writable-tmpfs --bind ./env:/opt/actions-runner/.env
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using temporary mode I get the log of the runner in my terminal. How do I get the log from running instance?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the info.

ubuntu-singularity/README.md Show resolved Hide resolved
ubuntu-singularity/README.md Outdated Show resolved Hide resolved
ubuntu-singularity/README.md Outdated Show resolved Hide resolved
ubuntu-singularity/entrypoint.sh Outdated Show resolved Hide resolved
@@ -14,3 +14,4 @@ jobs:
uses: docker://github/super-linter:v3
env:
VALIDATE_ALL_CODEBASE: true
VALIDATE_DOCKER: false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why disable? ubuntu-docker/Dockerfile should still be linted.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i fiddled with this extensively see #127

Summary:

Alternative approaches:

  • avoid using sudo as a string in the Dockerfile, maybe using an env var or something.
  • include all VALIDATE_[LANGUAGE] keys in the workflow file and setting them to true (excpet Docker)

Neither approach seems good, hence I decided to wait.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed sudo command from the Docker file


## Using on a HPC Cluster

In oder to use the Singularity image on a HPC cluster, we first need to create a jobscript. The job script will be handled by the scheduler and eventually it will set up the runner when it is executed.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you don't have root-permission on the HPC cluster you will need to build the github-actions-runner-singularity.sif file on your own machine and copy it to the cluster. Can a build+copy hint be added?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added scp command

@fdiblen fdiblen merged commit 88b3787 into master Nov 6, 2020
@fdiblen fdiblen deleted the 84-singularity branch November 6, 2020 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants