Docker experience

[DANAJK]

I'm relaying my experience with Docker SIRF to either get feedback or guide improvements in our documentation.

I've tried with some success to get the Docker version of SIRF going on my MacBook Pro. I decided to use a Virtual Machine to avoid risking my Mac set up. Arrangement:
MacBook Pro running macOS Catalina (10.15.7)
VMware Fusion Version 11.5.6

From VMware Fusion, created a new Ubuntu-64bit machine.
From Ubuntu web site, downloaded ubuntu-20.04.1-desktop-amd64.iso
I forget exactly, but I think I had to make this iso a CD in the VM to get it to install.

Within the VM, typed 'docker' and followed the instructions for how to install Docker.
Followed the link in the SIRF SuperBuild/docker README.md file to the Git release site with docker-compose, I clicked on 1.28.0-rc1 and from the Assets section, downloaded docker-compose-Linux-x86_64. (Note there is duplicate documentation on the SIRF Wiki that should probably just point to the README)
I did some faffing with paths etc on the VM but eventually as sudo I moved this binary to /usr/local/bin (possibly an incorrect move - I now see that it can be installed easily using sudo apt install docker-compose).

Cloned the SIRF-SuperBuild.

From here on, most docker or docker-compose or ./sirf-compose commands would only run successfully to completion if I preceded them with sudo. (Even docker ps complains about permission denied for a Docker daemon socket unless I sudo it.)

We need a note that the first time you run sirf-compose it will take a long time to install everything.

I was unable to run the notebook due to lack of disk space. I extended my VM size from 20Gb to 40Gb. Then, within the Ubuntu VM, used the Disks app to increase the root filesystem partition size. (This seemed to work despite others on the internet having trouble - note I had to first Resize an overlapping partition before I could Resize the partition I needed).

I failed to realise at first the distinction between sirf-compose and sirf-compose-server.
However, once clear, the Jupyter notebook works and I ran a SIRF example (note the delay before localhost:9999 will work can be more than 'a few seconds'). This was a PET example. When I tried to start Gadgetron from a Jupyter terminal, there was a conflicting port for the REST port, I tried a different number and got a bit further but still an error.

Following the README instructions, the SIRF command line interface works, including Gadgetron. I noticed that the grappa_basic.py creates a file called output.h5.h5 so something is wrong there.

So, thanks for setting up all the Docker stuff. Any feedback on the above would be welcome.

[KrisThielemans]

hi @DANAJK I've converted this to a Discussion (new github feature, I've just enabled it for this repo) which seems more appropriate.

Your experiments prompted me to try this as well. Reactions and some comments from my own:

• I've tried to improve the README a bit according to what you suggest in Small Docker additions #463
• I've pointed the Wiki page to the README such that there's only one copy of the information.
• I first thought "why did David not just use Docker on Mac", but then I tried this on Windows 10 Pro, and discover that VirtualBox still doesn't like having Hyper-V enabled, so did exactly the same thing! It's amazing that it works really. Presumably a bit slower, but still fine.
• I do not have the sudo problems. Possibly something to do with your docker-compose installation, or maybe you cloned with sudo? However, I do have a possibly related issue with permissions: docker images and permissions problems for /devel #464

I haven't tried starting gadgetron on the docker service image yet.

[DANAJK]

I tried it first in a VM in case I ended up doing sudo etc and didn't want to mess up a new Mac.

I have since been trying it on another Mac Book using Docker directly without an explicit VM. Installation was smooth but at the moment I can't get Gadgetron running...

[DANAJK]

Another reason for trying it in a Linux VM was that I thought it might be more similar to eventual XNAT deployment.

[KrisThielemans]

good thought, although we probable will have to use Singularity as opposed to Docker.

[DANAJK]

I have now done Docker's 101 tutorial - which I can recommend: https://www.docker.com/101-tutorial . (If you are on a Mac, I suggest regularly having a look at the Docker Dashboard to see the effect of the command line commands). I have a couple of questions:
1.) In the 101 tutorial, docker compose is used to form a collection of docker containers that make up an app. In SIRF, although we use a lot of external functionality, in a running version, there seems to be only one container - why is this?

2.) I see that the possibility of doing docker pull synerbi/sirf:<DOCKER_TAG> is mentioned in our README. Does this pull from Docker Hub a complete pre-built image? What are the advantages/disadvantages of pulling compared to following the other instructions in the README?

[KrisThielemans]

1.) In the 101 tutorial, docker compose is used to form a collection of docker containers that make up an app. In SIRF, although we use a lot of external functionality, in a running version, there seems to be only one container - why is this?

Because we need the external functionality as link stage. It should be possible to have one container for the jupyter server+SIRF, and one for gadgetron. This is WIP #215

2.) I see that the possibility of doing docker pull synerbi/sirf:<DOCKER_TAG> is mentioned in our README. Does this pull from Docker Hub a complete pre-built image? What are the advantages/disadvantages of pulling compared to following the other instructions in the README?

yes, I guess advantage of pull is it is faster but disadvantage is a ~6GB download.

[KrisThielemans]

I currently experience very slow start-up of a new container. (./sirf-compose up --no-start sirf is fast, but docker start -ai sirf the first time takes ~3-4 minutes). I don't quite understand what it's running then, as I thought the machine would be prebuild.

When doing this with a service instance and pressing Ctrl-C a few times, it turned out it was busy doing the conda install in

SIRF-SuperBuild/docker/user_service-ubuntu.sh

Line 10 in f4b8f14

conda install -c conda-forge -y --file requirements-service.txt || \

I have therefore 2 questions:

• why is user_service-ubuntu.sh run at this time?
• conda install can be very slow these days. Any reason to prefer it over pip?

[casperdcl]

ah nobody @ mentioned me so never saw this thread.

• remove need for sudo: https://docs.docker.com/engine/install/linux-postinstall/
• pretty sure docker-compose comes with the latest version of docker. If not, it can be installed via apt, pip, or even direct download
• gadgetron is auto-started when running service containers so that's why there's a port conflict (with itself) if you try to start a second instance manually
• first startup can take a while as it downloads & unzips SIRF-Exercises required data
• conda install is more reliable than pip in terms of version conflict handling (which is why it's slower - it checks for conflicts). However, pip is increasingly getting slower and more stable
• docker pull versus letting docker-compose build: yes, the choice is between large download or long build time

[KrisThielemans]

nobody @ mentioned me so never saw this thread.

didn't want to disturb you 😉

gadgetron is auto-started when running service containers so that's why there's a port conflict (with itself) if you try to start a second instance manually

aha. We should make that clear in the README then.

first startup can take a while as it downloads & unzips SIRF-Exercises required data

ok. I might vote for now downloading data then. We should at least write a message. However, that doesn't seem to be what is creating my problem, as that should happen after installation of jupyter etc I believe.

conda install is more reliable than pip in terms of version conflict handling (which is why it's slower - it checks for conflicts).

ok, but it shouldn't take 2-3 minutes (although I'm not sure that's what's going on, as I don't know a way to monitor the docker instance start-up, aside from docker logs but that doesn't show anything).

I'm probably biased against conda these days due to an open issue where it just hangs on install for many (including myself on my previous laptop), see conda/conda#10216

[KrisThielemans]

Closing this as I believe all of this has been resolved, or is out-of-date