Changes in $HOME lost after restarting workspace

[Adamtaranto]

Bug description

I have been using the Dockerfile below for the last year without issue to create workspaces with miniconda.

Miniconda is installed into the $HOME (/home/gitpod) dir. Recently changes in this dir such as installed packages or new envs have been failing to persist after restarting the workspace.

Has something changed on your end?
Should I install miniconda somewhere else now?

Steps to reproduce

Create new Gitpod workspace using these config files. Create a test env (i.e. conda env create -n testenv). Stop and restart the workspace.

Startup will give error on "jupyter nbextension" commands from .gitpod.yml as packages previously install by init: pip3 install -r requirements.txt are gone. Also conda env list will show that testenv is gone.

.gitpod.Dockerfile

# Use the Gitpod base image for Python 3.11
FROM gitpod/workspace-python-3.11

# Set user
USER gitpod

# Install Miniconda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
    /bin/bash ~/miniconda.sh -b -p $HOME/miniconda && \
    rm ~/miniconda.sh

# Add Miniconda to PATH
ENV PATH="$HOME/miniconda/bin:$PATH"

# Initialize conda in bash config fiiles:
RUN conda init bash

# Set up Conda channels
RUN conda config --add channels conda-forge && \
    conda config --add channels bioconda && \
    conda config --set channel_priority strict

# Set libmamba as solver
RUN conda config --set solver libmamba

# Remove the undesired default Python location from PATH
RUN export PATH=$(echo $PATH | tr ':' '\n' | grep -v '/home/gitpod/.pyenv/shims' | tr '\n' ':')

.gitpod.yml

image:
  file: .gitpod.Dockerfile

tasks:
  - name: Install Python packages from requirements.txt
    init: pip3 install -r requirements.txt
    command: |
      jupyter nbextension install --sys-prefix --py nbgrader
      jupyter nbextension enable --sys-prefix --py nbgrader
      jupyter labextension enable nbgrader

vscode:
  extensions:
    - eamodio.gitlens
    - janisdd.vscode-edit-csv
    - mechatroner.rainbow-csv
    - ms-python.python

requirements.txt

nbgrader==0.8.5
numpy
pandas
seaborn
matplotlib
pysam
biopython
regex
scipy

Workspace affected

melbournebi-comp9001620-96dowtuzt7i

Expected behavior

New envs and installed packages in $HOME should persist after restarting a stopped workspace.

Example repository

No response

Anything else?

This seems to have become an issue some time after November last year.

[Adamtaranto]

Also edits to ~/.condarc are lost after restarting the workspace.

[axonasif]

Hi @Adamtaranto, I've followed up on your support ticket, please take a look there.

[Adamtaranto]

Hi @axonasif, please reopen this issue. It is not solved.

I have tried installing miniconda into the root directory of my project using the env variable $GITPOD_REPO_ROOT in place of $HOME in the Dockerfile, but the build fails. I am also not able to write files to $GITPOD_REPO_ROOT from the Dockerfile. Maybe that variable does not yet exist during build?

How can I specify a location for miniconda that will persist between restarts?

[axonasif]

Hi @Adamtaranto, please try this dockerfile:

# Image source code: https://github.com/axonasif/workspace-images/tree/tmp
# Also see https://github.com/gitpod-io/workspace-images/issues/1071
FROM axonasif/workspace-python@sha256:f5ba627a31505ea6cf100abe8e552d7ff9e0abd6ba46745b6d6dab349c001430

# Set user
USER gitpod

# Install Miniconda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py311_24.1.2-0-Linux-x86_64.sh -O ~/miniconda.sh && \
    /bin/bash ~/miniconda.sh -b -p $HOME/miniconda && \
    rm ~/miniconda.sh

# Add Miniconda to PATH
ENV PATH="$HOME/miniconda/bin:$PATH"

# Initialize conda in bash config files:
RUN conda init bash

# Set up Conda channels
RUN conda config --add channels conda-forge && \
    conda config --add channels bioconda && \
    conda config --set channel_priority strict

# Set libmamba as solver
RUN conda config --set solver libmamba

# Persist ~/miniconda
RUN echo 'create-overlay $HOME/miniconda' > "$HOME/.runonce/1-miniconda"

Or, you can also create a workspace from here to try: https://github.com/allforks/nbgrader-gitpod-template/tree/patch

I have tried installing miniconda into the root directory of my project using the env variable $GITPOD_REPO_ROOT in place of $HOME in the Dockerfile, but the build fails. I am also not able to write files to $GITPOD_REPO_ROOT from the Dockerfile. Maybe that variable does not yet exist during build?

Yes, it doesn't. It's only mounted after you enter a running workspace

[Adamtaranto]

Hi Aminur,

Thanks! It looks like changes in ~/miniconda/pkgs now persist after restarting the workspace.

Is it possible to also preserve changes to the ~/.condarc config file between sessions?

[axonasif]

Hi @Adamtaranto

Yes! Just add the following to your dockerfile:

# Persist ~/.condarc
RUN echo 'create-overlay $HOME/.condarc' > "$HOME/.runonce/2-condarc"

The command is essentially this: create-overlay $HOME/.condarc.

But here we're creating a startup script at $HOME/.runonce, scripts put inside that dir will only run once without race conditions.

[Adamtaranto]

I had to do the same thing with /lib as some conda pkgs were installing stuff there.

Is it safe to remove the pyenv dirs from my PATH? I forget what the exact issue was but at some point having /home/gitpod/.pyenv/shims in there was causing problems.

PATH = /home/gitpod/miniconda/bin:/home/gitpod/miniconda/condabin:/home/gitpod/.pyenv/shims:/ide/bin/remote-cli:/home/gitpod/miniconda/bin:/home/gitpod/.pyenv/bin:/home/gitpod/.pyenv/userbase/bin:/home/gitpod/.local/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Will the new workspace image at axonasif/workspace-python@sha256:f5ba627a31505ea6cf100abe8e552d7ff9e0abd6ba46745b6d6dab349c001430 be available forever? It would be great to get an official GitPod image that does this if possible.

[axonasif]

Is it safe to remove the pyenv dirs from my PATH? I forget what the exact issue was but at some point having /home/gitpod/.pyenv/shims in there was causing problems.

If I remember correctly, this custom image does not add the shims dir to the path, do you still see it in $PATH?

[axonasif]

Will the new workspace image at axonasif/workspace-python@sha256:f5ba627a31505ea6cf100abe8e552d7ff9e0abd6ba46745b6d6dab349c001430 be available forever? It would be great to get an official GitPod image that does this if possible.

As long as my dockerhub account exists, it should be there.

There is a proposal to add this patch to the official image: gitpod-io/workspace-images#1071

[Adamtaranto]

Is it safe to remove the pyenv dirs from my PATH? I forget what the exact issue was but at some point having /home/gitpod/.pyenv/shims in there was causing problems.

If I remember correctly, this custom image does not add the shims dir to the path, do you still see it in $PATH?

Yep, I just tested it with the https://github.com/Adamtaranto/nbgrader-gitpod-template repo and it produces a PATH as above.

[axonasif]

In that case it's probably coming from pyenv side and not something we hooked into. Maybe there's a flag to disable that on pyenv but I don't remember it now. What you're doing also works just fine if you need to remove it from PATH.

[Adamtaranto]

This was working earlier, but now /home/gitpod/.pyenv/shims is back in the path when the workspace starts up and pip3 install call in .gitpod.yml is using /home/gitpod/.pyenv/shims/pip3 and is installing packages into the wrong location.

# Remove the undesired default Python location from PATH
RUN export PATH=$(echo $PATH | tr ':' '\n' | grep -v '/home/gitpod/.pyenv/shims' | tr '\n' ':')

Could /home/gitpod/.pyenv/shims be getting added back into the path after I have removed it?

The current test repo is at:
https://github.com/Adamtaranto/nbgrader-gitpod-template

[Adamtaranto]

Definitely something weird going on. I'm using the code below in my .gitpod.yml where I check the path and remove the unwanted /home/gitpod/.pyenv/shims, but oddly it comes back by the start of the command block.

tasks:
  - name: Init Gitpod Workspace
    init: |
      echo $PATH
      export PATH=$(echo $PATH | tr ':' '\n' | grep -v '/home/gitpod/.pyenv/shims' | tr '\n' ':')
      echo $PATH
      /home/gitpod/miniconda/bin/pip3 install -r requirements.txt
      jupyter nbextension install --sys-prefix --py nbgrader
      jupyter nbextension enable --sys-prefix --py nbgrader
      jupyter labextension enable nbgrader
      conda install --yes -c bioconda bwa blast samtools --solver=libmamba
      conda clean --all --yes
      echo "alias gogo='$GITPOD_REPO_ROOT/utils/gogo.sh'" >> $HOME/.bashrc
      source $HOME/.bashrc
    command: |
      echo $PATH
      export PATH=$(echo $PATH | tr ':' '\n' | grep -v '/home/gitpod/.pyenv/shims' | tr '\n' ':')
      echo "Running: $GITPOD_WORKSPACE_URL"
      echo "Call 'gogo a' or 'gogo w' to edit nbgrader assignments or workshops."

First echo $PATH in init (shims 3rd item):

/home/gitpod/miniconda/bin:/home/gitpod/miniconda/condabin:/home/gitpod/.pyenv/shims:/ide/bin/remote-cli:/home/gitpod/miniconda/bin:/home/gitpod/.pyenv/bin:/home/gitpod/.pyenv/userbase/bin:/home/gitpod/.local/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Remove shims in init block (no shims):

/home/gitpod/miniconda/bin:/home/gitpod/miniconda/condabin:/ide/bin/remote-cli:/home/gitpod/miniconda/bin:/home/gitpod/.pyenv/bin:/home/gitpod/.pyenv/userbase/bin:/home/gitpod/.local/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:

Echo $PATH at start of command block (shims is back!):

/home/gitpod/.pyenv/shims:/home/gitpod/miniconda/bin:/home/gitpod/miniconda/condabin:/ide/bin/remote-cli:/home/gitpod/miniconda/bin:/home/gitpod/.pyenv/bin:/home/gitpod/.pyenv/userbase/bin:/home/gitpod/.local/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:.  

I'm now forcing packages to be installed with the miniconda pip3 with /home/gitpod/miniconda/bin/pip3 install -r requirements.txt which works but is not ideal.

@axonasif do you have any idea why shims is no longer removed from the path by this line in my Dockerfile?:

RUN export PATH=$(echo $PATH | tr ':' '\n' | grep -v '/home/gitpod/.pyenv/shims' | tr '\n' ':')

[Adamtaranto]

Also, the alias that I am adding to .bashrc in the init block does not seem to be persisting after workspace restart.

echo "alias gogo='$GITPOD_REPO_ROOT/utils/gogo.sh'" >> $HOME/.bashrc

I have this line in my Dockerfile to persist the .bashrc file:

RUN echo 'create-overlay $HOME/.bashrc' > "$HOME/.runonce/4-bashrc"

[Adamtaranto]

Update: The unwanted /home/gitpod/.pyenv/shims gets added back to the start of the PATH which I run source $HOME/.bashrc

I can't see anything in obvious in.bashrc that would do this. @axonasif can you take a look?

[axonasif]

@Adamtaranto talking a look now.

Edit: check the support ticket.

[axonasif]

FYI, for future readers, the fixes can be found here: https://github.com/allforks/nbgrader-gitpod-template