Skip to content

Mount GCS via single host gcsfuse and expose in container #2294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dlwh wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Mount GCS via single host gcsfuse and expose in container #2294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion infra/marin-big-run.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker us-central2-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-us-central2 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-cluster-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker {{REGION}}-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount {{BUCKET}} /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -83,7 +88,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-eu-west4-a.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker europe-west4-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-eu-west4 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-eu-west4.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker europe-west4-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-eu-west4 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-us-central1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker us-central1-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-us-central1 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-us-central2-staging.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker us-central2-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-us-central2 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-us-central2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker us-central2-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-us-central2 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-us-east1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker us-east1-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-us-east1 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount

worker_setup_commands:
# delete any old ray session data
Expand Down
9 changes: 8 additions & 1 deletion infra/marin-us-east5-a.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ docker:
initialization_commands:
- which docker || (curl -fsSL https://get.docker.com -o get-docker.sh; sudo sh get-docker.sh; sudo usermod -aG docker $USER; sudo systemctl restart docker -f)
- yes | gcloud auth configure-docker us-east5-docker.pkg.dev
- which gcsfuse || (sudo apt-get update && sudo apt-get install -y curl gnupg && sudo mkdir -p /etc/apt/keyrings && curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg && echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt gcsfuse-$(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list >/dev/null && sudo apt-get update && sudo apt-get install -y gcsfuse)
- sudo sh -c "grep -q '^user_allow_other$' /etc/fuse.conf 2>/dev/null || echo user_allow_other >> /etc/fuse.conf"
- if [ -e /tmp/gcsfuse_mount ] && [ ! -d /tmp/gcsfuse_mount ]; then rm -f /tmp/gcsfuse_mount; fi
- mkdir -p /tmp/gcsfuse_mount
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --dir-mode 0777 --file-mode 0666 -o allow_other --only-dir gcsfuse_mount marin-us-east5 /tmp/gcsfuse_mount || true
# always run this because ray doesn't run with sudo
- sudo usermod -aG docker $USER
# we want to launch docker containers from inside docker, which means we need to loosen the permissions on the docker
Expand Down Expand Up @@ -87,7 +92,9 @@ setup_commands:
- echo 'export TPU_MIN_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_STDERR_LOG_LEVEL=3' >> $HOME/.bashrc
- echo 'export TPU_LOG_DIR=disabled' >> $HOME/.bashrc
- gcsfuse --implicit-dirs --client-protocol grpc --cache-dir /dev/shm --file-cache-max-size-mb 160000 --only-dir gcsfuse_mount $BUCKET /opt/gcsfuse_mount || true
# gcsfuse runs on the host (mounted at /tmp/gcsfuse_mount); expose it at the conventional in-container path.
- if [ -e /opt/gcsfuse_mount ] && [ ! -L /opt/gcsfuse_mount ]; then sudo rm -rf /opt/gcsfuse_mount; fi
- sudo ln -sfn /tmp/gcsfuse_mount /opt/gcsfuse_mount
Comment on lines +96 to +97
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The symlink created at /opt/gcsfuse_mount will not be accessible inside the Docker container because /opt is not mounted into the container. Only /tmp is mounted (line 48). For the mount to be accessible at /opt/gcsfuse_mount inside the container, you need to add a volume mount like "-v /opt:/opt" to both head_run_options and worker_run_options.

Copilot uses AI. Check for mistakes.
Copy link
Member Author

@dlwh dlwh Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lies

lrwxrwxrwx 1 root root 18 Jan  6 22:37 /opt/gcsfuse_mount -> /tmp/gcsfuse_mount
bash: gcsfuse_mount: command not found
a
dedupe
gcsfuse_mount
helmet-data
huggingface-cache
marin-us-central2
medu-models
models
nfliu
nvidia--Llama-Nemotron-Post-Training-Dataset-v1-ed905e6


worker_setup_commands:
# delete any old ray session data
Expand Down
Loading