Updating apt repositories fails on Slurm images based on Ubuntu 20.04

[tpdownes]

Describe the bug

Cannot update apt repository metadata on VMs using the Slurm image based upon Ubuntu 20.04.

Steps to reproduce

Steps to reproduce the behavior:

1. apt-get update (or apt update) fails with a message similar to

E:Failed to fetch https://downloads.whamcloud.com/public/lustre/latest-feature-release/ubuntu2004/client/Packages 404 Not Found [IP: 138.197.209.17 443], E:Some index files failed to download. They have been ignored, or old ones used instead

Expected behavior

apt-get update should succeed without error

Workaround

Run sudo rm -f /etc/apt/sources.list.d/lustre-client.list and re-run apt-get update.

Planned fix

We will update the Whamcloud (DDN) lustre repository URL in future builds of Slurm Ubuntu 20.04 images. We are also planning a migration for the a3-highgpu-8g blueprint away from Ubuntu 20.04 to a more recent Ubuntu release.

[tpdownes]

This runner in as the first runner in startup script can implement this:

      - type: shell
        destination: workaround_apt.sh
        content: |
          #!/bin/bash
          # this script is no longer necessary on the most recent TCPX A3
          # images, however it is included for backwards compatibility
          set -e -o pipefail
          rm -f /etc/apt/sources.list.d/lustre-client.list
          apt-get update

See these links for usage of startup scripts on Slurm infrastructure:

• cluster-toolkit/examples/machine-learning/a3-highgpu-8g/ml-slurm-a3-2-cluster.yaml

  Lines 108 to 111 in 26fafe0

  	- id: a3_startup
  	source: modules/scripts/startup-script
  	settings:
  	# When shutting down a VM with local SSD disks, we strongly recommend the

• cluster-toolkit/examples/machine-learning/a3-highgpu-8g/ml-slurm-a3-2-cluster.yaml

  Lines 165 to 171 in 26fafe0

  	- id: a3_nodeset
  	source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
  	use:
  	- sysnet
  	- gpunets
  	- compute_sa
  	- a3_startup

[github-actions]

This issue is stale because it has been open for 30 days with no activity.