Prepare for Docker Hub limits

[melange396]

In doing a recent release of delphi-epidata, during the CI "build" process, an error was encountered from docker stating "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading" (more complete error text can be found in a collapsed section at the bottom of this message). Docker's documentation suggests this should not have affected us ("The Docker Hub plan limits will take effect on March 1, 2025"), so I am guessing that they prematurely, accidentally, and temporarily enabled rate limiting. A subsequent re-run of the build ran successfully.

If [when] a similar error occurs in the future (the Docker limits are scheduled to be enforced starting 4 weeks from today), in addition to creating/upgrading our Docker Hub user tier(*), we may have change our GitHub Actions workflows to do a Docker login step (like we do for the GitHub Container Registry) before any operations like the one that errored.

Alternatively, we might potentially be able to switch to self-hosted runners and deal with the unauthenticated, IP address based rate limit of 10 pulls per hour per address. We will surely have very infrequent success at best if we try to rely on the address based limits inside the GitHub Actions runner pool.

There are other, more drastic ways of skirting these limits (like not using Docker at all) but they are probably not worth jumping into without more detailed investigation.

(*)NOTE: I don't know what kind of Docker or Docker Hub account we have nor what subscriptions are attached to it, or if we even have such a thing already.

expanded error text:

from https://github.com/cmu-delphi/delphi-epidata/actions/runs/13040470808/job/36380975909?pr=1589#step:12:1561

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 156B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/redis:latest
#2 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/redis/manifests/sha256:ca65ea36ae16e709b0f1c7534bc7e5b5ac2e5bb3c97236e4fec00e3625eb678d: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
------
 > [internal] load metadata for docker.io/library/redis:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM redis
   2 |     
   3 |     ENV REDIS_PASSWORD=$REDIS_PASSWORD
--------------------
ERROR: failed to solve: redis: failed to resolve source metadata for docker.io/library/redis:latest: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/redis/manifests/sha256:ca65ea36ae16e709b0f1c7534bc7e5b5ac2e5bb3c97236e4fec00e3625eb678d: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Error: Process completed with exit code 1.

[korlaxxalrok]

Looks like they've updated the deadline to April 1, so we have a bit of time still.

I think we still probably fall into the category of the free user and should target something like the following:

Updated Pull Rate Limits for Free & Unauthenticated Users – To ensure a reliable and seamless experience for all users, we are updating authenticated and free pull limits:

    Unauthenticated users: Limited to 10 pulls per hour (as announced previously)
    Free authenticated users: Increased to 100 pulls per hour (up from 40 pulls / hour)
    System accounts & automation: As previously shared, automated systems and service accounts can easily authenticate using Personal Access Tokens (PATs) or Organizational Access Tokens (OATs), ensuring access to higher pull limits and a more reliable experience for automated authenticated pulls.

[korlaxxalrok]

I set up a personal/free user and generated a token. These details are stored in Bitwarden.

Need to figure out:

• Where in Actions to auth and use the token
• Where in our container system we need to auth and use the token
  • Is there something specific to do to use both our private repo via GitHub and Docker Hub's public one? We pull images from both in some cases.

[korlaxxalrok]

We have a suitable authenticated free account to most likely get around most pull rate issues we might encounter.

These credentials are available in Actions, we just need to put them into use if/when necessary.

@melange396 Is this "good enough" for now, or do we want make any preemptive changes to existing CI processes?

[melange396]

Yeah, that is sufficient for now. I will try to trigger it on Tuesday to see what happens. For posterity, ill repeat here the identifiers for the credentials secrets that you shared in slack:

${{ secrets.DELPHI_DOCKER_HUB_USER }}
${{ secrets.DELPHI_DOCKER_HUB_TOKEN }}

[melange396]

I half-heartedly tried to trigger this earlier, but was not affected by any limiting. Perhaps they have not turned on enforcement, perhaps i got lucky, or maaaaybe github actions will be exempt from this? Documentation for this is very hard to find, but comments at https://news.ycombinator.com/item?id=43127689 imply that it could be the case.