Using ECR as a Private Image Registry

[metaskills]

I've read this page (https://docs.github.com/en/codespaces/codespaces-reference/allowing-your-codespace-to-access-a-private-image-registry#common-image-registry-servers) several times and gotten Docker Hub working correctly, but I can not get ECR working. I've created several users and IAM credentials using full access to ECR and still can not get the image to be read in via Docker or a service via dockerComposeFile.

Can someone confirm that values for each? For example, the docs say the server should be in the format of <aws_account_id>.dkr.ecr.<region>.amazonaws.com and I have done that. Am I correct that the <*>_CONTAINER_REGISTRY_USER is the Access Key ID and <*>_CONTAINER_REGISTRY_PASSWORD is the secret key? Also, has anyone gotten this to work for ECR?

[asciimike]

I don't believe that AWS ECR supports using the Access Key ID/secret key paradigm, I think you have to generate a short lived authorization token and refresh it (per their docs). We are planning on offering a secrets API (like what Actions provides) to enable you to programmatically refresh that token.

[metaskills]

So @asciimike are you saying that the feature on this page with regard to ECR a) does not work or b) will if I or Codespaces does something different?

My expectation was that given the key and secret I provided, Codespaces would have done the API calls needed to do the ECR login. For example, we use this GitHub Action (https://github.com/marketplace/actions/docker-login#aws-elastic-container-registry-ecr) and we already did the work to make sure the key/secret has IAM permissions to do sts:GetServiceBearerToken and everything in the Action just works.

In fact, I think we recently had to fix a bug where we had to add that sts:GetServiceBearerToken capabilities to our user. Let me test again real quick.

[asciimike]

We are not planning to expose the *_CONTAINER_REGISTRY_* variables, so the behavior will remain the same. The AWS_* ones would be regular secrets, and thus accessible by folks though (presumably people will want to use them for other tools).

[metaskills]

you set the following secrets: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
The AWS_* ones would be regular secrets...

I missed this the first time around... when you say AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY I assume you mean at the organization level? If so, that sounds like a bad idea. It would set this very specific use case (ECR Login) to a very general (and global) AWS credential namespace. That feels like a leaky abstraction. As an organization, what I would like is to be able to do is set *_CONTAINER_REGISTRY_* ENVs for ECR and have everything done for me and hidden from users. I feel open to solutions but not ones that leak into the user accessible ENV vars. Especially if those ENV vars could conflict with other use cases.

[asciimike]

You could provide it at a user or repo level to apply for a subset of users.

My initial thought was that ECR is a subset of the AWS CLI, and that having the AWS CLI credentials be plumbed through would be generally useful, as I've had requests for providing az and gcloud credentials as well. I recognize that these may make more sense to be set individually though, as it's a slightly different use case.

As I understand it, your preference would be to have *_CONTAINER_REGISTRY_USER=AWS_ACCESS_KEY_ID and *_CONTAINER_REGISTRY_PASSWORD= AWS_SECRET_ACCESS_KEY?

[metaskills]

As I understand it, your preference would be to have *_CONTAINER_REGISTRY_USER=AWS_ACCESS_KEY_ID and *_CONTAINER_REGISTRY_PASSWORD=AWS_SECRET_ACCESS_KEY?

That is correct. That would be the expected behavior and the least DX friction. The documentation could mention that this user needs the sts:GetServiceBearerToken to work. As is now I am doing a hacky rotate of the key/secret every hour via the worlds worst automation and duct tape. I would imagine I am the only person doing this for ECR integration and that changing the interface around to not require posting new keys every 12 hours would be a welcome change for anyone wanting the as advertised private registry integration.

[asciimike]

I agree, my only concern was folks would think that the user has to be aws rather than letting us do the key exchange. Unfortunately, this is probably going to be one of those features where you are forced to read the docs to see how it works.

I do think that we'll offer easier credential access for AWS/GCP/Azure tooling in the future though, since I'm seeing a lot of questions come up around this, but it likely be distinct from this feature.

[Chuxel]

@metaskills I have a near term workaround that takes advantage of initalizeCommand in devcontainer.json and uses the AWS CLI to sign in whenever a codespaces is created.

See https://github.com/chuxel/codespaces-ecr

There's slightly different secret names you set, and you can customize the names of the secrets in the event you need more than one across repos, etc.

We need to get first class support in place here, but I think this should unblock you. /cc: @asciimike

[Chuxel]

Hmmm. That's odd. Can you share your devcontainer.json?

[metaskills]

I simplified it a lot while debugging. All the way down to just an "image" property vs a compose, etc. I will email the create log output to Jonathan Carter and he can maybe share with you? Always ended here.

...
 * Starting OpenBSD Secure Shell server sshd
docker process exited with exit code 0
(*) Starting ssh server.

[Chuxel]

Yep that works. You can also try changing set -e to set +e in the event the script is erroring out in an odd way. Not quite sure why this would work in my ECR case, so it's actually possible something unrelated is going on as well. Does the image have a custom entrypoint or command?

[metaskills]

I think our script runs fine as the entire dev container is built. Not sure about the entrypoint, but if we are simply changing the method of docker login with the same image, how might it come into play?

[Chuxel]

Just adding this to the public issue for others since we've corresponded offline. It is unclear why this would be happening given the image build is complete and initalizeCommand fires on the host VM. Something in this particular image (or possibly dotfiles) seems to be tripping things up. We're having trouble reproducing. If anyone else hits this problem, if you send across a codespace ID, we can take a look at the codespace to dig in further.

[asciimike]

Per all the discussion here, we've solved this in a more elegant way which allows you to provide an access key ID and secret. The docs have been updated accordingly: https://docs.github.com/en/codespaces/codespaces-reference/allowing-your-codespace-to-access-a-private-image-registry#accessing-aws-elastic-container-registry

[metaskills]

THANK YOU! The docs look good. I did notice there were a few lower case _container_ that would be more consistent as _CONTAINER_.

[samsarkleio]

Hi @asciimike,

Thanks for this! I just wanted to double check what the intent and workflow are now after this change.

My assumptions are that if:

• we set *_CONTAINER_REGISTRY_SERVER=<aws_account_id>.dkr.ecr.<region>.amazonaws.com
• and we set *_CONTAINER_REGISTRY_USER = <AWS_ACCESS_KEY_ID>
• and we set *_CONTAINER_REGISTRY_PASSWORD = <AWS_SECRET_ACCESS_KEY>

...then during a Codespace initialization, something equivalent to the following is going to happen:

• aws ecr get-login-password --region <region> | sudo docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com.

By equivalent, I mean it may not literally use the aws cli, but it will hit the API in some fashion (maybe this API as you mentioned above) to get temporary login credentials, and pipe them into docker login.

And to make this all work, the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY of the user in question needs to have sts:GetServiceBearerToken rights.

Is that correct?

[asciimike]

@samsarkleio you're exactly right (and yes, we use the API rather than the CLI). You'll also need to make sure the user in question has read access on the image as well as the token swap permissions.

[samsarkleio]

@asciimike Awesome thank you!

One more question, is there any startup/initialization path where the *_CONTAINER_REGISTRY_* variables could ever be seen by users? Is it ever exposed in logs, or can it be grabbed from the box itself?

[asciimike]

I don't believe these variables are exposed, and even if they were, we'll have swapped them out for the short lived token by the time they could be available, so there isn't a risk of exposing the access key ID/secret key.