OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint #357
Comments
|
Additionally (this may be completely unrelated, if so I will open a new issue) intermittently we are getting this error from the action |
|
I found a tweet. https://twitter.com/arkadiyt/status/1481418230082248714
|
|
Thanks @suzuki-shunsuke I've also noticed there’s a PR open already, however the thumbprints are different! https://github.com/aws-actions/configure-aws-credentials/pull/355/files |
|
I could obtain the new thumbprint as the AWS official doc https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html. % curl https://token.actions.githubusercontent.com/.well-known/openid-configuration
{"issuer":"https://token.actions.githubusercontent.com","jwks_uri":"https://token.actions.githubusercontent.com/.well-known/jwks","subject_types_supported":["public","pairwise"],"response_types_supported":["id_token"],"claims_supported":["sub","aud","exp","iat","iss","jti","nbf","ref","repository","repository_owner","run_id","run_number","run_attempt","actor","workflow","head_ref","base_ref","event_name","ref_type","environment","job_workflow_ref"],"id_token_signing_alg_values_supported":["RS256"],"scopes_supported":["openid"]}%
% openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443
CONNECTED(00000005)
...
1 s:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
-----BEGIN CERTIFICATE-----
MIIE6jCCA9KgAwIBAgIQCjUI1VwpKwF9+K1lwA/35DANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
QTAeFw0yMDA5MjQwMDAwMDBaFw0zMDA5MjMyMzU5NTlaME8xCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxKTAnBgNVBAMTIERpZ2lDZXJ0IFRMUyBS
U0EgU0hBMjU2IDIwMjAgQ0ExMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAwUuzZUdwvN1PWNvsnO3DZuUfMRNUrUpmRh8sCuxkB+Uu3Ny5CiDt3+PE0J6a
qXodgojlEVbbHp9YwlHnLDQNLtKS4VbL8Xlfs7uHyiUDe5pSQWYQYE9XE0nw6Ddn
g9/n00tnTCJRpt8OmRDtV1F0JuJ9x8piLhMbfyOIJVNvwTRYAIuE//i+p1hJInuW
raKImxW8oHzf6VGo1bDtN+I2tIJLYrVJmuzHZ9bjPvXj1hJeRPG/cUJ9WIQDgLGB
Afr5yjK7tI4nhyfFK3TUqNaX3sNk+crOU6JWvHgXjkkDKa77SU+kFbnO8lwZV21r
eacroicgE7XQPUDTITAHk+qZ9QIDAQABo4IBrjCCAaowHQYDVR0OBBYEFLdrouqo
qoSMeeq02g+YssWVdrn0MB8GA1UdIwQYMBaAFAPeUDVW0Uy7ZvCj4hsbw5eyPdFV
MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
EgYDVR0TAQH/BAgwBgEB/wIBADB2BggrBgEFBQcBAQRqMGgwJAYIKwYBBQUHMAGG
GGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBABggrBgEFBQcwAoY0aHR0cDovL2Nh
Y2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xvYmFsUm9vdENBLmNydDB7BgNV
HR8EdDByMDegNaAzhjFodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vRGlnaUNlcnRH
bG9iYWxSb290Q0EuY3JsMDegNaAzhjFodHRwOi8vY3JsNC5kaWdpY2VydC5jb20v
RGlnaUNlcnRHbG9iYWxSb290Q0EuY3JsMDAGA1UdIAQpMCcwBwYFZ4EMAQEwCAYG
Z4EMAQIBMAgGBmeBDAECAjAIBgZngQwBAgMwDQYJKoZIhvcNAQELBQADggEBAHer
t3onPa679n/gWlbJhKrKW3EX3SJH/E6f7tDBpATho+vFScH90cnfjK+URSxGKqNj
OSD5nkoklEHIqdninFQFBstcHL4AGw+oWv8Zu2XHFq8hVt1hBcnpj5h232sb0HIM
ULkwKXq/YFkQZhM6LawVEWwtIwwCPgU7/uWhnOKK24fXSuhe50gG66sSmvKvhMNb
g0qZgYOrAKHKCjxMoiWJKiKnpPMzTFuMLhoClw+dj20tlQj7T9rxkTgl4ZxuYRiH
as6xuwAwapu3r9rxxZf+ingkquqTgLozZXq8oXfpf2kUCwA/d5KxTVtzhwoT0JzI
8ks5T1KESaZMkE4f97Q=
-----END CERTIFICATE-----
...
# saved the root certificate as cert.crt
% openssl x509 -in cert.crt -fingerprint -noout | sed -e 's/://g'
SHA1 Fingerprint=6938FD4D98BAB03FAADB97B34396831E3780AEA1
% openssl x509 -in Downloads/cert.crt -fingerprint -noout | sed -e 's/://g' | tr '[:upper:]' '[:lower:]'
sha1 fingerprint=6938fd4d98bab03faadb97b34396831e3780aea1It seems the new thumbprint is |
|
FYI you can also run this command: openssl s_client -connect token.actions.githubusercontent.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31Just remove the |
|
How to get new thumbprint in advance? This will break once every year. |
|
@jakejscott that command is giving a different fingerprint because you take the first cert of the chain, not the last as doc suggests.
|
|
Does anyone know if this was this replaced due to expiration of the old cert, or a security issue? I'd like to know whether to add the new cert in addition to the old one, or should I replace the old thumbprint. |
|
For those using terraform to manage the OIDC provider in AWS: data "tls_certificate" "github" {
url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
client_id_list = ["sts.amazonaws.com"]
} |
|
I understand there's a new thumbprint, but how exactly do we apply this to the github action? Can we provide the new thumbprint here, or what exactly must be done? Does this get applied on the AWS side? A bit confused. but still researching. |
|
@altjx has to be updated in AWS IAM --> Identity Providers -->(token.actions.githubusercontent.com) |
|
FYI, even after 12h+ since the change, I'm still getting two different cert chains from the same hostname. Simple test: while true; do echo | openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443 2>&1 | grep -c 'BEGIN CERTIFICATE'; sleep .1; doneIn ~30% of cases, the chain has 3 certificates, instead of 2: vs Fortunatelly, this is not a problem for OIDC, since Amazon is getting the same cert chain every time. At least from what I have seen. This problem may be ISP/country specific. |
|
Is there recommended course of action for mitigation? Spent an hour thinking I broke something until I found this thread. |
|
There is a viable mitigation here: rotating certificates includes the possibility of changing trust chains in the course of that. What Github could do is publish the trust bundles for their expected CAs, users retrieve that bundle, extract the thumbprints for all valid roots. This is a list, so you can provide multiple fingerprints so that you can tolerate getting different CAs per connection while a rotation is happening, since this is a distributed consistency problem that shouldn't be taken for an atomic operation for clients. Publishing trust bundles is in fact what Google does for the GCP IdP, but here it seems that the AWS resource definition requires not just the trust vector but enumeration of all the certificates used. Trust vector and name really should be sufficient, but Github could still give users relief by publishing the certs with their chains. This is preferable to having to look at CT to look for all issuance against the host name, retrieving all valid roots and passing those, as CT is a detective rather than a preventive control against incorrect issuance. If you want to do pinning right, which you generally should for an IdP, you need to verify the root of trust for first use, can use that root to pull the bundle thereafter for updates. You could also pull published trust bundles then look at what's in CT to determine, but this really points to trust qualification being fairly wonky in the OIDC/OAuth world, which generally expect you either to trust a name via whatever WebPKI universe you accept, or you trust a specific key. AWS wants the latter, the sane middle ground is that you want a name and a specific trust vector, such that you can give trust bundles as opposed to individual HTTPS certificate. Whatever should be done, there's a bunch of complexity that really should be hidden via an IdP datasource provider. |
|
@buffyg aws allows upto 5 thumbprint . (just a consideration ) |
|
Larger point:
But the point is that getting all possible server certificates during a rotation while pinning is not something that a client has ready visibility into because you aren't supposed to need to know that. Such are the mechanics of TLS with WebPKI: you don't look up a key or set of keys for a name, you connect to the thing with that DNS name, you retrieve a cert, validate it against your set of trust anchors. Self-selected published trust vectors so that you pin to issuance path rather than individual certificates, that's the most tractable form because it follows the design of TLS+WebPKI, and it still needs publication by a side channel because that's not what JOSE/JWKS provides for. Github as things stand just does plain WebPKI, as best as I can see. All possible server certificates, that's still potentially tractable by adding a side channel for publishing all certificates that are meant to be in use, but it's not at all how things were designed to work in the first place, is treating WebPKI like self-signed keys which therefore all need to be published, hence the fragility in handling rotations. |
|
I'm pretty confident I've accurately described the way CA pinning and related trust bundle publication is supported by the GCP IdP because the details are my best recollection of what I got directly from @agl, but I'll tag him here in case I'm mistaken. |
|
Duplicate of #355. |
That's a "Pull Request" and this is an "Issue" |
|
👋🏾 Thanks all for spotting this and sorry about the issue. We've issued a change log here: https://github.blog/changelog/2022-01-13-github-actions-update-on-oidc-based-deployments-to-aws/. This is something we'll be looking into to ensure this doesn't happen again. |
|
👋 To expand on Andy's comment, I'd like to answer a few questions raised.
@danieljamesscott This was a routine rotation due to upcoming expiration. As such, you do not need to remove the previous thumbprint, but may do so as it is no longer in use.
@stevie- This issue was caused by the intermediate CA changing in our certificate's trust chain, which we don't expect to necessarily occur during every SSL rotation/renewal. We're investigating here to determine how to avoid this changing when possible, and will ensure adequate communication in the future if it does need to change and user action is required.
@buffyg Thanks for your thorough analysis, we'll be looking into options here. The AWS documentation implies there may be some options here to reduce the need for customers to update the thumbprint themselves, although our engineering team will need to investigate if they are feasible. Again, I'd like to apologize for this issue and reiterate that we are taking steps to ensure this doesn't happen again. |
|
While it's always annoying to have something like this break deployments, I will say it's quite nice that I was able to:
We've all had to deal with support hell and black boxes when it comes to issues and resolutions before. The community, collaboration, and transparency here is refreshing! |
I really like that solution, but how secure is it, especially in terms of man-in-the-middle attacks? |
What are the security implications of having the wrong fingerprint? The Idp queries will still go to token.actions.githubusercontent.com ? Also out of interest why are we querying https://token.actions.githubusercontent.com/.well-known/openid-configuration rather than just https://token.actions.githubusercontent.com to get the cert information? It works well for me by the way (using both urls), thanks! |
@zarenner I've also talked to AWS about making the underling trust ceremony corporate as opposed to per-customer. They already trust a number of IdPs, suggested they do the same with you, the mechanics can be between your engineering orgs. I suggested using WebPKI issuance pins, that's what AWS says, which is then garbled by Terraform (looking at the AWS docs, it starts out talking about using CA pins, but the Terraform resource docs instead suggest these should be server thumbprints, not CA, "A list of server certificate thumbprints for the OpenID Connect (OIDC) identity provider's server certificate(s)"). The AWS doc, which gives the procedure for relying on root CAs, is https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html and the Terraform resource doc is https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_openid_connect_provider. I've since been pointed to a write-up from @sleevi on considerations for OIDC FastFed, suggesting that subsetting WebPKI issuance has viability issues and that Private PKI should be used instead. Also should be pointed out that DigiCert has reduced the lifetime of their ICAs precisely to discourage pinning via ICA: https://knowledge.digicert.com/alerts/DigiCert-ICA-Update.html Sleevi's notes are here: https://docs.google.com/document/d/e/2PACX-1vQQ-FhNjW0ZhZVTnK1VG_87IBNZKBaJmweYZb1VBRdQCMAWXekqKfxdNl8wL6FWFDJ9pxXxpr66-GZp/pub |
|
@zarenner mentioned wanting to explore ways to avoid having the issuing CA change. To echo and emphasize @buffyg’s comment, the WebPKI doesn't provide that guarantee, somewhat intentionally. DigiCert has been explicit about this, and it’s a huge credit to them for implementing organizational changes to help surface these risks during routine situations (such as rotations), where folks are aware and can respond to consequences, because the alternative is these sorts of things get surfaced during non-routine situations, like responses to CA compromise or distrust. A big part of this philosophy was seeing how long it took organizations to successfully transition off certificates from DigiNotar, WoSign/StartCom, and Symantec, three CAs that demonstrated significant security control failures. For an unfortunately large number of users, they had trouble rotating certificates precisely because they had end users who had pinned. Beyond such cases, though, the WebPKI ecosystem has intentionally been moving to shorter (effective) lifetimes to root certificates. When you use a 10-year old root, for example, you also inherit 10 years of any mistakes it has made potentially causing harm to your organization. The old Verizon PKI demonstrated this best: Verizon did not even know everyone they had issued sub-CAs to, any of which could have maliciously issued certificates that anyone who trusted the Verizon roots would accept. DigiCert, when they acquired Verizon, spent an incredible amount of time investigating and revoking those certificates, because that’s a huge risk. As one of the authors of RFC 7469, and as one who helped drive removing support from Chrome, I’ve definitely been trying to help educate folks about the dangers of pinning. Since we’re engineers here, the way I would describe Certificate pinning is a bit like reaching into an opaque C struct: you’re reaching into implementation details that aren’t ABI or API guaranteed. That is, if you had a typedef struct FOO_internal FOO;then it would be inherently unsafe in your libraries void baz(FOO* foo) {
*((int*)((char*)(foo) + 12)) = 42;
}because you’re reaching in to the internal implementation details and “violating” the API contract. Pinning to the WebPKI is basically that: reaching into implementation details of the trust graph that, by design, aren’t guaranteed to be stable. If you need a stable API, then private PKI let’s you define those contracts as you need. Hopefully this doesn’t come off as a lecture 😅 It’s just very much “here be dragons”, and any pinning to the Web PKI is going to (eventually) explode, because just like software itself needs to get regularly updated to respond to security risks, so does the WebPKI. Note: while Google does publish a root bundle at https://pki.goog, that’s not really for encouraging pinning, and you’ll note it’s a broad selection of CAs. However, even with that broad selection, there have still been operational issues where Google ends up getting a certificate outside of that set, and interoperability issues can come up. So publishing a set of trust anchors isn’t much of a solution: it’s primarily there to ensure at least a minimum set or “everyone Google may decide to do business with” is included. |
|
@adudek blame is on AWS, not GitHub: #357 (comment) Not trusting TLS certificate chains issued by CAs and requiring users to hardcode fingerprints of certificates they don't operate is poor design by AWS. |
|
@n1ngu by this extension, blame all OIDC implementers that are not willing to support github out of the box, just because github doesn't understand contracts nor it provides means to update those automatically. Do I understand correctly, that we expect every vendor to read github announcements and copy-paste its configuration to their solution, also bending their release cycles to satisfy githubs lack of insight? |
|
Technically GitHub did nothing wrong here. Nowhere does OIDC spec mandate pinning to intermediate TLS certificates.
We should. because if they're doing pinning they're not following OIDC by the letter. OIDC actually says |
I'm willing to accredit blame to AWS if GitHub will provide means to automatically supply all required OIDC information (yes, all certificates in use) and if it will not rely on reading their announcements. In the current situation even if AWS wanted - it could not implement it properly, because someone needs to be informed that the change was made. In this scenario, all blame would be put on AWS. Is it 90's again or what? |
That would be necessary if GitHub was deploying self-signed certificates. The whole point of using CA certificates is not having to announce certificate rotations. If AWS trusted CA certificates none of this would happen. And this is unrelated to OIDC specs. |
GitHub already publishes all the information needed programatically and in compliance with OIDC. Namely PKI. their jwks_uri is signed by a globally trusted root CA. AWS should perform TLS verification like everyone else in the world and not come up with alternative schemes like pinning intermediate certificates. Only thing AWS needs to do is verify the certificate chain as per the TLS spec. That they don't do this is extremely weird. |
I can only assume it is because of services like EKS that use self-signed certificates for their control plane and OIDC endpoints. |
What if PKI Consortium decides to host (part of) its services with AWS? Would You trust this solution? I'd trust only pinned information that was vetted. |
|
How are you suggesting to retrieving that pin from GitHub? Through TLS. And we're back to square one. WebPKI is safe. Certificate Transparency log makes abuse obvious. Pinning is unnecessary complexity. |
|
Hi everyone, let's try and keep the discussion here related to this action only. The way thing are today:
The path forward to making this a better experience involves changing how IAM works, or how GitHub is managing their certificates, or both. I don't have any information to share today other than the IAM team is aware of this sharp edge and we want to make it better in the future. If I have anything to announce there, I'll share that as soon as possible. We've updated the readme and the example CloudFormation template for this action's documentation. I'd also note that if you're using the CDK OpenIdConnectProvider construct, you will need to manually configure thumbprints as well. If anyone has any suggestions on how to make our documentation and setup process clearer, we'd be happy to take that feedback. |
|
I had updated our OIDC provider with the new fingerprints but still ran into the same issue. Don't forget to update not only the OIDC provider in the account where the runners are hosted, but also in other accounts from which roles are being assumed. |
Thanks! Works for me |
Confirmed this also worked for me, I have both |
peterwoodworth
added
service-limitation
|
I am using a GitHub self-hosted runner in an AWS AutoScaling Group (ASG). The IAM Role that is used has both thumbprints.
But I was still receiving the error below.
I had to recreate the EC2 Instance by scaling down and up the ASG to get the error to go away. Has anyone else experienced this issue with self-hosted runners? |
|
@hxrsmurf I saw the exact same error persist for a few jobs on the GitHub hosted runners even though I had the full history or thumbprints configured for OIDC:
See:
Note that it was only a few jobs out of many and some of the jobs were in workflows for which there were other successful jobs. The errors have only stopped in the last couple of days (not exactly sure when as I left it for a while). Strangely all the errors were from jobs targeting All I can say is all my jobs on GitHub hosted runners are now working but there were peristent errors for a few days despite having all thumbprints configured. |
|
Hello! @wellsiau-aws beat me to announcing this slightly, but I have some changes to share: the IAM team have released an update for thumbprint configuration for GitHub (and only GitHub at this time) that will permanently solve this issue! As of July 6, 2023, it is no longer necessary to specify a specific thumbprint in your IAM Identity Provider configuration. AWS will secure communication with tokens.actions.githubusercontent.com using our library of trusted CAs rather than using a certificate thumbprint. You should see a message in your AWS account if you look at the configured identity provider confirming that this is the case. The API still appears to require that some thumbprint is configured, since that will require other API and UI changes, but your thumbprint will be completely ignored and OIDC will still work. The message in the IAM console will persist to remind you of this. I successfully tested this out in my own AWS account by setting my configured thumbprint to Note: this change will not affect GHES customers, since they have a different hostname and issuer from their IdP. You should not need to make any changes to your existing configuration with this change. Please open up a separate issue in this repo if you're continuing to have problems with the OIDC configuration, or reach out to AWS Support. |
|
Comments on closed issues are hard for our team to see. |
|
It would be lovely to get screenshots of the before and after in the console. Just so we know what to expect. This sounds like an amazing improvement. How does this affect the terraform side of the world? |
|
@jensenbox here's the after. I removed one of the fingerprints, and overwrote the other with the thumbprint you see. Still works! |
and others
Hey,
For the past hour or so I’ve been getting this error while using this action:
Error: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprintI’ve checked the thumbprint and it’s correctly set, would this be an issue with GitHub, are there any known issues at the moment?
The text was updated successfully, but these errors were encountered: