Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Importing Ca certificates with USE_SYSTEM_CA_CERTS is slower than it used to be #612

Open
achimwinter opened this issue Jul 24, 2024 · 15 comments · Fixed by #572
Open
Assignees
Labels
bug Something isn't working

Comments

@achimwinter
Copy link

achimwinter commented Jul 24, 2024

Please add the exact image (with tag) that you are using

21-jre-alpine

Please add the version of Docker you are running

Docker version 26.1.3, build b72abbb

What happened?

Since yesterday (23.07.2024) i get a exception connecting to my documentDB Database on AWS.
My setup is like this: in a builder stage i download the global-bundle.pem from aws and copy that to /certificates/aws-ca.crt.
Setting the Environment variable
ENV USE_SYSTEM_CA_CERTS=1
Which worked for me so far, except for images from yesterday on.
Since then i get a whole lot of log statements in the container about all the certificates imported, but the application fails when connecting to the database.

I also tried to use: "JRE_CACERTS_PATH" (from https://github.com/docker-library/docs/pull/2445/files) added to the CMD, but that also didnt work. (I am not using a non-root image)

Using an older image works still fine.

Relevant log output

2024-07-24T13:21:07.052+02:00	Importing keystore /tmp/tmp.KbBNoA to /opt/java/openjdk/lib/security/cacerts...
2024-07-24T13:21:07.754+02:00	Entry for alias naverglobalrootcertificationauthority successfully imported.
2024-07-24T13:21:07.755+02:00	Entry for alias teliasonerarootcav1 successfully imported.
2024-07-24T13:21:07.756+02:00	Entry for alias vtruseccrootca successfully imported.
2024-07-24T13:21:07.757+02:00	Entry for alias emsignrootca-g1 successfully imported.
2024-07-24T13:21:07.757+02:00	Entry for alias quovadisrootca3g3 successfully imported.
2024-07-24T13:21:07.758+02:00	Entry for alias secureglobalca successfully imported.
2024-07-24T13:21:07.758+02:00	Entry for alias microsoftrsarootcertificateauthority2017 successfully imported.
2024-07-24T13:21:07.759+02:00	Entry for alias ssl.comevrootcertificationauthorityecc successfully imported.
2024-07-24T13:21:07.759+02:00	Entry for alias szafirrootca2 successfully imported.
2024-07-24T13:21:07.759+02:00	Entry for alias ssl.comtlsrsarootca2022 successfully imported.
2024-07-24T13:21:07.760+02:00	Entry for alias quovadisrootca1g3 successfully imported.
2024-07-24T13:21:07.760+02:00	Entry for alias atostrustedrootrootcarsatls2021 successfully imported.
2024-07-24T13:21:07.840+02:00	Entry for alias autoridaddecertificacionfirmaprofesionalcifa62634068 successfully imported.
2024-07-24T13:21:07.842+02:00	Entry for alias securesignrootca11 successfully imported.
2024-07-24T13:21:07.843+02:00	Entry for alias telekomsecuritytlsrsaroot2023 successfully imported.
2024-07-24T13:21:07.843+02:00	Entry for alias isrgrootx2 successfully imported.
2024-07-24T13:21:07.843+02:00	Entry for alias isrgrootx1 successfully imported.
2024-07-24T13:21:07.845+02:00	Entry for alias digicertglobalrootca successfully imported.
2024-07-24T13:21:07.845+02:00	Entry for alias sectigopublicserverauthenticationrootr46 successfully imported.
2024-07-24T13:21:07.846+02:00	Entry for alias bjcaglobalrootca2 successfully imported.
2024-07-24T13:21:07.846+02:00	Entry for alias globalsignroote46 successfully imported.
2024-07-24T13:21:07.846+02:00	Entry for alias bjcaglobalrootca1 successfully imported.
2024-07-24T13:21:07.847+02:00	Entry for alias starfieldservicesrootcertificateauthority-g2 successfully imported.
2024-07-24T13:21:07.847+02:00	Entry for alias actalisauthenticationrootca successfully imported.
2024-07-24T13:21:07.848+02:00	Entry for alias tubitakkamusmsslkoksertifikasi-surum1 successfully imported.
2024-07-24T13:21:07.849+02:00	Entry for alias amazonrootca4 successfully imported.
2024-07-24T13:21:07.850+02:00	Entry for alias amazonrootca3 successfully imported.
2024-07-24T13:21:07.851+02:00	Entry for alias amazonrootca2 successfully imported.
2024-07-24T13:21:07.851+02:00	Entry for alias amazonrootca1 successfully imported.
2024-07-24T13:21:07.851+02:00	Entry for alias affirmtrustpremium successfully imported.
2024-07-24T13:21:07.852+02:00	Entry for alias haricatlsrsarootca2021 successfully imported.
2024-07-24T13:21:07.852+02:00	Entry for alias entrustrootcertificationauthority-g4 successfully imported.
2024-07-24T13:21:07.852+02:00	Entry for alias entrustrootcertificationauthority-g2 successfully imported.
2024-07-24T13:21:07.853+02:00	Entry for alias gdcatrustauthr5root successfully imported.
2024-07-24T13:21:07.853+02:00	Entry for alias atostrustedrootrootcaecctls2021 successfully imported.
2024-07-24T13:21:07.854+02:00	Entry for alias telekomsecuritytlseccroot2020 successfully imported.
2024-07-24T13:21:07.854+02:00	Entry for alias emsigneccrootca-g3 successfully imported.
2024-07-24T13:21:07.855+02:00	Entry for alias atostrustedroot2011 successfully imported.
2024-07-24T13:21:07.855+02:00	Entry for alias d-trustevrootca12020 successfully imported.
2024-07-24T13:21:07.855+02:00	Entry for alias anfsecureserverrootca successfully imported.
2024-07-24T13:21:07.856+02:00	Entry for alias certignarootca successfully imported.
2024-07-24T13:21:07.856+02:00	Entry for alias swisssignsilverca-g2 successfully imported.
2024-07-24T13:21:07.857+02:00	Entry for alias vtrusrootca successfully imported.
2024-07-24T13:21:07.857+02:00	Entry for alias digicerttlsrsa4096rootg5 successfully imported.
2024-07-24T13:21:07.859+02:00	Entry for alias comodoecccertificationauthority successfully imported.
2024-07-24T13:21:07.860+02:00	Entry for alias securetrustca successfully imported.
2024-07-24T13:21:07.860+02:00	Entry for alias cadisigrootr2 successfully imported.
2024-07-24T13:21:07.860+02:00	Entry for alias aaacertificateservices successfully imported.
2024-07-24T13:21:07.861+02:00	Entry for alias starfieldrootcertificateauthority-g2 successfully imported.
2024-07-24T13:21:07.861+02:00	Entry for alias buypassclass2rootca successfully imported.
2024-07-24T13:21:07.862+02:00	Entry for alias tuntrustrootca successfully imported.
2024-07-24T13:21:07.939+02:00	Entry for alias buypassclass3rootca successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias epkirootcertificationauthority successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias entrust.netcertificationauthority(2048) successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias certigna successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias cfcaevroot successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias emsignrootca-c1 successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias certumtrustednetworkca successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias securitycommunicationrootca3 successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias securitycommunicationrootca2 successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias oistewisekeyglobalrootgcca successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias usertrustrsacertificationauthority successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias trustwaveglobaleccp384certificationauthority successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias firmaprofesionalcaroot-aweb successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias swisssigngoldca-g2 successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias globalsign-3 successfully imported.
2024-07-24T13:21:07.943+02:00	Entry for alias certsignrootca successfully imported.
2024-07-24T13:21:07.943+02:00	Entry for alias globalsign-2 successfully imported.
2024-07-24T13:21:07.943+02:00	Entry for alias globalsign-1 successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias certumec-384ca successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias hipkirootca-g1 successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias twcaglobalrootca successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias trustwaveglobaleccp256certificationauthority successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias globalsignrootr46 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias entrustrootcertificationauthority-ec1 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias emsigneccrootca-c3 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias digicerttrustedrootg4 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias quovadisrootca2g3 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias trustwaveglobalcertificationauthority successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr4 successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr3 successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr2 successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr1 successfully imported.
2024-07-24T13:21:07.947+02:00	Entry for alias hellenicacademicandresearchinstitutionseccrootca2015 successfully imported.
2024-07-24T13:21:07.947+02:00	Entry for alias d-trustrootclass3ca22009 successfully imported.
2024-07-24T13:21:07.948+02:00	Entry for alias commscopepublictrustrsaroot-02 successfully imported.
2024-07-24T13:21:07.948+02:00	Entry for alias e-szignorootca2017 successfully imported.
2024-07-24T13:21:07.949+02:00	Entry for alias commscopepublictrustrsaroot-01 successfully imported.
2024-07-24T13:21:07.949+02:00	Entry for alias affirmtrustcommercial successfully imported.
2024-07-24T13:21:07.950+02:00	Entry for alias godaddyclass2certificationauthority successfully imported.
2024-07-24T13:21:07.950+02:00	Entry for alias digicertassuredidrootg3 successfully imported.
2024-07-24T13:21:07.950+02:00	Entry for alias affirmtrustnetworking successfully imported.
2024-07-24T13:21:07.951+02:00	Entry for alias digicertassuredidrootg2 successfully imported.
2024-07-24T13:21:07.951+02:00	Entry for alias d-trustrootclass3ca2ev2009 successfully imported.
2024-07-24T13:21:07.951+02:00	Entry for alias baltimorecybertrustroot successfully imported.
2024-07-24T13:21:07.952+02:00	Entry for alias comodocertificationauthority successfully imported.
2024-07-24T13:21:07.953+02:00	Entry for alias starfieldclass2certificationauthority successfully imported.
2024-07-24T13:21:07.953+02:00	Entry for alias usertrustecccertificationauthority successfully imported.
2024-07-24T13:21:07.953+02:00	Entry for alias quovadisrootca3 successfully imported.
2024-07-24T13:21:07.954+02:00	Entry for alias sectigopublicserverauthenticationroote46 successfully imported.
2024-07-24T13:21:07.955+02:00	Entry for alias quovadisrootca2 successfully imported.
2024-07-24T13:21:07.955+02:00	Entry for alias trustasiaglobalrootcag4 successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias trustasiaglobalrootcag3 successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias twcarootcertificationauthority successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias d-trustbrrootca12020 successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias commscopepublictrusteccroot-02 successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias commscopepublictrusteccroot-01 successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias certumtrustedrootca successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias ucaglobalg2root successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias ssl.comrootcertificationauthorityecc successfully imported.
2024-07-24T13:21:07.958+02:00	Entry for alias certainlyrootr1 successfully imported.
2024-07-24T13:21:07.958+02:00	Entry for alias identrustcommercialrootca1 successfully imported.
2024-07-24T13:21:07.958+02:00	Entry for alias izenpe.com successfully imported.
2024-07-24T13:21:07.959+02:00	Entry for alias ucaextendedvalidationroot successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias microsece-szignorootca2009 successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias acraizfnmt-rcmservidoresseguros successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias digicerttlseccp384rootg5 successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias certsignrootcag2 successfully imported.
2024-07-24T13:21:07.961+02:00	Entry for alias globalsignrootca successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias acraizfnmt-rcm successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias certainlyroote1 successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias affirmtrustpremiumecc successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias xrampglobalcertificationauthority successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias teliarootcav2 successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias netlockarany(classgold)ftanstvny successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias ssl.comrootcertificationauthorityrsa successfully imported.
2024-07-24T13:21:07.964+02:00	Entry for alias entrustrootcertificationauthority successfully imported.
2024-07-24T13:21:07.964+02:00	Entry for alias digicertassuredidrootca successfully imported.
2024-07-24T13:21:08.039+02:00	Entry for alias digicertglobalrootg3 successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias digicertglobalrootg2 successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias certumtrustednetworkca2 successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias oistewisekeyglobalrootgbca successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias comodorsacertificationauthority successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias haricatlseccrootca2021 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias ssl.comtlseccrootca2022 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias securitycommunicationeccrootca1 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias identrustpublicsectorrootca1 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias digicerthighassuranceevrootca successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias accvraiz1 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias godaddyrootcertificateauthority-g2 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias microsofteccrootcertificateauthority2017 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias t-telesecglobalrootclass3 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias t-telesecglobalrootclass2 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias globalsign successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias hongkongpostrootca3 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias ssl.comevrootcertificationauthorityrsar2 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias hellenicacademicandresearchinstitutionsrootca2015 successfully imported.
2024-07-24T13:21:08.056+02:00	Import command completed: 147 entries successfully imported, 0 entries failed or cancelled
2024-07-24T13:21:08.750+02:00	Warning: use -cacerts option to access cacerts keystore
2024-07-24T13:21:09.353+02:00	Certificate was added to keystore
2024-07-24T13:21:09.579+02:00	WARNING: ca-cert-aws-ca.pem does not contain exactly one certificate or CRL: skipping
@achimwinter achimwinter added the bug Something isn't working label Jul 24, 2024
@achimwinter achimwinter changed the title [Bug]: Importing Ca Certificates with USE_SYSTEM_CA_CERTS does not work [Bug]: Importing Ca certificates with USE_SYSTEM_CA_CERTS does not work Jul 24, 2024
@gdams
Copy link
Member

gdams commented Jul 24, 2024

This is likely due to the bump from alpine:3.19 to alpine:3.20. I'll do some digging and see if something needs changing

@gdams
Copy link
Member

gdams commented Jul 24, 2024

ping @rassie incase you have any ideas what might be going on here

@rassie
Copy link
Contributor

rassie commented Jul 24, 2024

From what I can see at the first glance, update-ca-certificates from alpine:3.19 also had this behaviour. Will need to dig a bit deeper.

@achimwinter can you give us the exact image versions you are using, i.e. working and breaking? Thanks.

@rassie
Copy link
Contributor

rassie commented Jul 24, 2024

Having dug around a bit, I think it's the changed algorithm around importing certificates. I'll try to figure out what to change and would add the fix to #572 or maybe a separate PR.

@achimwinter
Copy link
Author

achimwinter commented Jul 24, 2024

Sure 21.0.3_9-jre-alpine is definitely working for me. And i think i even tried it with 21-ubi9-minimal, but i would need to make sure of that. I also saw, that all these info statements at the beginning were also the like Entry for alias securitycommunicationeccrootca1 successfully imported. are only found in the broken images. Maybe that helps.

EDIT: I can confirm that the same error applies to e.g. 21.0.4_7-jre-ubi9-minimal which seems in line with the algorithm being the issue

@rassie
Copy link
Contributor

rassie commented Jul 24, 2024

Just for the record: the reason is that the previous algorithm generated system certificates with update-ca-certificates first and then extracted those to the Java trust store which worked with multiple certificates in a single file. Current algorithm doesn't play nice with multiple certificates per file. Have to figure out a workaround now.

@gdams
Copy link
Member

gdams commented Jul 24, 2024

Just for the record: the reason is that the previous algorithm generated system certificates with update-ca-certificates first and then extracted those to the Java trust store which worked with multiple certificates in a single file. Current algorithm doesn't play nice with multiple certificates per file. Have to figure out a workaround now.

Thanks for looking into this. Can you make sure that we add a test to prevent this being broken again in the future?

@rassie
Copy link
Contributor

rassie commented Jul 24, 2024

@gdams I've updated #572 to amend this problem. I've extended a test to check for multiple certificates per file.

@gdams
Copy link
Member

gdams commented Jul 24, 2024

@achimwinter @rassie has prepared a fix, we'll get his changes merged and roll out a new release.

@gdams gdams reopened this Jul 24, 2024
@gdams
Copy link
Member

gdams commented Jul 24, 2024

keeping this open until we've confirmed a fix. The images will be updated once docker-library/official-images#17249 is merged

@gdams
Copy link
Member

gdams commented Jul 25, 2024

@achimwinter can you please confirm if this is fixed now?

@achimwinter
Copy link
Author

Yes, it works again 👍 However one thing i should mention is that the certificate import needs now about 2-3 minutes on container start.
Previously it was like 5 seconds from this log Entry:
WARNING: ca-cert-aws-ca.pem does not contain exactly one certificate or CRL: skipping (Which didnt really skip the import)
to the first logs from the application

@quartje
Copy link

quartje commented Jul 26, 2024

I had the same problem. The fix works, but running __cacert_entrypoint.sh again results in a failure of the script with a exit code of 1.
These are the last lines:

Warning: Overwriting existing alias hellenicacademicandresearchinstitutionsrootca2015 in destination keystore
Entry for alias hellenicacademicandresearchinstitutionsrootca2015 successfully imported.
Import command completed:  148 entries successfully imported, 0 entries failed or cancelled
Warning: use -cacerts option to access cacerts keystore

I am using eclipse-temurin:17-jdk-alpine

@gdams
Copy link
Member

gdams commented Jul 30, 2024

21.0.3_9-jre-alpine

@rassie I'm also looking into this but do you have any idea why this might be so much slower? Could it be the csplit command?

@gdams gdams changed the title [Bug]: Importing Ca certificates with USE_SYSTEM_CA_CERTS does not work [Bug]: Importing Ca certificates with USE_SYSTEM_CA_CERTS is slower than it used to be Aug 13, 2024
quartje added a commit to OpenConext/OpenConext-deploy that referenced this issue Aug 19, 2024
This reverts the new cacert update script, which leads to errors when
restarting a container. See also: adoptium/containers#612
quartje added a commit to OpenConext/OpenConext-deploy that referenced this issue Aug 19, 2024
This reverts the new cacert update script, which leads to errors when
restarting a container. See also: adoptium/containers#612
quartje added a commit to OpenConext/OpenConext-deploy that referenced this issue Aug 19, 2024
This reverts the new cacert update script, which leads to errors when
restarting a container. See also: adoptium/containers#612
@MvEerd
Copy link

MvEerd commented Aug 30, 2024

I had the same problem. The fix works, but running __cacert_entrypoint.sh again results in a failure of the script with a exit code of 1.

I believe I am running into a similar issue. Since using the latest 'eclipse-temurin:21-jdk' image it works initially but restarting the container results in a lot of 'Entry for alias ... successfully imported.' and then never continues to boot our spring boot application using java -jar as specified in the command, the container exits instead.

I'm unable to find a changelog that describes what we have to change to keep our cartificate import working like it did in image a2bedb741020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants