Skip to content

Conversation

@gbhat618
Copy link
Contributor

@gbhat618 gbhat618 commented Mar 11, 2025

See JENKINS-75409


Self signed TLS certificate passing was not working with WebSocket agents.

  • either passing the -cert (also affected TCP agent)
    passing the -cert is actually a bit tricky need to pass as,
     # this will put the multi-line root-ca.pem into a single line text into CA_ROOT
     CA_ROOT="$(cat /path/to/root-ca.pem)"
     
     # need ensure double quote enclosing, otherwise parsing fails
     java -jar agent.jar -cert "$CA_ROOT" ...
    
  • or -noCertificateCheck (this worked in TCP agent)

Only passing the trustStore with the locally created CA was working -Djavax.net.ssl.trustStore=mytruststore.jks.

Testing done

Automated testing in jenkinsci/jenkins#10396

  • Failure simulated for these tests

    • tcpWithCertByValue: fails due to sslContext typo

      log snippet
      INFO: Could not locate server among [https://localhost:41597/jenkins/]; waiting 10 seconds before retry
      java.io.IOException: Failed to connect to https://localhost:41597/jenkins/tcpSlaveAgentListener/: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:230)
          at hudson.remoting.Engine.innerRun(Engine.java:958)
          at hudson.remoting.Engine.run(Engine.java:586)
      Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
      
    • webSocketNoCertificateCheck: fails due to pingSuccessful method falling to default java ssl, not the sslContext with certificate check disabled.

      log snippet
      WARNING: Certificate validation for HTTPs endpoints is disabled
      Mar 14, 2025 7:15:22 AM hudson.remoting.Engine startEngine
      INFO: Using Remoting version: 3291.vb_131b_dc231fa_
      Mar 14, 2025 7:15:22 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using /home/jenkins/agent/workspace/Core_jenkins_PR-10396/test/target/agent0-work15367093348594500537/remoting as a remoting work directory
      Mar 14, 2025 7:15:22 AM hudson.remoting.Launcher$CuiListener status
      INFO: https://localhost:38687/jenkins/login is not ready
      javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
      
    • webSocketWithCertByValue: fails due to sslContext typo and pingSuccessful method not using the sslContext object.

      log snippet
      Mar 14, 2025 7:36:16 AM hudson.remoting.Launcher createEngine
      INFO: Setting up agent: agent0
      Mar 14, 2025 7:36:16 AM hudson.remoting.Engine startEngine
      INFO: Using Remoting version: 3291.vb_131b_dc231fa_
      Mar 14, 2025 7:36:16 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using /home/jenkins/agent/workspace/Core_jenkins_PR-10396/test/target/agent0-work7308492121214090973/remoting as a remoting work directory
      Mar 14, 2025 7:36:16 AM hudson.remoting.Launcher$CuiListener status
      INFO: https://localhost:45923/jenkins/login is not ready
      javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
          at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
      
  • core IT is passing after this incremental remoting version is used in there. (link to build test report)

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

@gbhat618
Copy link
Contributor Author

logged in here
https://issues.jenkins.io/browse/JENKINS-75409

@gbhat618 gbhat618 closed this Mar 12, 2025
@gbhat618 gbhat618 reopened this Mar 13, 2025
@gbhat618 gbhat618 changed the title [DEBUG] TLS Certificate passing debug [JENKINS-75409] SSLContext typo fix and consider in pingSuccessful method Mar 13, 2025
@gbhat618 gbhat618 changed the title [JENKINS-75409] SSLContext typo fix and consider in pingSuccessful method [JENKINS-75409] SSLContext typo fix and consider it in pingSuccessful method Mar 13, 2025
@jglick
Copy link
Member

jglick commented Mar 13, 2025

passing the -cert is actually a bit tricky

Does it not accept -cert @/path/to/root-ca.pem? That seems like a bug or at least missing feature.

Copy link
Member

@jglick jglick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both changes look right.

@gbhat618
Copy link
Contributor Author

Does it not accept -cert @/path/to/root-ca.pem? That seems like a bug or at least missing feature

it is logged in here https://issues.jenkins.io/browse/JENKINS-75005

Also reproducing in test after IAR introduces cert option in jenkinsci/jenkins-test-harness#935

reproduce diff
diff --git a/src/test/java/org/jvnet/hudson/test/RealJenkinsRuleHttpsTest.java b/src/test/java/org/jvnet/hudson/test/RealJenkinsRuleHttpsTest.java
index 1fe49593..967167e0 100644
--- a/src/test/java/org/jvnet/hudson/test/RealJenkinsRuleHttpsTest.java
+++ b/src/test/java/org/jvnet/hudson/test/RealJenkinsRuleHttpsTest.java
@@ -64,6 +64,17 @@ public class RealJenkinsRuleHttpsTest {
         iar.createAgent(rr, options.build());
     }

+    @Test
+    public void inboundAgentWithCertByPath() throws Throwable {
+        var pathValue = "@" + getClass().getResource("/https/test-cert.pem").getPath();
+        var options = InboundAgentRule.Options
+                .newBuilder()
+                .name("remote")
+                .webSocket()
+                .cert(pathValue);
+        iar.createAgent(rr, options.build());
+    }
+
     private static void log(JenkinsRule r) throws IOException {
         LOGGER.info("Running on " + r.getURL().toExternalForm());
     }
(END)
failure log
[INFO] Running org.jvnet.hudson.test.RealJenkinsRuleHttpsTest
=== Starting inboundAgentWithCertByPath(org.jvnet.hudson.test.RealJenkinsRuleHttpsTest)
Will load plugins: RealJenkinsRuleInit.jpi
JENKINS_HOME=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/j h13115419354382663287 /Users/gbhat/.asdf/installs/java/temurin-17.0.14+7/bin/java -ea -Dhudson.Main.development=true -DRealJenkinsRule.classpath=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/RealJenkinsRule7152871084278840276/cp.txt -DRealJenkinsRule.location=file:/Users/gbhat/CBProjects/jenkins-test-harness/target/classes/ -DRealJenkinsRule.description=inboundAgentWithCertByPath(org.jvnet.hudson.test.RealJenkinsRuleHttpsTest) -DRealJenkinsRule.token=f596477b-7a23-40c9-8c78-ddae70aa697d -Dwinstone.portFileName=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/RealJenkinsRule7152871084278840276/jenkins-port.txt -Djava.io.tmpdir=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/ -jar /Users/gbhat/.m2/repository/org/jenkins-ci/main/jenkins-war/2.479/jenkins-war-2.479.war --enable-future-java --httpListenAddress=127.0.0.1 --prefix=/jenkins --httpPort=-1 --httpsPort=0 --httpsKeyStore=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/j h17593278634634351002/test-keystore.p12 --httpsKeyStorePassword=changeit --webroot=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/webroot12935166699664343631 --pluginroot=/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/pluginroot1982906886656888673
Running from: /Users/gbhat/.m2/repository/org/jenkins-ci/main/jenkins-war/2.479/jenkins-war-2.479.war
2025-03-14 03:08:42.108+0000 [id=1]	INFO	winstone.Logger#logInternal: Beginning extraction from war file
2025-03-14 03:08:43.026+0000 [id=1]	INFO	winstone.Logger#logInternal: Exclude Protocols [SSL, SSLv2, SSLv2Hello, SSLv3]
2025-03-14 03:08:43.026+0000 [id=1]	INFO	winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]
2025-03-14 03:08:43.033+0000 [id=1]	INFO	org.eclipse.jetty.server.Server#doStart: jetty-12.0.13; built: 2024-09-03T03:04:05.240Z; git: 816018a420329c1cacd4116799cda8c8c60a57cd; jvm 17.0.14+7
2025-03-14 03:08:43.213+0000 [id=1]	INFO	o.e.j.e.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /jenkins, did not find org.eclipse.jetty.ee9.jsp.JettyJspServlet
2025-03-14 03:08:43.232+0000 [id=1]	INFO	o.e.j.s.DefaultSessionIdManager#doStart: Session workerName=node0
2025-03-14 03:08:43.388+0000 [id=1]	INFO	hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/j h13115419354382663287 found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2025-03-14 03:08:44.046+0000 [id=1]	INFO	o.e.j.s.handler.ContextHandler#doStart: Started oeje9n.ContextHandler$CoreContextHandler@328902d5{Jenkins v2.479,/jenkins,b=file:///private/var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/webroot12935166699664343631/,a=AVAILABLE,h=oeje9n.ContextHandler$CoreContextHandler$CoreToNestedHandler@72e789cb{STARTED}}
2025-03-14 03:08:44.051+0000 [id=1]	INFO	o.e.j.util.ssl.SslContextFactory#load: x509=X509@29629fbb(1,h=[localhost, self-signed ca],a=[],w=[]) for Server@681adc8f[provider=null,keyStore=null,trustStore=null]
2025-03-14 03:08:44.191+0000 [id=1]	INFO	o.e.j.server.AbstractConnector#doStart: Started ServerConnector@7cb8437d{SSL, (ssl, http/1.1)}{0.0.0.0:65223}
2025-03-14 03:08:44.203+0000 [id=1]	INFO	org.eclipse.jetty.server.Server#doStart: Started oejs.Server@676cf48{STARTING}[12.0.13,sto=0] @2305ms
2025-03-14 03:08:44.205+0000 [id=30]	INFO	winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2025-03-14 03:08:44.255+0000 [id=35]	INFO	jenkins.InitReactorRunner$1#onAttained: Started initialization
2025-03-14 03:08:44.347+0000 [id=34]	INFO	jenkins.InitReactorRunner$1#onAttained: Listed all plugins
RealJenkinsRule ready
2025-03-14 03:08:45.118+0000 [id=48]	INFO	jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2025-03-14 03:08:45.120+0000 [id=48]	INFO	jenkins.InitReactorRunner$1#onAttained: Started all plugins
2025-03-14 03:08:45.120+0000 [id=34]	INFO	jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2025-03-14 03:08:45.176+0000 [id=50]	INFO	jenkins.InitReactorRunner$1#onAttained: System config loaded
2025-03-14 03:08:45.176+0000 [id=42]	INFO	jenkins.InitReactorRunner$1#onAttained: System config adapted
2025-03-14 03:08:45.177+0000 [id=55]	INFO	jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2025-03-14 03:08:45.177+0000 [id=55]	INFO	jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2025-03-14 03:08:45.183+0000 [id=35]	INFO	jenkins.model.Jenkins#setInstallState: Install state transitioning from: null to: TEST
2025-03-14 03:08:45.228+0000 [id=51]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
2025-03-14 03:08:45.255+0000 [id=26]	INFO	hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
Checking status
Jenkins is running at https://localhost:65223/jenkins/
Mar 14, 2025 8:38:45 AM org.jvnet.hudson.test.InboundAgentRule start
INFO: Running: [/Users/gbhat/.asdf/installs/java/temurin-17.0.14+7/bin/java, -Xmx512m, -XX:+PrintCommandLineFlags, -Djava.awt.headless=true, -jar, /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/agent8450341961110542793.jar, -url, https://localhost:65223/jenkins/, -secret, b79f3790b77631d343f8916d635b0e47db9a3f0c083e97bf041c2efe55cb967e, -name, remote, -webSocket, -cert, @/Users/gbhat/CBProjects/jenkins-test-harness/target/classes/https/test-cert.pem, -workDir, /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/remote-work12714929846590684492]
2025-03-14 03:08:45.718+0000 [id=83]	INFO	o.jvnet.hudson.test.JenkinsRule#waitOnline: Waiting for remote to come online…
-XX:ConcGCThreads=3 -XX:G1ConcRefinementThreads=10 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=301989888 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=536870912 -XX:MinHeapSize=6815736 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC
"-----END CERTIFICATE-----" is not a valid option
java -jar agent.jar [options...]
 -agentLog FILE                         : Local agent error log destination
                                          (overrides workDir)
 -auth user:pass                        : (deprecated) unused; use -credentials
                                          or -proxyCredentials
 -cert VAL                              : Specify additional X.509 encoded PEM
                                          certificates to trust when connecting
                                          to Jenkins root URLs. If starting
                                          with @ then the remainder is assumed
                                          to be the name of the certificate
                                          file to read. (default: -----BEGIN
                                          CERTIFICATE-----)
 -connectTo HOST:PORT                   : (deprecated) make a TCP connection to
                                          the given host and port, then start
                                          communication.
 -credentials (-jnlpCredentials)        : HTTP BASIC AUTH header to pass in for
 USER:PASSWORD                            making HTTP requests.
 -direct (-directConnection) HOST:PORT  : Connect directly to this TCP agent
                                          port, skipping the HTTP(S) connection
                                          parameter download. For example,
                                          "myjenkins:50000".
 -failIfWorkDirIsMissing                : Fails the initialization if the
                                          requested workDir or internalDir are
                                          missing ('false' by default)
                                          (default: false)
 -headless                              : (deprecated; now always headless)
 -help                                  : Show this help message (default:
                                          false)
 -instanceIdentity VAL                  : The base64 encoded InstanceIdentity
                                          byte array of the Jenkins controller.
                                          When this is set, the agent skips
                                          connecting to an HTTP(S) port for
                                          connection info.
 -internalDir VAL                       : Specifies a name of the internal
                                          files within a working directory
                                          ('remoting' by default) (default:
                                          remoting)
 -jar-cache DIR                         : Cache directory that stores jar files
                                          sent from the controller
 -jnlpUrl URL                           : instead of talking to the controller
                                          via stdin/stdout, emulate a JNLP
                                          client by making a TCP connection to
                                          the controller. Connection parameters
                                          are obtained by parsing the JNLP file.
 -loggingConfig FILE                    : Path to the property file with
                                          java.util.logging settings
 -name VAL                              : Name of the agent. (default: remote)
 -noCertificateCheck (-disableHttpsCert : Ignore SSL validation errors - use as
 Validation)                              a last resort only. (default: false)
 -noKeepAlive                           : Disable TCP socket keep alive on
                                          connection to the controller.
                                          (default: false)
 -noReconnect (-noreconnect)            : Doesn't try to reconnect when a
                                          communication fail, and exit instead
                                          (default: false)
 -noReconnectAfter DURATION             : Bail out after the given time after
                                          the first attempt to reconnect
 -ping                                  : (deprecated; now always pings)
 -protocols VAL                         : Specify the remoting protocols to
                                          attempt when instanceIdentity is
                                          provided.
 -proxyCredentials USER:PASSWORD        : HTTP BASIC AUTH header to pass in for
                                          making HTTP authenticated proxy
                                          requests.
 -secret HEX_SECRET                     : Agent connection secret. (default:
                                          b79f3790b77631d343f8916d635b0e47db9a3f
                                          0c083e97bf041c2efe55cb967e)
 -tcp FILE                              : (deprecated) instead of talking to
                                          the controller via stdin/stdout,
                                          listens to a random local port, write
                                          that port number to the given file,
                                          then wait for the controller to
                                          connect to that port.
 -text                                  : encode communication with the
                                          controller with base64. Useful for
                                          running agent over 8-bit unsafe
                                          protocol like telnet
 -tunnel HOST:PORT                      : Connect to the specified host and
                                          port, instead of connecting directly
                                          to Jenkins. Useful when connection to
                                          Jenkins needs to be tunneled. Can be
                                          also HOST: or :PORT, in which case
                                          the missing portion will be
                                          auto-configured like the default
                                          behavior.
 -url URL                               : Specify the Jenkins root URLs to
                                          connect to. (default: https://localhos
                                          t:65223/jenkins/)
 -version                               : Shows the version of the remoting jar
                                          and then exits (default: false)
 -webSocket                             : Make a WebSocket connection to
                                          Jenkins rather than using the TCP
                                          port. (default: true)
 -webSocketHeader NAME=VALUE            : Additional WebSocket header to set,
                                          eg for authenticating with reverse
                                          proxies. To specify multiple headers,
                                          call this flag multiple times, one
                                          with each header
 -workDir FILE                          : Declares the working directory of the
                                          remoting instance (stores cache and
                                          logs by default)



^C[INFO]
[INFO] Results:
[INFO]

I will check on that after the IAR PR above is merged.

@gbhat618 gbhat618 marked this pull request as ready for review March 14, 2025 12:42
@gbhat618
Copy link
Contributor Author

core IT is passing (link to build)

@gbhat618
Copy link
Contributor Author

(CI has failed in git checkout stderr: fatal: unable to access 'https://github.com/jenkinsci/remoting.git/': schannel: next InitializeSecurityContext failed: CRYPT_E_REVOCATION_OFFLINE (0x80092013) - The revocation function was unable to check revocation because the revocation server was offline., checking to re-trigger the build)

@gbhat618
Copy link
Contributor Author

gbhat618 commented Mar 17, 2025

(ig, I will just do git commit -m 'Trigger CI: empty commit' --allow-empty, there other non task related commits like branch update, so likely this PR needs to be Squash merged)

@gbhat618
Copy link
Contributor Author

(The same issue happened again, probably something to do with the network config of the windows controller) - I will re-trigger CI tomorrow.

Copy link
Member

@timja timja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@timja timja added the bug For changelog: Fixes a bug. label Mar 17, 2025
@timja timja merged commit 0d0d069 into jenkinsci:master Mar 17, 2025
14 checks passed
@jglick
Copy link
Member

jglick commented Mar 17, 2025

something to do with the network config of the windows controller

FTR no, the Windows agents need to

git config --add --global http.sslBackend openssl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug For changelog: Fixes a bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants