Enable CD

[basil]

Now that #116 has landed, the next step is to enable JEP-229 CD for this repository, using GitHub Actions (as is standard for CD in the Jenkins project). The CD work should largely follow https://www.jenkins.io/doc/developer/publishing/releasing-cd/ but may be slightly more involved in that it needs to run on a Windows 2019 system in GitHub Actions rather than a Linux system in GitHub Actions (as the instructions currently describe).

The standard process used in Jenkins is to run CD after CI has passed, skipping the tests in CD. Since this CD build is more complicated, and since the test suite runs in milliseconds, it may be desirable to also run the tests again during CD.

One complication is that CD needs to sign the binaries after building them but before putting them in the JAR. The signing should take place as described here with some commands like:

WinSDK/bin/signtool sign /f path-to-certificate.pfx /p PASSWORD /t http://timestamp.digicert.com native\x64\Release\winp.dll
WinSDK/bin/signtool sign /f path-to-certificate.pfx /p PASSWORD /t http://timestamp.digicert.com native\x64\Release\winp.x64.dll

It may be necessary to engage with the Jenkins infrastructure team for access to a certificate. The password for the certificate can be stored by a GitHub administrator as a secret in the "Actions secrets and variables" configuration for this repository, where it may be consumed by the CD (but not CI!) build.

[basil]

CC @pbo-linaro @slide

[basil]

References:

• https://github.com/jenkins-infra/github-reusable-workflows/blob/f771babb1fd571cb03e36d875eb61184c796aa4b/.github/workflows/maven-cd.yml
• https://github.com/jenkins-infra/jenkins-maven-cd-action
• https://federicoterzi.com/blog/automatic-codesigning-on-windows-using-github-actions/
• https://github.com/skymatic/code-sign-action

[pbo-linaro]

Just to ensure we are on the same page, will the workflow be:

• Run CI on GitHub Actions
• If pass, run CD
  ?

[basil]

Not quite—https://github.com/jenkins-infra/github-reusable-workflows/blob/f771babb1fd571cb03e36d875eb61184c796aa4b/.github/workflows/maven-cd.yml#L19-L24 first verifies that the Jenkins CI build was passing, using https://github.com/jenkins-infra/verify-ci-status-action, and this is why we first asked you to get the Jenkins build up and running. Then, if various other conditions are met, like the pull request being something other than a routine dependency update (tested via https://github.com/jenkins-infra/github-reusable-workflows/blob/f771babb1fd571cb03e36d875eb61184c796aa4b/.github/workflows/maven-cd.yml#L35-L41) we proceed to do the CD release.

All of the above should work out of the box for you, but where we run into differences for this component is in the CD release itself. https://github.com/jenkins-infra/github-reusable-workflows/blob/f771babb1fd571cb03e36d875eb61184c796aa4b/.github/workflows/maven-cd.yml#L43 certainly won't work here, as this needs to run on Windows 2019 rather than ubuntu-latest. https://github.com/jenkins-infra/github-reusable-workflows/blob/f771babb1fd571cb03e36d875eb61184c796aa4b/.github/workflows/maven-cd.yml#L56-L61 will then invoke Maven via https://github.com/jenkins-infra/jenkins-maven-cd-action/blob/5f5529707ac2bef1ff86da2553ce465ed669aa65/run.sh#L8, and that currently won't sign the binaries before deploying them, so some work is needed there as well. (It also uses -Pquick-build to skip the tests, which we may or may not want to do in this context.)

Ideally we would find a way to improve or enhance https://github.com/jenkins-infra/jenkins-maven-cd-action to accomodate this workflow rather than copying and pasting the code into this repository.

[basil]

One thing I just thought of that might help you is that when deploying the CD release with Maven https://github.com/jenkins-infra/jenkins-maven-cd-action/blob/5f5529707ac2bef1ff86da2553ce465ed669aa65/run.sh#L8 you will need to have the certificate and password defined as environment variables (stored as secrets in GitHub), so you can key off the presence of those environment variables in the Maven build to do the signing. Something like e.g.

<profiles>
  <profile>
    <id>sign-native</id>
    <activation>
      <os>
        <family>Windows</family>
      </os>
      <property>
        <name>env.CERTIFICATE</name>
      </property>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>exec-maven-plugin</artifactId>
          <executions>
            <execution>
              <id>sign-native</id>
              <goals>
                <goal>exec</goal>
              </goals>
              <phase>process-resources</phase>
              <configuration>
                <executable>sign.cmd</executable>
              </configuration>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

although note that I haven't tried to run this code.

[pbo-linaro]

To be honest, it seems very complicated and will involve again new discussions with other teams. On the other way, you seem to know exactly what it needs and seems like a few hours of work. So my polite request, could you help on this and do this change?

[basil]

I can't commit to doing this task. Parameterizing the CD workflow to allow for a non-Ubuntu VM seems like the easiest task, so I would start there. I have no experience with Windows key signing and didn't sign the currently shipping binaries, so I have no idea what that actually looks like beyond the hints Oleg left in this repository's native/signing.txt file. I'm also not a member of the infrastructure team and don't manage certificates for the Jenkins project. But once we have the key and password, I can add it as a GitHub secret and we can easily consume it from Maven as explained above. Plumbing that into the CD workflow is then the last step, which is a problem we can solve once we arrive there (at which point hopefully the solution will be clearer).

[pbo-linaro]

I'll take a look onto that thus.

[pbo-linaro]

After taking a look at this on my side, I opened a jenkins-infra issue (jenkins-infra/helpdesk#4076) to see if we can have an official action supporting this on Windows.

[basil]

The expectation is that you would write or modify any GitHub Actions based CD workflows, reaching out to the infrastructure team only if a certificate is needed.

it's a bit hard for me to figure what is the correct step and expected work

Then I will give you directions, step-by-step. At a high-level, the first task will be for you to get the entire CD workflow working using copy and pasted code in this repository, and the second task will be to figure out how to generalize this back into the reusable workflows. The rest of this comment will focus on the first high-level task, since the second is too far off to envision clearly.

Start by reverse engineering the signing process and coming up with a design for how this should be signed. Research the common ways of code signing Windows binaries, and list them. Then reverse engineer how the existing shipping binaries were signed. Determine if we should sign these binaries the same way or a different way. Report your conclusions, and then when consensus has been reached, we can reach out to the infrastructure team for help with a certificate (if a certificate is needed).

Codewise, create a pull request in jenkins-infra/repository-permissions-updater as described in https://www.jenkins.io/doc/developer/publishing/releasing-cd/#enable-cd-permissions. Then start working on a pull request for jenkinsci/winp:

• Follow https://www.jenkins.io/doc/developer/publishing/releasing-cd/#configure-cd-workflow, then inline the maven-cd workflow rather than consuming it from github-reusable-workflows, since we'll need to make changes to it. Once inlined, make the Linux -> Windows change I suggested.
• Follow https://www.jenkins.io/doc/developer/publishing/releasing-cd/#configure-release-drafter
• Follow https://www.jenkins.io/doc/developer/publishing/releasing-cd/#configure-dependabot
• Follow https://www.jenkins.io/doc/developer/publishing/releasing-cd/#update-maven-pom-and-config
• Update the Maven build to sign the binary as in Enable CD #117 (comment)

[basil]

Especially, as an external contributor, it's hard to justify spending time modify jenkins internal actions scripts associated to this, as our original work, was simply to support a new architecture (windows-arm64).

Allow me to make it easier to justify, then. The justification is as follows: the existing shipping binaries were signed manually by a maintainer who is not currently active. To ship changes to these binaries without causing a regression, we need to ensure that the new release is at feature parity with the old one (i.e., if the old release was signed, the new release must be signed as well). And to sign in a way that is sustainable (i.e., one that doesn't cause the same maintenance challenges in the future when a maintainer moves on), the signing process should be automated.