Add Snyk scanning for Logstash release artifacts #18557
Conversation
Logstash artifacts are all based on a logstash tarball. This tarball contains all of the gems/jars that are installed as logstash dependencies. Gradle struggles to find all these gems/jars (shadowed jars and default jruby gems for example). This adds a "shim" layer which produces and SBOM that can be directly handed to snyk. In addition to an SBOM there are some CSV files that show where every artifact was found on disk and does some duplicat analysis. This PR Implements automated vulnerability scanning of published Logstash tarballs. It extracts JARs and gems from artifacts, generates an SBOM, and reports to Snyk via `snyk monitor`. Daily Buildkite pipeline scans all releases and snapshots from logstash-versions.yml. Local extraction available via `./gradlew extractArtifactVersions` which follows the pattern of having all CI tasks go through a gradle task.
🤖 GitHub commentsJust comment with:
|
|
This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
|
| end | ||
|
|
||
| # Add evidence of location | ||
| component[:properties] = [ |
There was a problem hiding this comment.
Note: these are not required for snyk, but i figured we could leave it in the json for analysis if ever there is a question wehre version was extracted from.
| VERSIONS_URL = "https://raw.githubusercontent.com/logstash-plugins/.ci/1.x/logstash-versions.yml" | ||
|
|
||
|
|
||
| def fetch_logstash_versions() -> dict: |
There was a problem hiding this comment.
Build the matrix based on https://github.com/logstash-plugins/.ci/blob/1.x/logstash-versions.yml This will ensure we are in sync with the unified release jobs around release time etc.
|
|
||
| echo "--- Running Snyk monitor for Logstash {version}" | ||
| # NOTE: we may need to check if our enterprise account allows SBOM yet... There may be some other options if not | ||
| ./snyk monitor --experimental --file=output_sbom.json --org=logstash --target-reference={version} --project-name="logstash-artifact-{version}" |
There was a problem hiding this comment.
It may be an enterprise feature to use sbom as input. Need to double check that is feasible. There may be alternative ways.
The main purpose of using sbom instead of pom.xml or Gemfile.lock is that there are duplicate versions. These are difficult to express in those formats.
| "key": step_key, | ||
| "command": command, | ||
| "artifact_paths": [ | ||
| ".buildkite/scripts/snyk/artifact-scan/output*.csv", |
There was a problem hiding this comment.
The idea of these CSV files is that CI will be generating more human readable versions. If you want to look in to version paths, duplicates, etc you can download a csv from buildkite and search through it in a spreadsheet tool of your choice.
| puts "Installing dependencies..." | ||
| Dir.chdir(script_dir) do | ||
| require 'bundler' | ||
| Bundler.with_unbundled_env do |
There was a problem hiding this comment.
I'm still exploring patterns for the use case where we have a top level gradle task which calls a rake task that has dependencies managed with bundler OUTSIDE that of which we manage with logstash artifacts. I have a similar use case in #18536 where i'm shelling out in gradle.
💛 Build succeeded, but was flaky
Failed CI Steps |
2 participants
Release notes
[rn:skip]
What does this PR do?
Logstash artifacts are all based on a logstash tarball. This tarball contains all of the gems/jars that are installed as logstash dependencies. Gradle struggles to find all these gems/jars (shadowed jars and default jruby gems for example). This adds a "shim" layer which produces and SBOM that can be directly handed to snyk. In addition to an SBOM there are some CSV files that show where every artifact was found on disk and does some duplicat analysis.
This PR Implements automated vulnerability scanning of published Logstash tarballs. It extracts JARs and gems from artifacts, generates an SBOM, and reports to Snyk via
snyk monitor.Daily Buildkite pipeline scans all releases and snapshots from logstash-versions.yml. Local extraction available via
./gradlew extractArtifactVersionswhich follows the pattern of having all CI tasks go through a gradle task.Why is it important/What is the impact to the user?
N/A
How to test this PR locally