Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

kostasb · 2024-12-27T17:07:48Z

https://github.com/elastic/search-developer-productivity/issues/3547

Description

The goal of this PR is to:

move "extensible Dockerfiles" to a distroless base image. Dockerfile and Dockerfile.ftest are not used in Elastic's docker image releases but are rather provided as a stepping stone for users to build custom images on.
add a pipeline for vulnerability scanning of built images

This is based on @acrewdson 's draft. It uses wolfi-base images and adds buildkite pipelines to build, test and scan the resulting docker images using trivy.

A notification method for vulnerability reports from Trivy is TBD.

Checklists

Pre-Review Checklist

Changes Requiring Extra Attention

Security-related changes (encryption, TLS, SSRF, etc)
New external service dependencies added.

Release Note

The extensible Dockerfile provided in the connectors repo has been updated to use a different base image for improved security. Users who build custom docker images based on this Dockerfile may need to review their configuration for compatibility with the new base image.

acrewdson

great start! I added a few comments

acrewdson · 2024-12-27T18:01:29Z

Dockerfile

-RUN make clean install
-RUN ln -s .venv/bin /app/bin
+RUN make clean install && ln -s .venv/bin /app/bin
+


I had this line

RUN apk del make git

so that we could remove now-unnecessary packages from the final images, with the assumption that we do not need make or git at runtime, but we could confirm with someone from @elastic/search-extract-and-transform that it makes sense

@acrewdson My thought process for omitting that line was:

since we're going to scan the built image we'd want to be able to detect vulnerabilities on these packages, even if they're not required at runtime.

they've been included in the Dockerfile so removing them now can cause a "breaking" change for users who already build on top of this. Changing the base image itself can be a much more substantial breaking change though so perhaps that shouldn't be factored in here.

Let me know what you think?

I'm ok with removing make and git. I expect very little being extended on the image, and more likely that the connectors code is what is seeing extension. We shouldn't need make or git for that.

+1. git is needed at build-time, but for run-time we need neither git nor make

catalog-info.yaml

acrewdson · 2024-12-27T18:22:05Z

.buildkite/dockerfiles-pipeline.yml

+      # ----
+      # Dockerfile build and tests on amd64
+      # ----
+      - label: "Build amd64 image from Dockerfile"


instead of only building amd64 images, we could consider building a multi-arch image using buildah or drivah, like we are doing here, and that way we could also potentially avoid using these hefty ec2 instances

Thank you for the suggestion about a multi-arch image, reading in more detail how buildah/drivah multiarch push works, it pulls the already built amd64/arm64 images from the previous steps and combines them into a common manifest which references both builds. Unfortunately doesn't save us any build resources.

It means that we'd build the image on both architectures to create a multiarch image that isn't going to be published or used for any other purpose. I do not see the need for this since the images aren't published, what do you think?

I added pipeline steps to build and test both amd64 and arm64 architectures. Given the above analysis I don't see a use for a multiarch image step since we aren't publishing that artifact either way, let me know if you have any concerns.

acrewdson · 2024-12-27T18:27:22Z

catalog-info.yaml

+      schedules:
+        Daily main:
+          branch: main
+          cronline: '@daily'


wondering if we should consider running this less frequently? I think it would depend on what @elastic/search-extract-and-transform prefer, but the scenario I picture is something like:

this job runs once a week, or maybe even once every two weeks

if Trivy identifies any vulnerabilities, a message is sent to a Slack channel, or a GitHub issue is created

let's see what folks think 👍

if Trivy identifies any vulnerabilities, a message is sent to a Slack channel, or a GitHub issue is created

I think these notifications could potentially be a duplicate of existing Snyk issues detected on the official container image. What's the value added by these new notifications?

is Snyk going to scan the images produced by these CI jobs?
I think this PR is doing 3 things, but maybe doesn't need to.

switches our Dockerfile to use chainguard (has customer value)

Adds CI to validate the Dockerfile works (has customer value)

Adds CI to fail if a vuln is found (internal value, with indirect customer value)

I think (3) could probably be done separately, and reuse whatever machinery we use to scan other artifacts, especially since (1) and (2) will significantly cut down on false-positives that we'd have if we tried to do (3) on its own today.

The CI jobs in this PR do not push images to the docker registry, instead it relies on Trivy for vulnerability scanning, which runs within the context of the pipeline.

The alternative approach is to push the images to an internal namespace on the docker registry and request for these to be added to Snyk. This would result in duplicate reports though as we're already scanning the docker.elastic.co/integrations/elastic-connectors with Snyk built from the Dockerfile.wolfi image which is technically the exact same base OS build.

acrewdson · 2024-12-27T18:30:02Z

.buildkite/dockerfiles-pipeline.yml

+        command: |-
+          mkdir -p .artifacts
+          buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_dockerfile_image_amd64
+          trivy --version
+          env | grep TRIVY
+          find .artifacts -type f -name '*.tar.gz*' -exec trivy image --quiet --input {} \;


nice 👍

I'm thinking it could also be nice to add one additional stage to this pipeline, if it's not too difficult:

a smoke-test/acceptance-test that verifies that connectors actually starts up in the resultant image, just as a way of gaining confidence that the built image does what we expect it to do

Good point, this pipeline includes the same test step that is used for the production docker images built from Dockerfile.wolfi here so that the testing approach is aligned on both built images.

The pipeline step test_extensible_dockerfile_image_amd64 runs the .buildkite/publish/test-docker.sh which performs basic sanity checks and verifies the output of elastic-ingest --version here.

Since adding functional tests at the connector level is a more involved task which could benefit both the production and the "extensible" image I'd suggest we create a dedicated issue for that, what do you think?

.buildkite/dockerfiles-pipeline.yml

Dockerfile

kostasb · 2024-12-30T17:08:26Z

Since in this PR we are building a Dockerfile on a different base image (cgr.dev/chainguard/wolfi-base) there are package discrepancies comparing to the base image used in the official built images wolfi/python:3.11-dev.

For reference, here is a list of packages which, among others (like bash and curl), are available in the wolfi python-3.11 image used for the production image builds.

Do we want this list of packages added to the wolfi-base image? Are connectors requiring some, or all, of them? @elastic/search-extract-and-transform

artem-shelkovnikov · 2024-12-31T14:13:19Z

I'm not sure if we need the packages mentioned, we really only need dependencies for 3.11 python + base64 available.

kostasb · 2024-12-31T15:12:56Z

I'm not sure if we need the packages mentioned, we really only need dependencies for 3.11 python + base64 available.

Thanks, then I'll stick with the bare minimum packages and we can iterate if needed.

I added both amd64 and arm64 image builds, mainly for the purpose of smoke testing (with .buildkite/publish/test-docker.sh) on both architectures. I don't expect deviations in the vulnerability testing between architectures.

Do we need some sort of release note for the new base image? It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

kostasb · 2025-01-02T14:06:12Z

Update re. the packages included with the base image vs the python one that we do docker image builds on: @oli-g suggested to check whether installing python-3.11 on the wolfi-base image installes those as dependencies. It does install most of them as part of the Dockerfile step that adds python (RUN apk add --no-cache python3=~${python_version} make git), with some exceptions (linux-headers) which are unlikely to be relevant for connectors. That said, there shouldn'be any concerns regarding the base image diff.

artem-shelkovnikov · 2025-01-02T16:12:38Z

It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

Just to check - cgr.dev/chainguard/wolfi-base is publicly available, right?

kostasb · 2025-01-02T16:32:34Z

It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

Just to check - cgr.dev/chainguard/wolfi-base is publicly available, right?

Yes, we've confirmed that this is a publicly available image that can be downloaded by unauthenticated (not logged in to any registry) docker instances.

artem-shelkovnikov · 2025-01-02T16:37:15Z

Do we need some sort of release note for the new base image? It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

We can mention it in our release notes indeed. I don't expect a lot of customers to build customer docker images and likely it's just gonna work for them, but it would not hurt to mention that we've updated our docker images recently to make them more secure with potential incompatible changes that the customers will have to fix

kostasb · 2025-01-03T08:50:02Z

Do we need some sort of release note for the new base image? It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

We can mention it in our release notes indeed. I don't expect a lot of customers to build customer docker images and likely it's just gonna work for them, but it would not hurt to mention that we've updated our docker images recently to make them more secure with potential incompatible changes that the customers will have to fix

Thank you, I added a release note in the issue description.

kostasb requested a review from a team December 27, 2024 17:07

github-actions bot added auto-backport v9.0.0 labels Dec 27, 2024

acrewdson reviewed Dec 27, 2024

View reviewed changes

kostasb force-pushed the kostasb/switch-dockerfile-image branch from 84d8123 to cdfd17b Compare December 31, 2024 15:03

switch extensible Dockerfiles to wolfi-base image

a1b50bd

kostasb force-pushed the kostasb/switch-dockerfile-image branch from 25f6298 to a1b50bd Compare December 31, 2024 17:33

kostasb marked this pull request as ready for review December 31, 2024 17:34

kostasb requested a review from a team as a code owner December 31, 2024 17:34

kostasb added the release_note label Jan 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

kostasb commented Dec 27, 2024 •

edited

Loading

acrewdson left a comment

acrewdson Dec 27, 2024

kostasb Dec 30, 2024 •

edited

Loading

seanstory Jan 2, 2025

artem-shelkovnikov Jan 2, 2025

acrewdson Dec 27, 2024

kostasb Dec 30, 2024 •

edited

Loading

kostasb Jan 2, 2025

acrewdson Dec 27, 2024

oli-g Dec 30, 2024

seanstory Jan 2, 2025

kostasb Jan 2, 2025 •

edited

Loading

acrewdson Dec 27, 2024

kostasb Dec 31, 2024

kostasb commented Dec 30, 2024 •

edited

Loading

artem-shelkovnikov commented Dec 31, 2024

kostasb commented Dec 31, 2024 •

edited

Loading

kostasb commented Jan 2, 2025

artem-shelkovnikov commented Jan 2, 2025

kostasb commented Jan 2, 2025

artem-shelkovnikov commented Jan 2, 2025

kostasb commented Jan 3, 2025

Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

Are you sure you want to change the base?

Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

Conversation

kostasb commented Dec 27, 2024 • edited Loading

Description

Checklists

Pre-Review Checklist

Changes Requiring Extra Attention

Release Note

acrewdson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasb Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasb Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasb Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasb commented Dec 30, 2024 • edited Loading

artem-shelkovnikov commented Dec 31, 2024

kostasb commented Dec 31, 2024 • edited Loading

kostasb commented Jan 2, 2025

artem-shelkovnikov commented Jan 2, 2025

kostasb commented Jan 2, 2025

artem-shelkovnikov commented Jan 2, 2025

kostasb commented Jan 3, 2025

kostasb commented Dec 27, 2024 •

edited

Loading

kostasb Dec 30, 2024 •

edited

Loading

kostasb Dec 30, 2024 •

edited

Loading

kostasb Jan 2, 2025 •

edited

Loading

kostasb commented Dec 30, 2024 •

edited

Loading

kostasb commented Dec 31, 2024 •

edited

Loading