Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

Merged
merged 9 commits into from
Jan 20, 2025
+129 −9
Merged

Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning #3063

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a1b50bd
switch extensible Dockerfiles to wolfi-base image
kostasb Dec 27, 2024
30f59fd
remove packages unnecessary in runtime
kostasb Jan 7, 2025
53ae3e1
remove packages unnecessary in runtime from Dockerfile.ftest
kostasb Jan 7, 2025
2334402
weekly schedule for main and 8.x
kostasb Jan 7, 2025
157b8e7
merge extensible dockerfiles into main pipeline
kostasb Jan 10, 2025
fa7c0b0
Merge branch 'main' into kostasb/switch-dockerfile-image
kostasb Jan 15, 2025
c9a5e46
add ftest image tests to pipeline dependencies
kostasb Jan 15, 2025
bfd0ef8
use oss convention and remove ftest
kostasb Jan 16, 2025
c243aa4
Merge branch 'main' into kostasb/switch-dockerfile-image
kostasb Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 208 additions & 0 deletions .buildkite/extensible-dockerfiles-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
steps:
- group: ":truck: Building, Testing and Scanning extensible Dockerfile and Dockerfile.ftest"
key: "build_test_scan_group"
if: "(build.branch == \"main\")"
steps:
# ----
# Dockerfile build and tests on amd64
# ----
- label: "Building amd64 Docker image from extensible Dockerfile"
agents:
provider: aws
instanceType: m6i.xlarge
imagePrefix: ci-amazonlinux-2
env:
ARCHITECTURE: "amd64"
DOCKERFILE_PATH: "Dockerfile"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile"
command: ".buildkite/publish/build-docker.sh"
key: "build_extensible_dockerfile_image_amd64"
artifact_paths: ".artifacts/*.tar.gz"
- label: "Testing amd64 image built from extensible Dockerfile"
agents:
provider: aws
instanceType: m6i.xlarge
imagePrefix: ci-amazonlinux-2
env:
ARCHITECTURE: "amd64"
DOCKERFILE_PATH: "Dockerfile"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile"
depends_on: "build_extensible_dockerfile_image_amd64"
key: "test_extensible_dockerfile_image_amd64"
commands:
- "mkdir -p .artifacts"
- buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_image_amd64
- ".buildkite/publish/test-docker.sh"

# ----
# Dockerfile.ftest build and tests on amd64
# ----
- label: "Building amd64 Docker image from extensible Dockerfile.ftest"
agents:
provider: aws
instanceType: m6i.xlarge
imagePrefix: ci-amazonlinux-2
env:
ARCHITECTURE: "amd64"
DOCKERFILE_PATH: "Dockerfile.ftest"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile-ftest"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile-ftest"
command: ".buildkite/publish/build-docker.sh"
key: "build_extensible_dockerfile_ftest_image_amd64"
artifact_paths: ".artifacts/*.tar.gz"
- label: "Testing amd64 image built from Dockerfile.ftest"
agents:
provider: aws
instanceType: m6i.xlarge
imagePrefix: ci-amazonlinux-2
env:
ARCHITECTURE: "amd64"
DOCKERFILE_PATH: "Dockerfile.ftest"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile-ftest"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile-ftest"
depends_on: "build_extensible_dockerfile_ftest_image_amd64"
key: "test_extensible_dockerfile_ftest_image_amd64"
commands:
- "mkdir -p .artifacts"
- buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_ftest_image_amd64
- ".buildkite/publish/test-docker.sh"

# ----
# Dockerfile build and tests on arm64
# ----
- label: "Building arm64 Docker image from extensible Dockerfile"
agents:
provider: aws
instanceType: m6g.xlarge
imagePrefix: ci-amazonlinux-2-aarch64
diskSizeGb: 40
diskName: '/dev/xvda'
env:
ARCHITECTURE: "arm64"
DOCKERFILE_PATH: "Dockerfile"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile"
command: ".buildkite/publish/build-docker.sh"
key: "build_extensible_dockerfile_image_arm64"
artifact_paths: ".artifacts/*.tar.gz"
- label: "Testing arm64 image built from extensible Dockerfile"
agents:
provider: aws
instanceType: m6g.xlarge
imagePrefix: ci-amazonlinux-2-aarch64
diskSizeGb: 40
diskName: '/dev/xvda'
env:
ARCHITECTURE: "arm64"
DOCKERFILE_PATH: "Dockerfile"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile"
depends_on: "build_extensible_dockerfile_image_arm64"
key: "test_extensible_dockerfile_image_arm64"
commands:
- "mkdir -p .artifacts"
- buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_image_arm64
- ".buildkite/publish/test-docker.sh"

# ----
# Dockerfile.ftest build and tests on arm64
# ----
- label: "Building arm64 Docker image from extensible Dockerfile.ftest"
agents:
provider: aws
instanceType: m6g.xlarge
imagePrefix: ci-amazonlinux-2-aarch64
diskSizeGb: 40
diskName: '/dev/xvda'
env:
ARCHITECTURE: "arm64"
DOCKERFILE_PATH: "Dockerfile.ftest"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile-ftest"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile-ftest"
command: ".buildkite/publish/build-docker.sh"
key: "build_extensible_dockerfile_ftest_image_arm64"
artifact_paths: ".artifacts/*.tar.gz"
- label: "Testing arm64 image built from Dockerfile.ftest"
agents:
provider: aws
instanceType: m6g.xlarge
imagePrefix: ci-amazonlinux-2-aarch64
diskSizeGb: 40
diskName: '/dev/xvda'
env:
ARCHITECTURE: "arm64"
DOCKERFILE_PATH: "Dockerfile.ftest"
DOCKER_IMAGE_NAME: "docker.elastic.co/ci-agent-images/elastic-connectors-extensible-dockerfile-ftest"
DOCKER_ARTIFACT_KEY: "elastic-connectors-extensible-dockerfile-ftest"
depends_on: "build_extensible_dockerfile_ftest_image_arm64"
key: "test_extensible_dockerfile_ftest_image_arm64"
commands:
- "mkdir -p .artifacts"
- buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_ftest_image_arm64
- ".buildkite/publish/test-docker.sh"

# ----
# Vulnerability scanning on amd64 extensible Dockerfile and Dockerfile.ftest built images
# ----
- label: "Trivy Scan amd64 extensible Dockerfile Artifacts"
timeout_in_minutes: 10
depends_on:
- test_extensible_dockerfile_image_amd64
key: "trivy-scan-amd64-extensible-dockerfile-image"
agents:
provider: k8s
image: "docker.elastic.co/ci-agent-images/trivy:latest"
command: |-
mkdir -p .artifacts
buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_image_amd64
trivy --version
env | grep TRIVY
find .artifacts -type f -name '*.tar.gz*' -exec trivy image --quiet --input {} \;
- label: "Trivy Scan amd64 Dockerfile.ftest Artifacts"
timeout_in_minutes: 10
depends_on:
- test_extensible_dockerfile_ftest_image_amd64
key: "trivy-scan-amd64-extensible-dockerfile-ftest-image"
agents:
provider: k8s
image: "docker.elastic.co/ci-agent-images/trivy:latest"
command: |-
mkdir -p release
buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_ftest_image_amd64
trivy --version
env | grep TRIVY
find .artifacts -type f -name '*.tar.gz*' -exec trivy image --quiet --input {} \;

# ----
# Vulnerability scanning on arm64 extensible Dockerfile and Dockerfile.ftest built images
# ----
- label: "Trivy Scan arm64 extensible Dockerfile Artifacts"
timeout_in_minutes: 10
depends_on:
- test_extensible_dockerfile_image_arm64
key: "trivy-scan-arm64-extensible-dockerfile-image"
agents:
provider: k8s
image: "docker.elastic.co/ci-agent-images/trivy:latest"
command: |-
mkdir -p .artifacts
buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_image_arm64
trivy --version
env | grep TRIVY
find .artifacts -type f -name '*.tar.gz*' -exec trivy image --quiet --input {} \;
- label: "Trivy Scan arm64 Dockerfile.ftest Artifacts"
timeout_in_minutes: 10
depends_on:
- test_extensible_dockerfile_ftest_image_arm64
key: "trivy-scan-arm64-extensible-dockerfile-ftest-image"
agents:
provider: k8s
image: "docker.elastic.co/ci-agent-images/trivy:latest"
command: |-
mkdir -p release
buildkite-agent artifact download '.artifacts/*.tar.gz*' .artifacts/ --step build_extensible_dockerfile_ftest_image_arm64
trivy --version
env | grep TRIVY
find .artifacts -type f -name '*.tar.gz*' -exec trivy image --quiet --input {} \;
13 changes: 10 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
FROM python:3.11-slim-bookworm
RUN apt -y update && apt -y upgrade && apt -y install make git
COPY . /app
FROM cgr.dev/chainguard/wolfi-base
ARG python_version=3.11

USER root
RUN apk add --no-cache python3=~${python_version} make git

COPY --chown=nonroot:nonroot . /app

USER nonroot
WORKDIR /app
RUN make clean install
RUN ln -s .venv/bin /app/bin

kostasb marked this conversation as resolved.
Show resolved Hide resolved
ENTRYPOINT []
13 changes: 9 additions & 4 deletions Dockerfile.ftest
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
FROM python:3.11-slim-bookworm
# RUN apt update && apt install make
RUN apt -y update && apt -y upgrade && apt -y install make git
COPY . /app
FROM cgr.dev/chainguard/wolfi-base
ARG python_version=3.11

USER root
RUN apk add --no-cache python3=~${python_version} make git

COPY --chown=nonroot:nonroot . /app

USER nonroot
WORKDIR /app
RUN make clean install
RUN .venv/bin/pip install -r requirements/ftest.txt
33 changes: 33 additions & 0 deletions catalog-info.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,39 @@ spec:
search-extract-and-transform: {}
search-productivity-team: {}

# Nightly build and scan of the connectors extensible Dockerfiles
---
apiVersion: "backstage.io/v1alpha1"
kind: "Resource"
metadata:
name: "connectors-extensible-dockerfiles"
description: "Nightly build and scan of the connectors extensible Dockerfiles"
spec:
type: "buildkite-pipeline"
owner: "group:search-extract-and-transform"
system: "buildkite"
implementation:
apiVersion: "buildkite.elastic.dev/v1"
kind: "Pipeline"
metadata:
name: "connectors-extensible-dockerfiles"
description: "Nightly build and scan of the connectors extensible Dockerfiles"
spec:
pipeline_file: ".buildkite/extensible-dockerfiles-pipeline.yml"
provider_settings:
trigger_mode: "none"
repository: "elastic/connectors"
schedules:
Daily main:
branch: main
cronline: '@daily'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if we should consider running this less frequently? I think it would depend on what @elastic/search-extract-and-transform prefer, but the scenario I picture is something like:

  • this job runs once a week, or maybe even once every two weeks
  • if Trivy identifies any vulnerabilities, a message is sent to a Slack channel, or a GitHub issue is created

let's see what folks think 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • if Trivy identifies any vulnerabilities, a message is sent to a Slack channel, or a GitHub issue is created

I think these notifications could potentially be a duplicate of existing Snyk issues detected on the official container image. What's the value added by these new notifications?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is Snyk going to scan the images produced by these CI jobs?
I think this PR is doing 3 things, but maybe doesn't need to.

  1. switches our Dockerfile to use chainguard (has customer value)
  2. Adds CI to validate the Dockerfile works (has customer value)
  3. Adds CI to fail if a vuln is found (internal value, with indirect customer value)

I think (3) could probably be done separately, and reuse whatever machinery we use to scan other artifacts, especially since (1) and (2) will significantly cut down on false-positives that we'd have if we tried to do (3) on its own today.

Copy link
Contributor Author

@kostasb kostasb Jan 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI jobs in this PR do not push images to the docker registry, instead it relies on Trivy for vulnerability scanning, which runs within the context of the pipeline.

The alternative approach is to push the images to an internal namespace on the docker registry and request for these to be added to Snyk. This would result in duplicate reports though as we're already scanning the docker.elastic.co/integrations/elastic-connectors with Snyk built from the Dockerfile.wolfi image which is technically the exact same base OS build.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the value added by these new notifications?

...

which is technically the exact same base OS build.

These are really good points that I hadn't considered. Given that with this PR we are:

  • switching the Dockerfile to become distroless, and making the resultant image considerably more secure
  • making the Dockerfile essentially the same as the Dockerfile.wofli, the latter of which produces the images that are pushed and scanned by Snyk
  • adding nothing in the Dockerfile except the bare minimum that we need to run connectors (assuming we get rid of git and make 👍)

... then I actually don't see much value in adding the Trivy step (let alone adding notifications based on its results). I wish I'd had these realizations sooner, apologies @kostasb.

What do you all think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kostasb thanks, I like the idea of leaving the Trivy step there, but not having it send any notifications. But I'll defer to E&T folks (@seanstory @artem-shelkovnikov) to say what they prefer 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no objection to leaving it (without notifications) but also I don't think a human will be manually looking at the outputs often. So if we're ok as treating it essentially as a source of debug logs that may or may not be examined, great.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the following might be a big direction change for this PR, but considering the feedback above, I think it might be worthy to explore: what if, instead of adding a new pipeline (with informative value only), we add the Trivy scan step to the existing pipeline? I think the informative value would be higher for developers working in PRs: the Trivy scan would be part of their feedback loop, and not something relegated to a different pipeline.

Similarly to what we did for ent-search, I would run the Trivy scan step in PRs to main, and on merge on main and other current release branches. What do you all think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging the extensible Dockerfile build/test/scan steps into the main pipeline makes sense to me, we can proceed this way if the team agrees.

Copy link
Contributor Author

@kostasb kostasb Jan 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a commit that merges the extensible Dockerfiles pipeline into the main pipeline (easy to revert if the team @seanstory @artem-shelkovnikov has a different opinion after all).

As a bonus point, this also enables testing as part of this PR (the new dedicated pipeline wouldn't be picked up until after merged).

message: "Runs daily `main` extensible Dockerfiles image builds"
teams:
everyone:
access_level: "READ_ONLY"
search-extract-and-transform: {}
search-productivity-team: {}

########
# Docker image build and publish - manual release
########
Expand Down