Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cpbr 1867 ubi9 minimal #618

Open
wants to merge 51 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
70e40c0
add trimmed down dockerfile
hk10111 Oct 30, 2024
2541a91
add docker refresh module
hk10111 Nov 6, 2024
409bb4c
copy binaries from cp-base-new
hk10111 Nov 6, 2024
169acae
create dockerfile with jre and ubi-micro
hk10111 Nov 6, 2024
f3c7fbf
add mvn build
hk10111 Nov 6, 2024
529ac03
minor fix to env var
hk10111 Nov 6, 2024
b104644
add remaining files
hk10111 Nov 6, 2024
d1014b9
add tests
hk10111 Nov 6, 2024
241633b
add tests for ub
hk10111 Nov 12, 2024
da30d3f
add ub tests
hk10111 Nov 12, 2024
ea65844
add cp-base-refresh to deploy pipeline
hk10111 Nov 12, 2024
42421f4
remove tag
hk10111 Nov 13, 2024
5b32376
remove tag
hk10111 Nov 13, 2024
bbaff1b
use cp-base-lite
hk10111 Nov 13, 2024
664c82c
add dedupe scripts
hk10111 Nov 20, 2024
c509524
add dedupe scripts and jars
hk10111 Nov 20, 2024
ce04ac7
minor fixes to user group
hk10111 Nov 20, 2024
cabebb2
replace ubi8-micro with ubi8-minimal
hk10111 Dec 20, 2024
a71423d
rename image to cp-base-jre17
hk10111 Jan 15, 2025
2f4faba
rename image to cp-base-jre17
hk10111 Jan 15, 2025
b16ec43
rename image to cp-base-jre17
hk10111 Jan 15, 2025
8101e30
add env var for ubi9 version
hk10111 Jan 15, 2025
9fecdd0
use env var for ubi9 minimal version
hk10111 Jan 15, 2025
c460fde
resolve merge conflicts
hk10111 Jan 15, 2025
df94173
use master branch
hk10111 Jan 15, 2025
c9204a1
change docker repo name in pipeline
hk10111 Jan 20, 2025
7cab88d
skip tests
hk10111 Jan 21, 2025
3a16eb2
correct image name
hk10111 Jan 21, 2025
1a10982
removed docker file from base
hk10111 Jan 22, 2025
e4e9987
base files for docker image
hk10111 Jan 27, 2025
db004c4
dockefile for cp-base-java and pom
hk10111 Jan 27, 2025
2bc7e3b
remove cp-base-jre17
hk10111 Jan 27, 2025
adf216c
add entry for base-java
hk10111 Jan 27, 2025
c9c12ff
add pom.xml for base lite
hk10111 Jan 27, 2025
4914e2b
add dockerfile for cp-base-lite
hk10111 Jan 27, 2025
d3d6e45
add log4j properties
hk10111 Jan 28, 2025
35b2f3a
rename test to base_java
hk10111 Jan 28, 2025
7acf42d
modify test for base-java
hk10111 Jan 28, 2025
21b8049
add test back for cp-base-lite
hk10111 Jan 29, 2025
edcd55c
change image name
hk10111 Jan 29, 2025
b20eec5
minor optimizations
hk10111 Jan 29, 2025
106d5ea
set default ubi minimal to use ubi9
hk10111 Jan 29, 2025
d36386e
update java to 21
hk10111 Jan 29, 2025
390a255
add yes to install yum
hk10111 Jan 29, 2025
203090f
fix issues due to symlink
hk10111 Jan 29, 2025
3743bc1
fix test for directory change
hk10111 Jan 29, 2025
3d28ec3
cherry pick #622
hk10111 Jan 29, 2025
3d1450d
update base to use java21
hk10111 Jan 29, 2025
3bea075
update tag for base-lite
hk10111 Jan 29, 2025
dc790b9
replace yum with microdnf
hk10111 Jan 29, 2025
1c770cd
resolve merge conflicts
hk10111 Jan 30, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .semaphore/semaphore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ global_job_config:
- export DOCKER_UPSTREAM_REGISTRY=$DOCKER_DEV_REGISTRY
- export LATEST_TAG=$BRANCH_TAG-latest
- export DOCKER_UPSTREAM_TAG="$LATEST_TAG"
- export DOCKER_REPOS="confluentinc/cp-base-new confluentinc/cp-base-lite confluentinc/cp-jmxterm"
- export DOCKER_REPOS="confluentinc/cp-base-java confluentinc/cp-base-new confluentinc/cp-base-lite confluentinc/cp-jmxterm"
- export DOCKER_DEV_TAG="dev-$BRANCH_TAG-$BUILD_NUMBER"
- export AMD_ARCH=.amd64
- export ARM_ARCH=.arm64
Expand Down Expand Up @@ -102,7 +102,7 @@ blocks:
- export PACKAGING_BUILD_ARGS="$PACKAGING_BUILD_ARGS -DCONFLUENT_PACKAGES_REPO=$OS_PACKAGES_URL"
- mvn -Dmaven.wagon.http.retryHandler.count=3 --batch-mode -P jenkins,docker clean install dependency:analyze validate -U -Ddocker.registry=$DOCKER_DEV_REGISTRY -Ddocker.upstream-registry=$DOCKER_UPSTREAM_REGISTRY
-DBUILD_NUMBER=$BUILD_NUMBER -DGIT_COMMIT=$GIT_COMMIT -Ddocker.tag=$DOCKER_DEV_TAG$OS_TAG$AMD_ARCH -Ddocker.upstream-tag=$DOCKER_UPSTREAM_TAG$OS_TAG -Darch.type=$AMD_ARCH -Ddocker.os_type=ubi8
$PACKAGING_BUILD_ARGS -Ddependency.check.skip=true
$PACKAGING_BUILD_ARGS -Ddependency.check.skip=true -Dmaven.test.skip
- . cache-maven store
- >-
for dev_image in $AMD_DOCKER_DEV_FULL_IMAGES;
Expand Down Expand Up @@ -135,7 +135,7 @@ blocks:
- ci-tools ci-update-version
- mvn -Dmaven.wagon.http.retryHandler.count=3 --batch-mode -P jenkins,docker clean install dependency:analyze validate -U -Ddocker.registry=$DOCKER_DEV_REGISTRY -Ddocker.upstream-registry=$DOCKER_UPSTREAM_REGISTRY
-DBUILD_NUMBER=$BUILD_NUMBER -DGIT_COMMIT=$GIT_COMMIT -Ddocker.tag=$DOCKER_DEV_TAG$OS_TAG$ARM_ARCH -Ddocker.upstream-tag=$DOCKER_UPSTREAM_TAG$OS_TAG -Darch.type=$ARM_ARCH -Ddocker.os_type=ubi8
$PACKAGING_BUILD_ARGS -Ddependency.check.skip=true
$PACKAGING_BUILD_ARGS -Ddependency.check.skip=true -Dmaven.test.skip
- . cache-maven store
- for image in $ARM_DOCKER_DEV_FULL_IMAGES; do echo "Pushing $image" && docker push $image; done
epilogue:
Expand Down
68 changes: 68 additions & 0 deletions base-java/Dockerfile.ubi8
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
ARG UBI_MICRO_VERSION
ARG TEMURIN_JDK_VERSION
ARG DOCKER_UPSTREAM_REGISTRY
ARG DOCKER_UPSTREAM_TAG
ARG GOLANG_VERSION
ARG UBI_MINIMAL_VERSION

FROM docker.io/golang:${GOLANG_VERSION} AS build-ub-package-dedupe
RUN useradd --no-log-init --create-home --shell /bin/bash appuser
WORKDIR /build/package_dedupe
COPY --chown=appuser:appuser package_dedupe/ ./
RUN go build -ldflags="-w -s" ./package_dedupe.go
WORKDIR /build/ub
COPY --chown=appuser:appuser ub/ ./
RUN go build -ldflags="-w -s" ./ub.go
USER appuser
RUN go test ./...

FROM registry.access.redhat.com/ubi9-minimal:${UBI_MINIMAL_VERSION} AS REFRESH
ARG PROJECT_VERSION
ARG ARTIFACT_ID

# Remember where we came from
LABEL io.confluent.docker.git.repo="confluentinc/common-docker"

ARG GIT_COMMIT
LABEL io.confluent.docker.git.id=$GIT_COMMIT

ARG BUILD_NUMBER=-1
LABEL io.confluent.docker.build.number=$BUILD_NUMBER

LABEL maintainer="[email protected]"
LABEL vendor="Confluent"
LABEL version=$GIT_COMMIT
LABEL release=$PROJECT_VERSION
LABEL name=$ARTIFACT_ID
LABEL summary="Common base image for new Confluent lightweight Docker images."
LABEL description="Common base image for Confluent lightweight Docker images."
LABEL io.confluent.docker=true
# This affects how strings in Java class files are interpreted. We want UTF-8 and this is the only locale in the
# base image that supports it
ENV LANG="C.UTF-8"

RUN printf "[temurin-jre] \n\
name=temurin-jre \n\
baseurl=https://packages.adoptium.net/artifactory/rpm/rhel/\$releasever/\$basearch \n\
enabled=1 \n\
gpgcheck=1 \n\
gpgkey=https://packages.adoptium.net/artifactory/api/gpg/key/public \n\
" > /etc/yum.repos.d/adoptium.repo

RUN echo "installing temurin-21-jre:${TEMURIN_JDK_VERSION}" \
&& microdnf install -y temurin-21-jre${TEMURIN_JDK_VERSION} \
&& microdnf clean all \
&& useradd --no-log-init --create-home --shell /bin/bash appuser

COPY --from=build-ub-package-dedupe /build/package_dedupe /usr/bin/package_dedupe
COPY --from=build-ub-package-dedupe /build/ub /usr/bin/ub


COPY target/${ARTIFACT_ID}-${PROJECT_VERSION}-package/share/doc/* /usr/share/doc/${ARTIFACT_ID}/
COPY target/${ARTIFACT_ID}-${PROJECT_VERSION}-package/share/java/${ARTIFACT_ID}/* /usr/share/java/${ARTIFACT_ID}/
COPY --chown=appuser:appuser include/etc/confluent/docker /etc/confluent/docker
COPY --chown=appuser:appuser include/etc/cp-base-java /etc/cp-base-java


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please include files under include directory

USER appuser
WORKDIR /home/appuser
File renamed without changes.
1 change: 1 addition & 0 deletions base-java/include/etc/cp-base-java/log4j.properties
1 change: 1 addition & 0 deletions base-java/license.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Copyright 2023 Confluent, Inc.
68 changes: 68 additions & 0 deletions base-java/package_dedupe/package_dedupe.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
package main

import (
"crypto/sha1"

"fmt"

"io"
"log"
"os"
"path/filepath"
)

func dedupe_packages(rootPath string) {
sha2path := make(map[string]string)
err := filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
return nil
}
sha, err := shaSum(path)
if err != nil {
return err
}
if orig, exists := sha2path[sha]; exists {
relPath, err := filepath.Rel(filepath.Dir(path), orig)
if err != nil {
return err
}
os.Remove(path)
err = os.Symlink(relPath, path)
if err != nil {
return err
}
log.Printf("DEDUP: ln -sf %s %s\n", orig, path)
} else {
sha2path[sha] = path
}
return nil
})
if err != nil {
log.Fatal(err)
}
}

func shaSum(path string) (string, error) {
file, err := os.Open(path)
if err != nil {
return "", err
}
defer file.Close()
hash := sha1.New()
if _, err := io.Copy(hash, file); err != nil {
return "", err
}
return fmt.Sprintf("%x", hash.Sum(nil)), nil
}

func main() {
if len(os.Args) != 2 {
fmt.Println("Usage: dedupe_packages <directory_name>")
os.Exit(1)
}
basePath := os.Args[1]
dedupe_packages(basePath)
}
118 changes: 118 additions & 0 deletions base-java/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--~
~ Copyright 2017 Confluent Inc.
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>io.confluent</groupId>
<artifactId>common-docker</artifactId>
<version>8.0.0-0</version>
</parent>

<packaging>pom</packaging>

<artifactId>cp-base-java</artifactId>

<description>Base for new Confluent lightweight Docker images</description>
<name>${project.artifactId}</name>

<properties>
<docker.skip-build>false</docker.skip-build>
<docker.skip-test>false</docker.skip-test>
<docker.pull-image>true</docker.pull-image>
</properties>

<dependencies>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>utility-belt</artifactId>
<version>${io.confluent.common-docker.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>${maven-resources-plugin.version}</version>
<executions>
<execution>
<id>copy-resources</id>
<phase>compile</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<overwrite>true</overwrite>
<outputDirectory>target</outputDirectory>
<resources>
<resource>
<directory>${basedir}/../docker-utils/target</directory>
<includes>
<include>docker-utils-${CONFLUENT_VERSION}-jar-with-dependencies.jar</include>
</includes>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<configuration>
<buildArgs>
<UBI_MINIMAL_VERSION>${ubi9.image.version}</UBI_MINIMAL_VERSION>
<TEMURIN_JDK_VERSION>-${ubi.temurin.jdk.version}</TEMURIN_JDK_VERSION>
<SKIP_SECURITY_UPDATE_CHECK>${docker.skip-security-update-check}</SKIP_SECURITY_UPDATE_CHECK>
<GOLANG_VERSION>${golang.version}</GOLANG_VERSION>
</buildArgs>
</configuration>
</plugin>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.43.4</version>
<configuration>
<images>
<image>
<build>
<args>
<UBI_MINIMAL_VERSION>${ubi9.image.version}</UBI_MINIMAL_VERSION>
<TEMURIN_JDK_VERSION>-${ubi.temurin.jdk.version}</TEMURIN_JDK_VERSION>
<SKIP_SECURITY_UPDATE_CHECK>
${docker.skip-security-update-check}
</SKIP_SECURITY_UPDATE_CHECK>
<GOLANG_VERSION>${golang.version}</GOLANG_VERSION>
</args>
</build>
</image>
</images>
</configuration>
</plugin>
</plugins>
</build>
</project>
File renamed without changes.
File renamed without changes.
28 changes: 28 additions & 0 deletions base-java/test/test_base_java_image.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import os
import unittest

import confluent.docker_utils as utils
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep this file in base-lite also to make sure base-lite is getting built properly also.



class BaseJavaImageTest(unittest.TestCase):

def setUp(self):
self.image = "{0}confluentinc/cp-base-java:{1}".format(os.environ["DOCKER_REGISTRY"], os.environ["DOCKER_TAG"])

def test_image_build(self):
self.assertTrue(utils.image_exists(self.image))

def test_ub_exists(self):
self.assertTrue(utils.path_exists_in_image(self.image, "/usr/bin/ub"))

def test_ub_runnable(self):
ub_cmd = "bash -c '/usr/bin/ub/ub -h'"
self.assertTrue(b"utility commands" in utils.run_docker_command(image=self.image, command=ub_cmd))

def test_kafka_ready_jar(self):
java_cmd = "bash -c 'java -cp \"/usr/share/java/cp-base-java/*\" io.confluent.admin.utils.cli.KafkaReadyCommand -h'"
self.assertTrue(b"Check if Kafka is ready" in utils.run_docker_command(image=self.image, command=java_cmd))


if __name__ == '__main__':
unittest.main()
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
36 changes: 8 additions & 28 deletions base-lite/Dockerfile.ubi8
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.

ARG UBI_MINIMAL_VERSION="latest"
ARG GOLANG_VERSION

FROM golang:${GOLANG_VERSION} AS build-ub
WORKDIR /build
RUN useradd --no-log-init --create-home --shell /bin/bash appuser
COPY --chown=appuser:appuser ub/ ./
RUN go build -ldflags="-w -s" ./ub.go
USER appuser
RUN go test ./...

FROM registry.access.redhat.com/ubi8/ubi-minimal:${UBI_MINIMAL_VERSION}
ARG DOCKER_UPSTREAM_REGISTRY
ARG DOCKER_TAG
FROM ${DOCKER_UPSTREAM_REGISTRY}confluentinc/cp-base-java:${DOCKER_TAG}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use new arg/env for the base image name as there are symlinks also added in the Dockerfile


ARG PROJECT_VERSION
ARG ARTIFACT_ID
Expand Down Expand Up @@ -56,24 +47,15 @@ ARG TEMURIN_JDK_VERSION=""

ENV UB_CLASSPATH=/usr/share/java/cp-base-lite/*

RUN printf "[temurin-jdk] \n\
name=temurin-jdk \n\
baseurl=https://packages.adoptium.net/artifactory/rpm/rhel/\$releasever/\$basearch \n\
enabled=1 \n\
gpgcheck=1 \n\
gpgkey=https://packages.adoptium.net/artifactory/api/gpg/key/public \n\
" > /etc/yum.repos.d/adoptium.repo

RUN microdnf --nodocs install yum \
&& yum --nodocs update -y \
&& yum --nodocs install -y --setopt=install_weak_deps=False \
USER root

RUN yum --nodocs update -y \
&& yum --nodocs install -y --setopt=install_weak_deps=False --allowerasing \
"curl${CURL_VERSION}" \
"temurin-21-jre${TEMURIN_JDK_VERSION}" \
&& microdnf clean all \
&& yum clean all \
&& rm -rf /tmp/* \
&& mkdir -p /etc/confluent/docker /usr/logs \
&& useradd --no-log-init --create-home --shell /bin/bash appuser \
&& chown appuser:appuser -R /etc/confluent/ /usr/logs

# This is a step that will cause the build to fail of the package manager detects a package update is availible and isn't installed.
Expand All @@ -88,12 +70,10 @@ RUN yum --disablerepo="temurin-jdk" check-update || "${SKIP_SECURITY_UPDATE_CHEC
COPY --chown=appuser:appuser target/${ARTIFACT_ID}-${PROJECT_VERSION}-package/share/doc/* /usr/share/doc/${ARTIFACT_ID}/
COPY --chown=appuser:appuser target/${ARTIFACT_ID}-${PROJECT_VERSION}-package/share/java/${ARTIFACT_ID}/* /usr/share/java/${ARTIFACT_ID}/

COPY --chown=appuser:appuser include/etc/confluent/docker /etc/confluent/docker
COPY --chown=appuser:appuser include/etc/cp-base-lite /etc/cp-base-lite
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please make this directory a symlink to /etc/cp-base-java.Similarly symlink /usr/share/java/cp-base-lite to /usr/share/java/cp-base-java

COPY --from=build-ub /build/ub /usr/bin

RUN mkdir /licenses
COPY license.txt /licenses

USER appuser
WORKDIR /home/appuser
WORKDIR /home/appuser
Loading