Refactor Docker Images, Evaluator Images Simple CI by JersyJ · Pull Request #823 · mrlvsb/kelvin

JersyJ · 2026-02-07T00:54:40Z

Evaluator Docker Image Docker deployment #503
Docker Compose Evaluator Scheduler service definition - configuration with scheduler (1 rq worker)
Docker Compose Evaluator CPU Workers service definition - configuration with CPU workers (32 by default , EVALUATOR_CPU_REPLICAS)
Docker Compose Evaluator GPU Workers service definition - configuration with GPU workers (32 by default, EVALUATOR_CUDA_REPLICAS)
EVALUATOR_REDIS__HOST
EVALUATOR_REDIS__PORT
Reworked Evaluator Docker images (base/run/java/dotnet/clang-tidy/cargo/pythonrun)
Kelvin: Updated pipeline configuration/validation to allow selecting a Docker image for the "tests" type (pipe).
CI for Evaluator Docker Images with build.py (Unable to use GitHub Action Cache due to docker-container driver design/architecture (Cannot give a local image to FROM when using docker-container moby/buildkit#2343), reverting to local docker driver for compatibility.)
Update CI to use native GitHub Actions Cache instead of the Registry cache
Ruff format with Python 3.12 f-string compatibility
Document undocumented environment variable and add logic for running Evaluator inside container (API_INTERNAL_BASEURL)
Updated pyproject.toml dev dependency group (removed deprecated syntax)
Migrate from pre-commit to Rust re-implementation called prek
Update documentation and fix instructions that no longer works

… refine Docker images and build processes.

…mage builds.

…:latest` tag from `kelvin/` parent image names in build script.

…PU, and CUDA workers, and update the evaluator Dockerfile entrypoint for command flexibility.

Copilot

Pull request overview

Refactors Kelvin’s Docker image setup (app + evaluator images) and CI workflows, while also updating developer tooling (switching from pre-commit to prek) and improving evaluator worker runtime behavior for Docker-based deployments.

Changes:

Reworked multiple Docker images (base/run/java/dotnet/clang-tidy/cargo/pythonrun) and added an evaluator worker image + build script.
Updated pipeline configuration/validation to allow selecting a Docker image for the tests pipe.
Updated CI caching strategy and developer tooling/docs (uv dependency groups, prek hooks).

Reviewed changes

Copilot reviewed 30 out of 31 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
web/views/student.py	String formatting style adjustments.
uv.lock	Removes `pre-commit` deps, adds `prek`, updates locked metadata.
pyproject.toml	Migrates dev deps to `[dependency-groups]`, adds `prek`.
frontend/src/PipelineValidation.js	Adds `tests.image` validation + description.
evaluator/testsets.py	Adjusts warning f-string quoting.
evaluator/pipelines.py	Adds configurable `tests` image and changes docker ulimit formatting.
evaluator/images/run/Dockerfile	Refactors apt install (leaner image, cleanup).
evaluator/images/pythonrun/Dockerfile	Switches base + pins pytest/flake8.
evaluator/images/java/entry.py	Removes runtime env-variable injection logic.
evaluator/images/java/Dockerfile	Refactors installs and sets JAVA/Maven env via `ENV`.
evaluator/images/gcc/entry.py	Adjusts PATH f-string quoting.
evaluator/images/gcc/Dockerfile	Minor formatting cleanup.
evaluator/images/dotnet/Dockerfile	Uses `kelvin/base:latest`, refactors apt install, adds env workaround.
evaluator/images/clang-tidy/Dockerfile	Switches from Alpine to Debian-based `kelvin/gcc` and installs tools via apt.
evaluator/images/cargo/entry.py	Adjusts warning string formatting.
evaluator/images/cargo/Dockerfile	Refactors apt install, bumps bleach.
evaluator/images/build.py	Major refactor into an image dependency scanner/builder CLI.
evaluator/images/base/Dockerfile	Consolidates base deps, locale env, and bleach install.
evaluator/Dockerfile	Adds a dedicated evaluator worker image that installs Docker CLI/engine packages.
docs/docs/01-intro/01-installation.mdx	Updates installation steps for prek + worker command.
docker-compose.yml	Adds internal base URL env, nginx aliasing, and evaluator worker services.
common/utils.py	Adds `API_INTERNAL_BASEURL` handling with DEBUG guard.
common/event_log.py	Fixes f-string quoting in `__str__`.
common/evaluate.py	Adds DEBUG TLS verify override and ensures temp dir is under system temp `/tmp/kelvin`.
api/views/default.py	Adjusts error f-string quoting.
Dockerfile	Reworks build/runtime stages, uv installation method, runtime deps, and user/group creation.
.pre-commit-config.yaml	Updates hook versions and consolidates ruff hooks.
.github/workflows/ci.yml	Uses job defaults for deployment_service; switches Docker cache to GHA.
.github/workflows/build-evaluator-images.yml	Adds workflow to build evaluator images when relevant files change.
.env.example	Documents API_INTERNAL_BASEURL and evaluator worker envs.
.dockerignore	Expands ignore patterns for Python/Node/editor artifacts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

evaluator/pipelines.py

evaluator/Dockerfile

Dockerfile

common/evaluate.py

docs/docs/01-intro/01-installation.mdx

…tation, and change default image in TestsPipe

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docker-compose.yml

evaluator/images/build.py

evaluator/Dockerfile

evaluator/pipelines.py

common/evaluate.py

evaluator/images/java/Dockerfile

…te Dockerfile comments for clarity

…nd deployment

…sections

Kobzol

Please separate a PR that only changes the CI, let's start with that.

Kobzol · 2026-02-09T13:21:14Z

.github/workflows/build-evaluator-images.yml

+  # On PRs, we need everything to be green, while deploy jobs are skipped.
+  # On master, we need everything to be green.
+  # ALL THE PREVIOUS JOBS NEED TO BE ADDED TO THE `needs` SECTION OF THIS JOB!
+  conclusion:


The conclusion job won't work like this, I think I wrote it to you before. We either have to merge the two workflows, or use a different name for the conclusion job here (e.g. conclusion-images), so that we can configure CI to wait for both jobs to be green.

Kobzol · 2026-02-09T13:22:46Z

.github/workflows/build-evaluator-images.yml

+on:
+  pull_request:
+  merge_group:
+  workflow_dispatch:


The main issue that we have with the images is not that they break after we change them, but that they break when something external changes, most often apt repositories. So it would be great to run CI periodically to detect that sooner.

One way of doing that is running them always in CI, without file change detection. That has the annoying property that it can break CI for unrelated PRs. Another possibility is to setup a cron, to run this e.g. once a week. I'd go with the cron for now (in addition to the existing triggers).

Kobzol · 2026-02-09T13:23:30Z

.github/workflows/ci.yml

        uses: docker/build-push-action@v6
        with:
-          cache-from: type=registry,ref=ghcr.io/mrlvsb/kelvin-ci-cache
+          cache-from: type=gha


Why was the switch made?

Due to maintenance of the registry storage (LRU is used there automatically) and also the readability/visibility of that registry. Also this is official way and recommendation from GitHub and Docker.

JersyJ · 2026-02-09T15:54:45Z

Closing due to #824 #825 #826 #827

JersyJ added 20 commits February 6, 2026 23:37

Introduce evaluator Docker-in-Docker setup, add SSL certificates, and…

4b1e0c1

… refine Docker images and build processes.

Refactor Evaluator Docker Images, Introduces simple CI

6b80e5b

Remove the evaluator service from docker-compose.yml

61fa33e

Rename workflow

bf8ec57

Formatting, pre-commit

a0b337e

Build context

1009628

Update name, and build context

25233f1

Try to use context use default

c323f44

feat: Inject build contexts for local 'kelvin/' dependencies during i…

fe00a11

…mage builds.

Another try

a0d9156

feat: explicitly tag Docker images with :latest

527eaaf

build: Explicitly specify 'latest' tag for base image in Dockerfile.

a619225

chore: Remove explicit docker context usage from workflow and strip `…

dfc016e

…:latest` tag from `kelvin/` parent image names in build script.

Try again

76b3825

Go back to the "docker" driver

dd0e1d8

ntroduce dedicated Docker Compose services for evaluator scheduler, C…

7537da6

…PU, and CUDA workers, and update the evaluator Dockerfile entrypoint for command flexibility.

Migrate to prek

bb9fec4

Try the default directory

e21081f

Improve internal API communication, add debug-mode SSL handling

f523239

Documentation for env.example

8faa012

JersyJ marked this pull request as ready for review February 8, 2026 15:17

Copilot AI review requested due to automatic review settings February 8, 2026 15:17

Copilot started reviewing on behalf of JersyJ February 8, 2026 15:18 View session

Copilot AI reviewed Feb 8, 2026

View reviewed changes

Update UV version in CI and Dockerfiles, improve installation documen…

1fe0b1f

…tation, and change default image in TestsPipe

JersyJ requested a review from Copilot February 8, 2026 16:26

Copilot started reviewing on behalf of JersyJ February 8, 2026 16:26 View session

Copilot AI reviewed Feb 8, 2026

View reviewed changes

JersyJ added 2 commits February 8, 2026 17:45

Refactor Docker configuration: remove unused network aliases and upda…

df15c62

…te Dockerfile comments for clarity

Implement evaluator image building via an entrypoint script

8424e37

JersyJ added 8 commits February 8, 2026 22:50

Healtcheck Docker Status mode for Deployment Service

0e60ace

Move evaluator Dockerfile into a multi-stage build, Evaluator builf a…

95a0109

…nd deployment

Fix Mypy issue

a8c126d

Simplify DooD socket access via socat and remove DOCKER_GROUP_ID

254f02a

Add fail-fast and unhealthy state to health_check

68eea93

Add health_check_timeout to settings and deployment request model

52f0e86

Add health check timeout option to deployment and documentation

c63b6b7

Add documentation for evaluator images and update tests and pipeline …

0199ceb

…sections

JersyJ force-pushed the evaluator-refactor-images branch from 498b373 to 0199ceb Compare February 9, 2026 00:46

Kobzol reviewed Feb 9, 2026

View reviewed changes

Merge branch 'evaluator-deployment' into evaluator-refactor-images

9ef32d5

JersyJ marked this pull request as draft February 9, 2026 14:00

JersyJ added 2 commits February 9, 2026 15:03

Rename conclusion job to conclusion-images for clarity in workflow

7424d46

Remove change detection and simplify image build process

16f8252

JersyJ closed this Feb 9, 2026

Conversation

JersyJ commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kobzol left a comment

Choose a reason for hiding this comment

Uh oh!

Kobzol Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Kobzol Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Kobzol Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

JersyJ Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

JersyJ commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JersyJ commented Feb 7, 2026 •

edited

Loading