Docker Deployment For Kelvin Evaluator by JersyJ · Pull Request #825 · mrlvsb/kelvin

JersyJ · 2026-02-09T15:03:47Z

Evaluator Docker Image Docker deployment #503
Docker Compose Evaluator Scheduler service definition - configuration with scheduler (1 rq worker)
Docker Compose Evaluator CPU Workers service definition - configuration with CPU workers (32 by default , EVALUATOR_CPU_REPLICAS)
Docker Compose Evaluator GPU Workers service definition - configuration with GPU workers (32 by default, EVALUATOR_CUDA_REPLICAS)
EVALUATOR_REDIS__HOST
EVALUATOR_REDIS__PORT
Document undocumented environment variable and add logic for running Evaluator inside container (API_INTERNAL_BASEURL)

Copilot

Pull request overview

Adds a Docker-deployable “Kelvin Evaluator” component (image + compose services) and updates backend logic/config so evaluator workers can run inside containers and communicate using an internal base URL.

Changes:

Introduces an evaluator Docker image target and an entrypoint that runs evaluator image builds before starting workers.
Extends docker-compose.yml with evaluator scheduler/CPU/GPU worker services and a Docker socket TCP proxy.
Updates backend utilities for Docker-internal URL building and evaluator job temp directory handling; updates CI to build/push the evaluator image.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
evaluator/evaluator-entrypoint.sh	New evaluator entrypoint that builds evaluator sub-images, then runs `manage.py` commands.
docker-compose.yml	Adds evaluator services and docker socket proxy; adds internal base URL env var; adjusts pull policies.
common/utils.py	Adds `API_INTERNAL_BASEURL` handling with a production safety guard.
common/evaluate.py	Disables TLS verification in DEBUG and changes evaluation temp dir base.
Dockerfile	Switches build base image approach; adds `evaluator` target with Docker tooling and entrypoint.
.github/workflows/ci.yml	Builds, uploads, loads, and pushes the new `kelvin-evaluator` image in CI/deploy.
.env.example	Documents evaluator-related env vars and `API_INTERNAL_BASEURL`.
.dockerignore	Expands ignore patterns for cleaner Docker build contexts.

Comments suppressed due to low confidence (1)

docker-compose.yml:15

pull_policy: never for the app service prevents pulling prebuilt images from GHCR and can cause prod deployments to use stale images or fail when the image tag isn’t present locally. If the deployment flow relies on ${APP_IMAGE_TAG} (as the comment suggests), this should remain always (or be configurable via an env var) rather than hard-coded to never.

    image: "ghcr.io/mrlvsb/kelvin:${APP_IMAGE_TAG:-latest}" # Interpolation for Deployment Service
    pull_policy: always

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docker-compose.yml

common/evaluate.py

docker-compose.yml

evaluator/evaluator-entrypoint.sh

Dockerfile

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docker-compose.yml

evaluator/evaluator-entrypoint.sh

docker-compose.yml

common/evaluate.py

docker-compose.yml

Kobzol

Left some comments.

common/evaluate.py

Kobzol · 2026-02-19T07:27:53Z

common/utils.py

+    # If the URL is the default Docker-internal one, only use it in DEBUG mode.
+    # This prevents Production from accidentally using the internal container hostname
+    # instead of the public domain, unless explicitly forced.
+    if base_uri == "https://nginx" and not settings.DEBUG:


This needs more explanation somewhere.

Added description

Ok, I think that I understand the issue now. There are a few things that are not ideal here:

We require HTTPS in nginx in local deployment (but this is more related to the s.verify = False thing, not this, though it is also related to this

We set the default value of the internal API to nginx; it is not clear if that is indeed a good default, especially given that we don't really expect to run the evaluators in the same Docker network as the web in production.

This function looks like it should be used for any general URL generation, but in reality it is only used for evaluators.

So I would suggest this:

Put API_INTERNAL_BASEURL=https://nginx to .env.example, so that it is used by default in local Docker deployment, without it being specified in docker-compose.yml

Remove the :-https://nginx default in docker-compose.yml; make the variable required

Move the variable loading to settings.py, an environment variable shouldn't be accessed randomly in one of Kelvin's functions, but rather be centralized in the website configuration

Rename the variable e.g. to EVALUATION_LINK_BASEURL

Only use the variable when generating an URL for evaluators (including LLM evaluation), not for "normal" public URL links, such as for e-mails. I think that this actually already happens, as the non-evaluator part of Kelvin uses request.build_absolute_uri directly. So it is enough to rename build_absolute_uri to e.g. build_evaluation_download_uri, or something like that, to make it clear for what it should (and shouldn't!) be used.

Changed mostly based on your suggestions, please review it, if it is acceptable

Kobzol · 2026-02-19T07:28:45Z

docker-compose.yml

      - KELVIN__HOST_URL=${KELVIN__HOST_URL}
+      # - Defaults to 'https://nginx' for local docker development (to fix loopback ref to 127.0.0.1)
+      # - IGNORED by app if value is 'https://nginx' AND DEBUG=False
+      - API_INTERNAL_BASEURL=${API_INTERNAL_BASEURL:-https://nginx}


What is this for?

Added more description

docker-compose.yml

Dockerfile

…-compose

Kobzol · 2026-02-19T15:37:09Z

common/utils.py

+    # If the URL is the default Docker-internal one, only use it in DEBUG mode.
+    # This prevents Production from accidentally using the internal container hostname
+    # instead of the public domain, unless explicitly forced.
+    if base_uri == "https://nginx" and not settings.DEBUG:


Ok, I think that I understand the issue now. There are a few things that are not ideal here:

We require HTTPS in nginx in local deployment (but this is more related to the s.verify = False thing, not this, though it is also related to this

We set the default value of the internal API to nginx; it is not clear if that is indeed a good default, especially given that we don't really expect to run the evaluators in the same Docker network as the web in production.

This function looks like it should be used for any general URL generation, but in reality it is only used for evaluators.

So I would suggest this:

Put API_INTERNAL_BASEURL=https://nginx to .env.example, so that it is used by default in local Docker deployment, without it being specified in docker-compose.yml

Remove the :-https://nginx default in docker-compose.yml; make the variable required

Move the variable loading to settings.py, an environment variable shouldn't be accessed randomly in one of Kelvin's functions, but rather be centralized in the website configuration

Rename the variable e.g. to EVALUATION_LINK_BASEURL

Only use the variable when generating an URL for evaluators (including LLM evaluation), not for "normal" public URL links, such as for e-mails. I think that this actually already happens, as the non-evaluator part of Kelvin uses request.build_absolute_uri directly. So it is enough to rename build_absolute_uri to e.g. build_evaluation_download_uri, or something like that, to make it clear for what it should (and shouldn't!) be used.

Kobzol · 2026-02-19T15:43:29Z

evaluator/evaluator-entrypoint.sh

+
+# Run the image builder to ensure all required images are present
+# Skip image build if running as scheduler (detected via --with-scheduler arg)
+if [[ "$*" != *"--with-scheduler"* ]]; then


--with-scheduler doesn't tell anything about whether the evaluator will need the images or not.

Let's just inline the whole command into docker-compose.yml, seems like the simplest solution without doing similar hacks.

docker-compose.yml

…ronment configurations

Kobzol

Thank you, left one comment.

Kobzol · 2026-02-23T10:23:29Z

docker-compose.yml

+  # or solve issues with socket permissions
+  docker_proxy:
+    container_name: kelvin_docker_proxy
+    profiles: [ prod,evaluator_cpu,evaluator_cuda ]


Hmm, this means that the Docker proxy will be running on the main server, even if there will be no evaluators there. Seems safer to just not do that.

Suggested change

profiles: [ prod,evaluator_cpu,evaluator_cuda ]

profiles: [ evaluator_cpu, evaluator_cuda ]

But it is required for deployment service :D

Kobzol

Ok, let's try. Thank you.

Copilot AI review requested due to automatic review settings February 9, 2026 15:03

Copilot started reviewing on behalf of JersyJ February 9, 2026 15:04 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

JersyJ requested a review from Copilot February 9, 2026 15:37

Copilot started reviewing on behalf of JersyJ February 9, 2026 15:37 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

JersyJ mentioned this pull request Feb 9, 2026

Refactor Docker Images, Evaluator Images Simple CI #823

Closed

Kobzol reviewed Feb 19, 2026

View reviewed changes

JersyJ added 5 commits February 19, 2026 12:51

Docker Deployment For Kelvin Evaluator

4b7bb73

Update pull policy to always for app and evaluator services in docker…

d05cf1f

…-compose

Revert folder prefix /kelvin

289d2ee

Depends-on proxy

84b87b4

Enhance clarifications and skip building images on scheduler container

592a9bd

JersyJ force-pushed the docker-evaluator branch from 3cba5e3 to 592a9bd Compare February 19, 2026 11:51

Kobzol reviewed Feb 19, 2026

View reviewed changes

Refactor evaluator setup for local Docker development and update envi…

f0695f2

…ronment configurations

JersyJ force-pushed the docker-evaluator branch from 2b00528 to f0695f2 Compare February 22, 2026 23:31

Kobzol reviewed Feb 23, 2026

View reviewed changes

Kobzol approved these changes Feb 23, 2026

View reviewed changes

Kobzol added this pull request to the merge queue Feb 23, 2026

Merged via the queue into mrlvsb:master with commit 3ebf97b Feb 23, 2026
10 checks passed

	profiles: [ prod,evaluator_cpu,evaluator_cuda ]
	profiles: [ evaluator_cpu, evaluator_cuda ]

Conversation

JersyJ commented Feb 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kobzol left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kobzol left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kobzol left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants