appendMirrors does not work as expected with RKE2 versions higher than v1.28

[x1ld3n]

Spegel version

v0.0.27

Kubernetes distribution

RKE2

Kubernetes version

v1.29.3

CNI

Calico

Describe the bug

First of all thanks for this awesome project, it helps a lot!

I have 2 RKE2 clusters with version 1.28.7 and one with 1.27.10 that pull images from a private gitlab registry with the content of /etc/rancher/rke2/registries.yaml:

configs:
  gitlab-registry:1337:
    tls:
      ca_file: /etc/rancher/rke2/certs/CA.crt

If i set appendMirrors: true in the helm chart values, it merges the hosts.toml file as expected:

server = 'https://gitlab-registry:1337'

[host.'http://10.5.13.134:30020']
capabilities = ['pull', 'resolve']

[host.'http://10.5.13.134:30021']
capabilities = ['pull', 'resolve']

[host.'https://gitlab-registry:1337/v2']
ca = ['/etc/rancher/rke2/certs/CA.crt']
capabilities = ['pull', 'resolve']

Now with other RKE2 clusters of versions 1.29.3, 1.30.1 and 1.30.5 it does not merge the files. The content of /etc/rancher/rke2/registries.yaml is exactly the same in all clusters but I end up with the config file like this:

server = 'https://gitlab-registry:1337'

[host.'http://10.5.23.52:30020']
capabilities = ['pull', 'resolve']

[host.'http://10.5.23.52:30021']
capabilities = ['pull', 'resolve']

while the hosts.toml file from _backup folder looks like this:

# File generated by rke2. DO NOT EDIT.

server = "https://gitlab-registry:1337/v2"
capabilities = ["pull", "resolve", "push"]

ca = ["/etc/rancher/rke2/certs/CA.crt"]

and I end up getting x509: certificate signed by unknown authority error when pulling new images from this private registry, while already existing images are pulled from other nodes as expected.

In the configuration container log it says that it appended the configuration:

{"time":"2024-11-25T19:36:06.3668089Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/oci.AddMirrorConfiguration","file":"/build/pkg/oci/containerd.go","line":376},"msg":"appending to existing Containerd mirror configuration","registry":"https://gitlab-registry:1337"}
{"time":"2024-11-25T19:36:06.366918714Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/oci.AddMirrorConfiguration","file":"/build/pkg/oci/containerd.go","line":387},"msg":"added Containerd mirror configuration","registry":"https://gitlab-registry:1337","path":"/var/lib/rancher/rke2/agent/etc/containerd/certs.d/gitlab-registry:1337/hosts.toml"}

here is my values file for all clusters:

image:
  tag: "v0.0.27"
spegel:
  logLevel: "DEBUG"
  registries:
    - https://cgr.dev
    - https://docker.io
    - https://ghcr.io
    - https://quay.io
    - https://mcr.microsoft.com
    - https://public.ecr.aws
    - https://gcr.io
    - https://registry.k8s.io
    - https://k8s.gcr.io
    - https://lscr.io
    - https://gitlab-registry:1337
  containerdMirrorAdd: true
  resolveLatestTag: false
  appendMirrors: true
  containerdSock: "/run/k3s/containerd/containerd.sock"
  containerdRegistryConfigPath: "/var/lib/rancher/rke2/agent/etc/containerd/certs.d"
  containerdContentPath: "/var/lib/rancher/rke2/agent/containerd/io.containerd.content.v1.content"

[dereknola]

RKE2 and K3s do not support standalone spegel deployments. Those distros manage their containerd config files, and will override the outside "interference" of spegel deployed as a helm chart. See https://docs.k3s.io/installation/registry-mirror (the same stuff applies to RKE2).

[brandond]

tl;dr spegel is built into rke2 already. Uninstall the spegel chart, add embedded-registry: true to config.yaml on your servers, and restart rke2.

Upgrade to a non-EOL release of RKE2 first though. Both 1.27 and 1.28 are end of life, and you're not even on the most recent patch release.

[x1ld3n]

Thank you for the suggestions. Can you help me how can I read the spegel logs when using embedded-registry: true because I can't get it to work with mirrors: "*" in registries.yaml

[dereknola]

Per the docs (with equivalent rke2 versions)

Wildcard support is available as of the March 2024 releases: v1.26.15+k3s1, v1.27.12+k3s1, v1.28.8+k3s1, v1.29.3+k3s1

[brandond]

I can't get it to work with mirrors: "*" in registries.yaml
version 1.28.7 and one with 1.27.10

Both 1.27 and 1.28 are end of life, and you're not even on the most recent patch release.

Upgrade!!!