Feature: cache all images of namespace

[1337andre]

Hi, thanks for great project.

We have separated our processes by namespaces.
e.g. monitoring, logging, metering, velero... each namespace have it's own CR imagecache and its own pullSecret.

Would it be possible to add a flag to take/cache all images in namespace? This way we don't need to maintain image in cache list and in deployment itself.

greetings
Andre

[tculp]

As an alternative approach, perhaps kube-fledged could watch for a certain pod annotation to automatically add the requested image to the cache, and subscribe the node to the image.

This functionality could work similar to cert-manager, which watches for a cert-manager.io/cluster-issuer annotation on ingresses to create a Certificate Resource, rather than requiring the user to create one manually.

[senthilrch]

Automatically determining which images to be cached is in the roadmap of kube-fledged. Would like to see more elaborative details on how best this could be implemented.

• should kube-fledged watch for annotations in Pods or Deployment/Statefulset/Daemonset?
• how can we generalize that only a subset of Pods running the cluster should be used to cache those images
• How kubefledged-controller would determine the nodes to which those images would be cached?

[vikas027]

Hey @senthilrch ,

What do you think about controlling the same through annotations?

• kube-fledged controller watches for annotations something like kubefledged.io/enable: "true".
• If the annotation is true with another annotation like kubefledged.io/node-selector: "somekey=somevalue" then the controller only creates cache on nodes matching the label. Also, if this annotation is present, then there is no need to check if kubefledged.io/enable is set or not.

[senthilrch]

• A Pod could have multiple init containers and app containers. We should have a means to specify whether to cache all the container images or only a subset of container images.
• It should be possible to set the nodeSelector for individual images in the Pod apart from setting a default nodeSelector applicable for all Pods.
• If no nodeSelector is specified, it implies images should be cached onto all the nodes.

[vikas027]

A Pod could have multiple init containers and app containers. We should have a means to specify whether to cache all the container images or only a subset of container images.

Right - maybe we can control the same using another annotation like kubefledged.io/pod-images: "image-*,image-*". If the annotation is present, only pull the mentioned images (with a wildcard), else pull all images in the pod.

[tculp]

Given that a pod requires all of its images to be present to start up, I think it would actually be a pretty rare occurrence to only want a subset of images on a pod to be pulled. Maybe that level of detail could come later?

[amitai-devops]

@senthilrch @1337andre and whoever is interested:

Until this feature is implemented, I created a wrapper chart for kube-fledged which manages ImageCaches automatically with kubernetes events. It's not yet perfect but it gets the job done:

https://github.com/explorium-ai/kube-auto-puller

[senthilrch]

@1337andre @tculp @vikas027 :

I propose to create a new CRD to allow harvesting images for the cache from Kubernetes objects. This allows a less-intrusive (resources needs not be labelled/annotated) and highly flexible way of declaring the "sources" for harvesting images. Even images from customer resources (e.g. ArgoWF, SeldonDeployment etc.) can be supported with this design approach:-

apiVersion: kubefledged.io/v1alpha1
kind: ImageCacheBuilder
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache2
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  - apiGroups:
      - "apps"
    resources:
      - statefulsets
    namespaceSelector:
      matchNames:
      - sock-shop
    selector:
      matchLabels:
        name: front-end     
  - apiGroups:
      - "apps"
    resources:
      - deployments
    namespaceSelector:
      matchNames:
      - sock-shop
    selector:
      matchLabels:
        name: front-end

Harvesting images from Deployments and Statefulsets will be supported in v0.11.0. i.e. Pods have to be created using these controllers. Images won't be harvested from Pods.