Clarification on "embedded etcd"

[rpivo]

After doing some digging, this is what I'm understanding by "embedded etcd":

A k8s cluster would require manual setup of etcd.
A k8s cluster would also have a dedicated etcd pod (on each control plane node?).
In k3s, etcd is "embedded" in the k3s server process that runs on control plane nodes as opposed to running in its own pod.

Is that right?

I'm guessing this is more obvious to seasoned Kubernetes folk, but it would be nice if there was some extra docs on that on the docs website. If there's a desire for this, I'm happy to type something up and create an initial PR for review.

[brandond]

K3s runs all the Kubernetes components in the main process. If you've enabled etcd, this includes etcd.

What specifically are you looking for documentation on? Have you looked at the Architecture page?

[tcordina]

Sorry for reviving a 9 months old topic, but I'm trying to understand how metrics exposure works in K3s and i came across this discussion, and this reply has me confused.

Quoting from the docs: "K3s runs all Kubernetes components in the main K3s process. Since Kubernetes uses a single Prometheus metric registry per process, metrics for all components are available via all exposed metrics endpoints.".

If etcd also runs in the main process, I fail to understand how it can have its own separated metrics endpoint, exposing only etcd related metrics, while other component endpoints like kubelet, kube-proxy or apiserver expose metrics for every component at the same time except etcd.

Based on that, it seems etcd lives in a separate process, yet your reply says otherwise.

Am I missing something here?

[brandond]

Yes, you are.

All the Kubernetes components share a common metrics library that wraps prometheus and is used to register and gather metrics: k8s.io/component-base/metrics/legacyregistry. When components are running in the same process, they are wired up through the same registerer/gatherer instance by way of the common library.

etcd is a different project, and registers/gathers metrics differently. So they are not comingled with the kubernetes metrics.

You could also just... look at a running K3s server and note that there is not a separate etcd process.

[tcordina]

Ok that makes sense, thanks for your quick reply and explanation.

[rpivo]

Ultimately I think I was confused by this diagram as it doesn't really distinguish the fact that the etcd process is not just a pod on the node but is instead inside the main process.
I didn't read all of the docs, so maybe this is just my impression given the pieces that I did read.

[brandond]

None of the Kubernetes components run as pods in K3s. This is kind of the whole thing about K3s that makes it unique.