diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 2b3c762..0000000 --- a/docs/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# FSM Reconciler Framework - -This guide has moved to the [Achilles docs website](https://pages.github.snooguts.net/reddit/achilles-docs/dev-guides/sdk-apply-objects/). diff --git a/docs/developer/updating_resources.md b/docs/developer/updating_resources.md deleted file mode 100644 index fe6af3d..0000000 --- a/docs/developer/updating_resources.md +++ /dev/null @@ -1,3 +0,0 @@ -# Updating Resources - -This guide has moved to the [Achilles docs website](https://pages.github.snooguts.net/reddit/achilles-docs/dev-guides/sdk-apply-objects/). diff --git a/docs/envtest.md b/docs/envtest.md new file mode 100644 index 0000000..d8e1656 --- /dev/null +++ b/docs/envtest.md @@ -0,0 +1,305 @@ +# envtest + +This document describes what `envtest` integration tests are, why they are valuable for testing Kubernetes controllers, +and how to write them. + +## What is `envtest`? + +`envtest` is a binary that runs an instance of `kube-apiserver` and `etcd`, enabling integration testing of Kubernetes +controllers and its interaction against the Kubernetes control plane. It is part of the [Kubebuilder](https://book.kubebuilder.io/) project, +and the `envtest` docs can be [found here](https://book.kubebuilder.io/reference/envtest.html). + +The Achilles SDK provides a Go wrapper around `envtest`, making it easy to programmatically start, stop, and integrate controllers being tested against +the test environment. + +## Why Use `envtest`? + +The bulk of the implementation of any Kubernetes controller, regardless of business logic, reduces down to CRUD operations +against the Kubernetes API server. Therefore, a controller's core correctness can be exercised as asserting that the +controller instantiates the expected Kubernetes state (i.e. child objects) in response to changes in the declared state (i.e. parent objects). + +Furthermore, the Kubernetes API server imposes a number of constraints that only surface at runtime when the client (i.e. the controller) sends the request to the server. + +Examples of these constraints include: + +1. RBAC control: Is your controller configured with the correct Kubernetes RBAC for CRUDing the resources it needs to? +2. Core API semantics + 1. Does your controller respect [Kubernetes naming constraints](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/)? + 2. Does your controller send updates correctly? For example, does it specify the [`metadata.resourceVersion`](https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions) of the objects it reads and writes? +3. Update semantics: Is your controller serializing Go structs to YAML correctly? + +Lastly, Kubernetes controllers are dynamic processes that communicate asynchronously with the kube-apiserver. The actual +state instantiated by the controller and the kube-apiserver are _eventually consistent_, meaning that the actual state +eventually converges to the desired state. It is difficult to faithfully reproduce an eventually consistent environment +without running these asynchronous processes in your test environment. + +## How to write `envtest` integration tests + +### Exercise a Single Controller Per Test + +We recommend writing a single `envtest` integration test for each controller you write. This test should only exercise +a single controller's logic, and should not test the interaction between multiple controllers. + +If your controller interacts with other controllers or other APIs, your test should _mock_ the behavior of those external +systems, similar to classical unit testing methodology. For instance, if your controller creates a Deployment object +and waits until that Deployment becomes ready, you would insert test logic that emulates the Kubernetes control plane +processing the Deployment and setting its status to `Ready` or `Failed`. + +### Setting Up Your Test + +We use the [Ginkgo](https://onsi.github.io/ginkgo/) testing framework to write our tests. Ginkgo is a BDD-style testing +framework that allows us to write tests in a behavior-driven format. + +We use the [Gomega](https://onsi.github.io/gomega/) matcher library to write assertions in our tests. It's especially +useful for ergonomically making asynchronous assertions, required in eventually consistent systems. In most cases you'll +use the `Eventually` matcher to assert that a condition will eventually be true, and the `Consistently` +matcher to assert that a condition will remain true for a period of time. + +To get started, install the `ginkgo` CLI: + +```shell +go install github.com/onsi/ginkgo/v2/ginkgo@latest +``` + +If ginkgo is already installed, make sure you are running ginkgo v2. If you are running v1, upgrade to ginkgo v2 by following [this guide](https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#upgrading-to-ginkgo-20). + +Next, create a new test suite by running: + +```shell +cd /path/to/your/controller/package +ginkgo bootstrap . +``` + +This will create a `*_suite_test.go` file, which will contain the setup for the test suite. +Inside of this file, we scaffold a new `envtest` IT with the following Go code: + +```golang +package mycontroller_test // we recommend using a different package than your controller package so your test exercises only the public interface of the controller package + +import ( + "context" + "path" + "path/filepath" + "testing" + "time" + + "github.com/fgrosse/zaptest" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "github.com/prometheus/client_golang/prometheus" + "github.snooguts.net/reddit/achilles-sdk/pkg/fsm/metrics" + "github.snooguts.net/reddit/achilles-sdk/pkg/logging" + libratelimiter "github.snooguts.net/reddit/achilles-sdk/pkg/ratelimiter" + "github.snooguts.net/reddit/achilles-sdk/pkg/test" + "go.uber.org/zap" + "k8s.io/apimachinery/pkg/runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + ctrllog "sigs.k8s.io/controller-runtime/pkg/log" + ctrlzap "sigs.k8s.io/controller-runtime/pkg/log/zap" + "sigs.k8s.io/controller-runtime/pkg/manager" + + "github.snooguts.net/reddit/mycontroller/internal/controllers/mycontroller" + "github.snooguts.net/reddit/mycontroller/internal/controlplane" + libtest "github.snooguts.net/reddit/mycontroller/internal/test" + ctrlscheme "github.snooguts.net/reddit/mycontroller/pkg/scheme" +) + +var ( + ctx context.Context + scheme *runtime.Scheme + log *zap.SugaredLogger + testEnv *test.TestEnv + testClient client.Client +) + +func TestMyController(t *testing.T) { + RegisterFailHandler(Fail) + + log = zaptest.LoggerWriter(GinkgoWriter).Sugar() // wires up the controller output to the Ginkgo test runner so logs show up in the shell output + ctrllog.SetLogger(ctrlzap.New(ctrlzap.WriteTo(GinkgoWriter), ctrlzap.UseDevMode(true))) + + RunSpecs(t, "mycontroller Suite") +} + +var _ = BeforeSuite(func() { + // default timeouts for "eventually" and "consistently" Ginkgo matchers + SetDefaultEventuallyTimeout(40 * time.Second) + SetDefaultEventuallyPollingInterval(200 * time.Millisecond) + SetDefaultConsistentlyDuration(3 * time.Second) + SetDefaultConsistentlyPollingInterval(200 * time.Millisecond) + + ctx = logging.NewContext(context.Background(), log) + rl := libratelimiter.NewDefaultProviderRateLimiter(libratelimiter.DefaultProviderRPS) + + // add test CRD schemes + scheme = ctrlscheme.MustNewScheme() + + // fetch external CRDs. + // TODO: this is optional. You only need this if your controller makes use of + // other codebases' resources. + externalCRDDirectories, err := test.ExternalCRDDirectoryPaths(map[string][]string{ + "github.com/some/other/repo/apis/v1alpha1": { + path.Join("config", "crd", "bases"), + }, + }, libtest.RootDir()) + Expect(err).ToNot(HaveOccurred()) + + testEnv, err = test.NewEnvTestBuilder(ctx). + WithCRDDirectoryPaths( + append(externalCRDDirectories, + // enumerate all directories containing CRD YAMLs + filepath.Join(libtest.RootDir(), "manifests", "base", "crd", "bases"), + )). + WithScheme(scheme). + WithLog(log.Desugar()). + WithManagerSetupFns( + func(mgr manager.Manager) error { + return mycontroller.SetupController( + ctx, + controlplane.Context{ + Metrics: metrics.MustMakeMetrics(scheme, prometheus.NewRegistry()), + }, + mgr, + rl, + ) // wires up controller being tested to the kube-apiserver + }, + ). + WithKubeConfigFile("./"). // test suite will output a kubeconfig file located in the specified directory + Start() // start invokes the `envtest` binary to start the `kube-apiserver` and `etcd` processes on the host + Expect(err).ToNot(HaveOccurred()) + testClient = testEnv.Client +}) + +// AfterSuite tears down the test environment by terminating the `envtest` processes once the test finishes +// Without this, the host will have orphaned `kube-apiserver` and `etcd` processes that will require manual cleanup +var _ = AfterSuite(func() { + By("tearing down the test environment", func() { + if testEnv != nil { + Expect(testEnv.Stop()).To(Succeed()) + } + }) +}) + +``` + +Now, you can set up envtests in `_test.go` files, just like other Go tests. Here's an example: + +```golang +package mycontroller_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// TODO: Put your env tests here. +var _ = Describe("mycontroller", Ordered, func() { + It("should work", func() { + Eventually(func(g Gomega) { + g.Expect(true).To(Equal(true)) + }).Should(Succeed()) + }) +}) + +``` + +When executing this test via `go test`, the `envtest` binary will start the `kube-apiserver` and `etcd` processes on the host. +The controller being tested will be wired up to the `kube-apiserver` and will be able to interact with the Kubernetes control plane. + +### Writing Your Test + +`envtest` ITs should be expressed in a behavioral manner, which is higher level than how you might express a unit test. +For controller automation, the general structure of a test would be to mimic how an actor would use the custom resource: + +1. Actor (human or program) creates the custom resource +2. Test asserts that the controller processes the custom resource and performs the expected actions: + 1. Creates expected child resources + 2. Updates the custom resource's status + 3. Performs expected actions against other integrated external systems (e.g. issues a request to a REST API) +3. Actor updates the custom resource +4. Test asserts that the controller processes the update and performs the expected actions +5. Actor deletes the custom resource +6. Test asserts that the controller performs expected cleanup actions + +Here is an example of a test that exercises the controller's behavior when a custom resource is created: + +```golang +var _ = Describe("MyController", func() { + Context("when a MyResource is created", func() { + It("should create a MyChildResource", func() { + By("creating a MyResource", func() { + myResource := &mycontroller.MyResource{ + ObjectMeta: metav1.ObjectMeta{ + Name: "my-resource", + Namespace: "default", + }, + Spec: mycontroller.MyResourceSpec{ + // fill in the spec fields + }, + } + Expect(orchClient.Create(ctx, myResource)).To(Succeed()) + }) + + By("waiting for the MyChildResource to be created", func() { + expected := &mycontroller.MyChildResource{ + // fill in expected state + } + + Eventually(func(g Gomega) { + actual := &mycontroller.MyChildResource{} + g.Expect(orchClient.Get(ctx, client.ObjectKey{Name: "my-child-resource", Namespace: "default"}, actual)).To(Succeed()) + g.Expect(actual).To(Equal(expected)) + }).Should(Succeed()) + }) + }) + }) +}) +``` + +Notice that we use an "Eventually" assertion. Because Kubernetes is an eventually consistent system, the test assertion +must poll for the expected state, with a user-configured timeout and polling interval. + +`envtest` ITs can also exercise controller failures modes by emulating conditions under which your controller will error +out. + +To see a full example of a `envtest` IT, refer to the [Achilles Federation controller test](https://github.snooguts.net/reddit/achilles/blob/f5f453b25216fd25f68c22b453345eb6777efbf0/orchestration-controller-manager/internal/controllers/federation/federation_reconciler_test.go). + +### Running Your Test + +To run your test, execute the following command: + +```shell +go test ./path/to/your/controller/package +``` + +### Debugging Your Test + +Controllers are more difficult to debug that single-threaded in-memory tests, both because of the asynchronous nature of +the controller and control plane, and because `envtest` runs the control plane as an out-of-band process (rather than embedded +in the test process). + +That being said, we have some extremely helpful methods for debugging controllers. + +#### Manually Interact with the Envtest Control Plane + +By calling `.WithKubeConfigFile(/path/to/kubeconfig)` in your envtest builder, the test suite will output a kubeconfig file that you can +use to interact with the control plane using `kubectl`. As long as the test suite is still running, the envtest control plane +will be accessible via `kubectl`. + +The recommended workflow is to execute the test in a debugger and pause the test execution at the point where you want to +inspect the control plane. Then, pass the kubeconfig to `kubectl` to inspect the state of the control plane: + +```shell +kubectl --kubeconfig /path/to/kubeconfig get pods -A +``` + +This allows the test author to inspect arbitrary state on the control plane, which is much more efficient than +instrumenting the test logic or controller logic to output all relevant state. + +#### Run the Tests in a Debugger + +Running tests in an interactive debugger allows you to pause the test execution and inspect the _in-memory_ state +of the controller, the control plane, and the test environment. + +If using Jetbrains GoLand, you can set breakpoints in your test code and run the test with the debugger. The test will pause +at the breakpoint, and you can inspect the state of the controller, the control plane, and the test environment. diff --git a/docs/guide.md b/docs/guide.md deleted file mode 100644 index 20a069c..0000000 --- a/docs/guide.md +++ /dev/null @@ -1,3 +0,0 @@ -# Creating a Controller - -This guide has moved to the [Achilles docs website](https://pages.github.snooguts.net/reddit/achilles-docs/dev/sdk/tutorial/). diff --git a/docs/imgs/fsm-flow.png b/docs/imgs/fsm-flow.png deleted file mode 100644 index 254348b..0000000 Binary files a/docs/imgs/fsm-flow.png and /dev/null differ diff --git a/docs/imgs/manager.png b/docs/imgs/manager.png deleted file mode 100644 index a232d87..0000000 Binary files a/docs/imgs/manager.png and /dev/null differ diff --git a/docs/imgs/objs.png b/docs/imgs/objs.png deleted file mode 100644 index 4702c1f..0000000 Binary files a/docs/imgs/objs.png and /dev/null differ diff --git a/docs/imgs/states.png b/docs/imgs/states.png deleted file mode 100644 index 25397ca..0000000 Binary files a/docs/imgs/states.png and /dev/null differ diff --git a/docs/operator/labels.md b/docs/operator/labels.md deleted file mode 100644 index eed280f..0000000 --- a/docs/operator/labels.md +++ /dev/null @@ -1,3 +0,0 @@ -# Labels - -This guide has moved to the [Achilles docs website](https://pages.github.snooguts.net/reddit/achilles-docs/runbooks/labels/). diff --git a/docs/sdk-apply-objects.md b/docs/sdk-apply-objects.md new file mode 100644 index 0000000..1622bc0 --- /dev/null +++ b/docs/sdk-apply-objects.md @@ -0,0 +1,307 @@ +# Applying Objects + +This document describes conventions and patterns around performing updates to Kubernetes resources. + +## OutputSet + +The `achilles-sdk`'s FSM reconciler supplies an [`OutputSet` abstraction](https://github.snooguts.net/reddit/achilles-sdk/blob/340f21a1aa4595651c8627fe754a44082bfab34b/pkg/fsm/types/output.go#L17) +that should satisfy _most_ use resource update use cases. + +The following illustrates the typical pattern: + +```golang +var state = &state{ + Name: "some-state", + Condition: SomeCondition, + Transition: func( + ctx context.Context, + r types.ReconcilerContext, + ctrlCtx controlplane.Context, + object *v1alpha1.SomeObject, + out *sets.OutputSet, + ) (*state, types.Result) { + // perform some logic and build output objects + appProject := &argov1alpha1.AppProject{ + ObjectMeta: ctrl.ObjectMeta{ + Name: AppProjectName(redditWorkload.Name, redditWorkload.Namespace), + Namespace: ctrlCtx.ArgoCDNamespace, + }, + Spec: argov1alpha1.AppProjectSpec{ + SourceRepos: []string{ + oci.HelmProdRegistryURL, + oci.HelmDevRegistryURL, + }, + SourceNamespaces: []string{redditWorkload.Namespace}, + Destinations: []argov1alpha1.ApplicationDestination{ + {Name: "*", Namespace: redditWorkload.Namespace}, + }, + }, + } + + // apply output objects via output set + out.Apply(appProject) + + return nextState, types.DoneResult() + }, +} +``` + +Our hypothetical controller builds an ArgoCD AppProject. The developer then simply passes +them into the `OutputSet` and the achilles-sdk handles applying those resources and their declared state to the Kubernetes +server. + +If the output object does not exist, it is created. If the output object already exists, it is updated to match the +configuration (metadata, spec, and status) constructed by your controller. + +The update only includes fields specified by the in-memory copy of the object. If using the SDK's `OutputSet` or `ClientApplicator.Apply(...)` abstractions, +we strongly recommend that you build the object from scratch rather than mutate a copy read from the server. This ensures +that you only update fields that you intend to update (and don't accidentally send data that you don't intend on having your +logic update). More advanced use cases may require a different "mutation-based" approach, discussed more below. + +Critically, on updates, only the fields specified by your code are updated on the server. **If you omit fields +whose values are pointer types, maps, or slices, they will not be updated on the server.** More detailed discussion on +apply semantics in below. + + +## Detailed Apply Semantics + +### Data Serialization + +There's a surprising amount of complexity in how clients send resource updates to the kube-apiserver. +Controllers are clients of the Kubernetes API Server. They send CRUD requests to the server to control Kubernetes +configuration. `Create` and `Update` request bodies contain the desired object state. These request bodies +are serialized from the underlying in-memory type to YAML (or JSON) before being sent to the server. + +For `Patch` update requests, only the fields that appear in the request body (which is determined by the underyling +implementation's serialization behavior) are updated. + +As discussed below, the Achilles SDK models updates in a manner that facilitates multiple actors gracefully managing +mutually exclusive fields on the same objects. This is achieved by controllers selectively serializing only the fields +they manage, and omitting fields they do not manage. + +### Achilles SDK Conventions + +Achilles controllers (and more generally, all Kubernetes controllers) should honor the following assumptions. Violation +of these assumptions leads to interoperability issues and can cause controllers to overwrite each other's updates. + +**Assumption 1: One Owner Per Field** + +For all Kubernetes objects that our controllers read and/or write, we must make the assumption that +every **field** in `metadata`, `spec`, and `status` has a single owner. "Owner" in this context refers to a program +or human that mutates the field. This constraint is not bespoke to the Achilles SDK; it is a general recommendation +for all Kubernetes controllers. The kube-apiserver can even enforce this constraint through a feature called +["server-side-apply"](https://kubernetes.io/docs/reference/using-api/server-side-apply/#field-management). We don't +currently use SSA in the Achilles SDK, but it is a feature that can be integrated in the future. For now, the onus of +following this convention rests on the controllers that manage a given resource. + +Violation of this assumption leads to conflicting updates where two owners fight over the same field, which can lead to +a malfunctioning system. + +**Assumption 2: For objects with multiple owners, all fields are pointer types, maps, or slices with the `omitempty` JSON tag** + +First, a quick primer on how Go serializes structs to JSON or YAML using the `encoding/json` package: + +1. Fields that are scalar types (int, string, bool, etc.) are always serialized to their JSON or YAML representation + i. There is no way to omit these fields from the serialized output. +2. Fields that are pointer types, maps, or slices and marked with the `omitempty` JSON tag[^1] are serialized to their JSON or YAML representation _only if the field is non-nil_ + i. If the field is nil, it is omitted from the serialized output. + ii. If the field is empty but non-nil (e.g. `[]` for slices, `{}` for maps and structs), it is serialized to its empty JSON representation (`[]` and `{}` respectively). + +Given these serialization mechanics, for objects that have multiple owners acting on mutually exclusive fields, we must ensure that +all `spec` and `status` fields are types that allow omission when serializing from Go types (pointer types, maps, or slices). +This means that actors can send updates while _omitting_ fields that they do not own, thus preventing collisions +with other owners. + +Following these two assumptions, we can optimistically apply all object updates without utilizing Kubernetes' [resource version](https://github.snooguts.net/reddit/reddit-helm-charts#versioning) +because there is no risk that any actor's update will conflict with or overwrite that of a different actor's. + +We also update all objects using [JSON merge patch semantics](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment), +([RFC here](https://tools.ietf.org/html/rfc7386)), which offers the simplest mental model for merging an update +against existing object state. + +**Deleting Fields** + +**zero value equivalent to field omission:** + +This applies in APIs where a field's zero value (which depends on the type) is semantically equivalent to the field +being absent. In these cases, setting the field to the zero value is equivalent to deleting the field. + +Using JSON merge patch semantics, deleting a field requires that the request body contain the field with an empty value +(`0` for numerics, `""` for strings, `false` for bools, `[]` for slices and `{}` for maps or structs). + +Assuming your field is a pointer type, map, or slice with the `omitempty` JSON tag, you can delete a field by setting it +to the empty but non-nil value for the given type. + +**zero value distinct from field omission:** + +In APIs where a field's zero value is not semantically equivalent to the field being absent, deleting the field requires +using an `Update` operation (rather than a `Patch` operation) to overwrite the entire object (same approach +as described below under "Deleting key-value pairs from map types"). + +## Advanced Apply Patterns + +The two assumptions above do not always hold, especially when using 3rd party CRDs whose types you do not control, +that are implemented by controllers whose behavior you don't control. + +Use the following workarounds when presented with these convention departures. + +**Kubernetes Resource Lock** + +Kubernetes CRDs are supplied with a resource lock, tracked by the `metadata.resourceVersion` field. +This field is an integer that represents the latest version of the resource that the server has persisted. +If an update or patch request specifies this field, the kube-apiserver will reject the request if the request's field does not match +the server's field. This guarantees that the client is operating on the latest version of the object. +Full details on the design and implementation [can be found here](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency). + +When would you want to use this feature? + +The backing Kubernetes client cache supplied by controller-runtime [can serve stale (i.e. outdated) data](https://github.com/kubernetes-sigs/controller-runtime/blob/main/FAQ.md#q-my-cache-might-be-stale-if-i-read-from-a-cache-how-should-i-deal-with-that). +If your logic reads data from the kube-apiserver and then uses that data to update the object, you may inadvertently +update the object with stale data. If sending stale data has negative side effects, you should use the resource lock +to guarantee that stale updates are never sent. + +For cases where an external controller is managing some fields in the `spec`, whose types _must_ be serialized (meaning that +you cannot avoid having your controller send some value for fields it doesn't manage because you don't control +the API struct definition and because the field is not a pointer type, map, or slice with the `omitempty` JSON tag), +the workaround is for your controller to perform a "read-modify-write" operation while using the Kubernetes resource lock, +which consists of: + +1. read the object from the server +2. modify only the fields that your controller manages +3. write the object back to the server while using the resource lock + +Usage of the resource lock (i.e. sending the update request with `metadata.resourceVersion` populated) ensures that +the Kubernetes server will reject the update if it's operating on a version of the object that is out of date. +This guarantees that your controller will not overwrite data managed by other controllers. + +To use the resource lock, do the following: + +```golang +import "github.snooguts.net/reddit/achilles-sdk/pkg/io" + +out.Apply(obj, io.WithResourceLock()) +``` + +**Deleting key-value pairs from map types** + +Since we're using JSON merge patch semantics by default, the only way to delete a KV pair from fields whose Go type is a +map is to overwrite the entire map.[^2] + +To perform a full object update, supply to `AsUpdate()` apply option like so: + + +```golang +import "github.snooguts.net/reddit/achilles-sdk/pkg/io" + +out.Apply(obj, io.AsUpdate()) +``` + +If this is a 3rd party CRD, you will likely need to pair the usage of `AsUpdate()` with `WithResourceLock()` to avoid +overwriting fields your controller does not manage. + +**Custom Management of Owner References** + +By default, the FSM reconciler adds an owner reference to all managed resources that links back to the reconciled object. +This enables Kubernetes-native garbage collection, whereby all managed resources will be deleted when the reconciled object +gets deleted. This default behavior makes sense in _most_ controller use cases. + +If your controller is intentionally managing owner references, you must disable this feature by using the `io.WithoutOwnerRefs()` +([link](https://github.snooguts.net/reddit/achilles-sdk/blob/dea76bcf6143aebce5ba0763f99c9f282b5b3415/pkg/io/options.go#L44)) +apply option. + +### Mutation Based Updates + +The Achilles SDK's `ClientApplicator` and `OutputSet` assume a "declarative-style" pattern where the update data is +built from scratch rather than mutated from a copy read from the server. However, there is a scenario where a mutation-based +approach is desirable—you want to minimize the performance cost of your controller updating a field with "stale" (i.e. outdated) data. + +The backing Kubernetes client cache supplied by controller-runtime [can serve stale (i.e. outdated) data](https://github.com/kubernetes-sigs/controller-runtime/blob/main/FAQ.md#q-my-cache-might-be-stale-if-i-read-from-a-cache-how-should-i-deal-with-that), +which can lead to extra rounds of reconciliation before your controller converges the managed object into a steady (i.e. unchanging) state. + +Ideally your controller must implement idempotent reconciliations and should be tolerant of actuation even if they are +caused by stale data. But if performance becomes a concern, or you wish to optimize your controller to reduce the number +of reconciliations, you can use a mutation-based approach. + + +```golang + // Create or Update the deployment default/foo + deploy := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} + + // NOTE: there's an analogous method for CreateOrPatch + op, err := controllerutil.CreateOrUpdate(ctx, r.Client, deploy, func() error { + // Deployment selector is immutable so we set this value only if + // a new object is going to be created + if deploy.ObjectMeta.CreationTimestamp.IsZero() { + deploy.Spec.Selector = &metav1.LabelSelector{ + MatchLabels: map[string]string{"foo": "bar"}, + } + } + + // update the Deployment pod template + deploy.Spec.Template = corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "foo": "bar", + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "busybox", + Image: "busybox", + }, + }, + }, + } + + return nil + }) + + if err != nil { + log.Error(err, "Deployment reconcile failed") + } else { + log.Info("Deployment successfully reconciled", "operation", op) + } +``` + +The implementation of `CreateOrUpdate` and `CreateOrPatch` is as follows: + +1. It performs a read from the server (fronted by the cache, so it can still be stale) +2. It executes the supplied mutation function to mutate the object +3. It sends the update or patch + +This implementation reduces the chance that stale data is sent to the server because of the initial read. But importantly, +it does not guarantee that stale data is not written. To have this guarantee, you must supply the Kubernetes resource lock +when sending your update (discussed above under "Kubernetes Resource Lock"). + +## FAQ + +1. What happens if my controller sends an update/patch request with stale data? + +Consider the following timeline: + +``` +t0: Controller A reads object from server +t1: Actor B updates object +t2: Controller A updates object with stale data +``` + +If the resource lock is used, the kube-apiserver will reject the update at `t2` because the resource version in the +update request does not match the server's resource version. + +If the resource lock is not used, the kube-apiserver will accept the update at `t2` and the object will be updated with +stale data. If Actor B responds by updating the object again, the object will be updated with Actor B's data, which will +in turn actuate Controller A. This cycle may repeat until Controller A eventually reads non-stale data and thus doesn't +overwrite Actor B. Cycles of this kind are essentially livelocks, which should be mitigated by using the resource lock. + +If Actor B does not react by updating the object again, then the object will remain in the state that Controller A set it to, +i.e. with stale data. There is no livelock in this scenario but the object is in an incorrect state. This can be mitigated +by using the resource lock. + +## References + +1. The full list of apply options lives under [`/pkg/io/options.go`](https://github.snooguts.net/reddit/achilles-sdk/blob/0a9d9df29d5201b0f3d689108547c3a97d819d82/pkg/io/options.go) +2. The client abstraction lives under [`/pkg/io/applicator.go`](https://github.snooguts.net/reddit/achilles-sdk/blob/0a9d9df29d5201b0f3d689108547c3a97d819d82/pkg/io/applicator.go) + +[^1]: Read more about [Go serialization here](https://pkg.go.dev/encoding/json#Marshal) +[^2]: We could theoretically implement or use a custom Go JSON marshaller that can output `key: null` to signal deletion of fields. diff --git a/docs/sdk-claim-claimed-reconcilers.md b/docs/sdk-claim-claimed-reconcilers.md new file mode 100644 index 0000000..b732ef0 --- /dev/null +++ b/docs/sdk-claim-claimed-reconcilers.md @@ -0,0 +1,35 @@ +# FSM Claim/Claimed Reconciler + +This guide walks you through the claim/claimed reconciler pattern available in the `achilles-sdk` and when to use it. + +## Background + +The `achilles-sdk` in addition to the base [FSM reconciler]({{< ref "dev/sdk/sdk-fsm-reconciler" >}}) also provides a claim/claimed reconciler pattern. +This pattern uses the same FSM semantics from a developer perspective. This pattern is built around two types of resources: +1. **Claim**: A claim resource is created by an end user or client claiming access to a resource. An example of this type of resource is [`RedisClusterClaim`](https://github.snooguts.net/reddit/achilles-redis-controllers/blob/main/api/storage/v1alpha1/redisclusterclaim_type.go) + which claims access to a `RedisCluster` instance. +2. **Claimed**: A claimed resource captures the actual instance of given object. An example of this type of resource is [`RedisCluster`](https://github.snooguts.net/reddit/achilles-redis-controllers/blob/main/api/storage/v1alpha1/rediscluster_type.go) + which captures the actual resources running in AWS. + +**The `claim` object is namespaced whereas the `claimed` object is cluster-scoped**. + + +## When to use the claim/claimed reconciler pattern? + +It's desirable for the `claim` object to be namespace scoped for one of the following reasons: + +1. The `claim` object is end-user facing and you want to ensure that users cannot mutate objects owned by other users by leveraging native Kubernetes RBAC, using the namespace as an isolation unit. +2. Your system organizes resources by namespace. For instance, all configuration pertaining to entity Y belongs in namespace Y. + 1. For example, Compute's `orchestration-controller-manager` controller provisions a namespace for each RedditCluster and places all resources pertaining to that RedditCluster in its own namespace. This allows easy inspection and cleanup of all state pertaining to a given RedditCluster. +3. You want to expose a namespace-scoped CR (the `claim` object) whose implementation (the `claimed` object) requires cluster-scoped resources. + 1. For instance, Storage's namespace-scoped `RedisClusterClaim` CR is implemented using cluster-scoped Crossplane managed resources. + 1. The `claimed` object being cluster scoped allows the `claim` object to manage and "own" both cluster-scoped resources like those exposed by Crossplane + and namespaced resources. Kubernetes explicitly disallows cross-namespace owner references ([ref](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/)). +4. You want to decouple the "request" (the `claim` object) from the "fulfillment" of that request (the `claimed` object), similar to Kubernetes' PVC and PV pattern. + 1. Note (April 17th, 2024) that the `achilles-sdk` hasn't implemented this decoupling pattern yet because no concrete use cases have required it. + +The `achilles-sdk` creates two independent reconcilers for the `claim` and `claimed` objects. +The `claim` reconciler is entirely managed by the sdk and does not require any work on the user's part ([ref](https://github.snooguts.net/reddit/achilles-sdk/blob/main/pkg/fsm/internal/reconciler_claim.go#L43)). +The `claim` reconciler is responsible for creating the `claimed` object if it does not exist and cascading a delete call when the `claim` object is deleted. +The developer is responsible for implementing the `claimed` reconciler using the exposed [FSM semantics]({{< ref "dev/sdk/sdk-fsm-reconciler" >}}). +As an example take a look at the [ASGRotator Controller](https://github.snooguts.net/reddit/achilles/blob/master/orchestration-controller-manager/internal/controllers/cloud-resources/asgrotator/controller.go). diff --git a/docs/sdk-finalizers.md b/docs/sdk-finalizers.md new file mode 100644 index 0000000..7cab77b --- /dev/null +++ b/docs/sdk-finalizers.md @@ -0,0 +1,39 @@ +# What are Finalizers? + +Finalizers is a feature in Kubernetes that prevent the deletion of an object until some conditions are met. Finalizers +alert controllers when an object is being deleted, allowing them to perform cleanup tasks before the object is removed. +More information about how finalizers work is available in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/). + +## Finalizers in the Achilles SDK + +The Achilles SDK has an optional feature to provide a `finalizerState` when creating a new controller. If a `finalizerState` is provided +the sdk automatically manages adding and removing the finalizer on the object being reconciled. The finalizer is added/updated on every reconcile +as long as the object is not being deleted. Once the object is being deleted (i.e. `metadata.DeletionTimestamp` is set), the sdk will call the +`finalizerState` provided by the controller. The finalizer will only be removed once the `finalizerState` returns a `types.DoneResult()`. +Until the finalizer is removed, the object will not be deleted and the sdk will call the `finalizerState` on every reconcile. + +Some examples of what the finalizerState can be used for are: +1. Cleaning up state in remote systems like Vault (e.g. deleting all managed Vault entities) +2. Deleting child Kubernetes objects in a particular order (i.e. deleting Crossplane InstanceProfiles before Roles due to a Crossplane limitation) + +Example to add a finalizer to an achilles controller: +```go +builder := fsm.NewBuilder( + ... +).WithFinalizerState(finalizerState) + +var finalizerState = &state{ + Name: "finalizer", + Condition: apicommon.Deleting(), + Transition: func(ctx context.Context, r types.ReconcilerContext, ctrlCtx controlplane.Context, secret *appv1alpha1.RedditSecret, out *types.OutputSet) (*state, types.Result) { + // cleanup logic + return nil, types.DoneResult() + }, +} + +``` +### Finalizer for claim/claimed reconciler + +For the claim/claimed reconciler, the sdk automatically adds a finalizer to the `claim` object. +This is to ensure the sdk is able to delete the `claimed` object when the `claim` object is deleted ([ref](https://github.snooguts.net/reddit/achilles-sdk/blob/main/pkg/fsm/internal/reconciler_claim.go)). +The behaviour of the finalizer on the `claimed` object is controlled by the `finalizerState` provided to the controller as explained above diff --git a/docs/sdk-fsm-reconciler.md b/docs/sdk-fsm-reconciler.md new file mode 100644 index 0000000..9b3aafa --- /dev/null +++ b/docs/sdk-fsm-reconciler.md @@ -0,0 +1,132 @@ +# FSM Reconciler + +This guide walks you through the FSM (finite state machine) controller framework, both the +programming mental model and common controller patterns with code examples. + +The goal of the FSM framework is to allow software engineers without extensive +experience building Kubernetes controllers to build correct and conventional declarative APIs, abstracting away +internal controller mechanics to allow the developer to focus on automation business logic. + +## Background + +For a brief conceptual overview of control-loop theory as it pertains to +Kubernetes, [see this document](https://kubernetes.io/docs/concepts/architecture/controller/). + +The FSM framework is built on top of [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime/), a +widely used SDK for building Kubernetes controllers. + +Controller-runtime reduces complexity of building controllers through simplifying opinions, the most important ones +being: + +- a controller is composed of two main parts, the trigger conditions and the reconciliation logic + - trigger conditions specify _when_ a control loop actuates and _which_ object gets operated upon + - reconciliation logic specifies _what_ operations to perform for the object being reconciled (henceforth referred + to as "parent object") +- each logical control loop is responsible for managing a single resource type (i.e. GVK) +- controller triggers are level-based, not edge-based + +Controller-runtime describe some of their opinions and +conventions [here](https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/doc.go). + +The FSM framework extends controller-runtime with additional opinions and structure that further simplify the process +of building controllers. These will be discussed below. + +## Finite State Machine Model + +The FSM framework provides additional structure over +controller-runtime's [monolithic "Reconcile()" method](https://github.com/kubernetes-sigs/controller-runtime/blob/dca0be70fd22d5200f37d986ec83450a80295e59/pkg/reconcile/reconcile.go#L93) +by modeling reconciliation as a finite state machine. Furthermore, the finite state machine has the additional +constraint +that **it must be a directed acyclic graph**. Cycles _within a single reconciliation invocation_ are detected and reported as runtime errors. Crucially, this does not prevent the controller from continuously reconciling on a periodic interval (especially for use cases that require polling some upstream system). + +Additionally, each reconciliation starts from the FSM's initial state rather than starting from the last reached state. +This implies the following: + +- all paths through the FSM graph must be idempotent +- all FSM states must be reachable by observing persisted state available to the controller + +These design constraints ensure controller correctness by forcing the developer to write their reconciliation logic in +a manner that is both idempotent and dependent on externally persisted state, rather than state internal to the +controller, which can easily diverge from the actual state of the world. The resulting control loop logic is resilient +to controller restarts and any runtime errors. + +## States + +Every state in the FSM maps to a [status condition](https://maelvls.dev/kubernetes-conditions/) on the parent object, +which is defined by the +developer [for each state](https://github.snooguts.net/reddit/achilles/blob/c0ddc4dadb6a7613552598da773bd77b80b15c0c/lib/fsm/types/transitions.go#L52) +. If the state completes successfully, the status condition's `Status` field will be set to true. Otherwise, in the case +of a +requeue result or error, the status field will be set to false. + +The tracking of states via status condition adheres to Kubernetes API best practices by providing an externally +observable +signal to dependencies of the API. Other actors (programs or humans) can treat the status conditions of FSM-backed APIs +as an authoritative source of truth on its status. + +## Transitioning Between States + +Each state defines the next state to transition to the current state completes successfully. The next state can vary +based on logical conditions, allowing the expression of branching paths. + +Each state +defines [a result type](https://github.snooguts.net/reddit/achilles/blob/c0ddc4dadb6a7613552598da773bd77b80b15c0c/lib/fsm/types/transitions.go#L118-L165) +. +Broadly speaking, there are three types of results: + +1. **done**—the state has finished successfully, the reconciler can transition to the next state or, in the case of a + terminal state, simply complete +2. **requeue**—instructs the reconciler to trigger again after a user-specified amount of time. This is used in cases + where + a controller is waiting for an external condition to be fulfilled. +3. **error**—the reconciler logs an error message and will retrigger, the delay of which is subject to exponential + backoff + +A requeue result is typically used over an error result when the external condition is expected to be eventually +consistent, and +thus its retrigger should not be subject to exponential delay. + +An error result is used when an external condition is not expected to be false. + +## Writing and Updating Managed Resources + +The majority of controllers involve creating and updating Kubernetes objects, whether they are CRDs or native resources. +The FSM framework provides +an [output object set abstraction](https://github.snooguts.net/reddit/achilles/blob/c0ddc4dadb6a7613552598da773bd77b80b15c0c/lib/fsm/types/transitions.go#L41) +for ensuring outputs. It provides the following functionality: + +- output objects are tracked via the parent object's status +- output objects have their owner references updated with the parent object + - this provides free garbage collection (i.e. the child objects will be deleted if the parent object is deleted) via + native Kubernetes garbage collection + +## Finalizer States + +[Kubernetes finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/) can be used +to ensure the execution of logic that triggers when an object is deleted. Objects with a finalizer will remain in a +terminating state, but not get deleted from Kubernetes state, until the finalizer is removed. + +The FSM provides a way to add a separate FSM triggered upon deletion of +objects, [see this example](https://github.snooguts.net/reddit/achilles/blob/36c3aa3bde5a2590f5d914918a8cefdf1ef953a7/lib/fsm/test/test_fsm_reconciler.go#L39) +. The FSM automatically manages the attachment and removal of the finalizer. The finalizer will only be removed if the +finalizer FSM terminates successfully. + +## Trigger Conditions + +The FSM exposes the same trigger conditions as controller-runtime. + +When building a new controller, +use `.Manages` ([source](https://github.snooguts.net/reddit/achilles/blob/e8f58f6d9a66ab799da21ae9eb1cdc373e56e2d2/lib/fsm/controller.go#L76)) +to specify the type of object that is being managed by the controller. Each controller can only manage a single object +type. + +The FSM automatically wires up triggers for all [managed resources](##Writing and Updating Managed Resources). + +Additional trigger conditions can be wired up for arbitrary events via +the [`.Watches` method](https://github.snooguts.net/reddit/achilles/blob/e8f58f6d9a66ab799da21ae9eb1cdc373e56e2d2/lib/fsm/controller.go#L109) +. + +## Example FSM Controllers + +See [this simple example](https://github.snooguts.net/reddit/achilles/blob/1499fc7d792c9d717572bea58e85ccb597245bb3/lib/fsm/test/test_fsm_reconciler.go) +for reference on how to implement an FSM controller. diff --git a/docs/sdk-metrics.md b/docs/sdk-metrics.md new file mode 100644 index 0000000..cbe70cc --- /dev/null +++ b/docs/sdk-metrics.md @@ -0,0 +1,122 @@ +# Metrics and Monitoring + +This guide describes the metrics provided by the SDK and how to use them to monitor the health and performance of your +system. + +## Controller-runtime Metrics + +The Achilles SDK +integrates [controller-runtime metrics](https://github.com/kubernetes-sigs/controller-runtime/blob/1ed345090869edc4bd94fe220386cb7fa5df745f/pkg/internal/controller/metrics/metrics.go). +Controller-runtime metrics provide foundational metrics for understanding the performance and health of your controller. + +These metrics can be viewed in +the ["Controller Runtime" Grafana dashboard](https://grafana.kubernetes.ue1.snooguts.net/d/Md5CPB44k/controller-runtime?orgId=1&refresh=30s&var-cluster=orch-1&var-prometheus=monitoring%2Finfrared-system&var-controller=All&var-webhook=All&from=1721981744523&to=1722003344523). + +## SDK Metrics + +The Achilles SDK provides additional metrics that leverage SDK conventions and structures to provide more detailed +insights into the health and performance of your controller. + +These metrics are displayed in the following Grafana dashboards: + +1. [Achilles Reconciler Metrics](https://grafana.kubernetes.ue1.snooguts.net/d/p_-RmaUVk/achilles-reconciler-metrics?orgId=1&from=1721960667563&to=1722003867564) + 1. Provides a high level overview of your controller and its custom resources +2. [Achilles Reconciler Detailed Metrics](https://grafana.kubernetes.ue1.snooguts.net/d/0gaENrwVk/achilles-reconciler-detailed-metrics?orgId=1&from=1721982323248&to=1722003923249) + 1. Provides a detailed overview of particular reconcile loops of your controller. + +### **`achilles_resource_readiness`** + +This metric is a gauge that maps to an Achilles object's status conditions. By default, the SDK instruments metrics for the +status condition of type "Ready". Users can instrument additional status conditions by declaring the following when +building their reconciler: + +```golang +WithReconcilerOptions( + fsmtypes.ReconcilerOptions[v1alpha1.Foobar, *v1alpha1.Foobar]{ + MetricsOptions: fsmtypes.MetricsOptions{ + ConditionTypes: []api.ConditionType{ + // user specifies custom status condition types here + MyCustomStatusCondition.Type, + }, + }, + }, +) +``` + +This metric is emitted for each Achilles object, allowing operators to monitor the readiness of each API object +in their system. + +The metric has the following labels. +```c +achilles_resource_readiness{ + group="app.infrared.reddit.com", // the Kubernetes group of the resource + version="v1alpha1", // the Kubernetes version of the resource + kind="FederatedRedditNamespace", // the Kubernetes kind of the resource + name="demo-namespace-1", // the name of the resource + namespace="", // the namespace of the resource (empty for cluster-scoped CRDs) + status="True", // the status condition's "Status" field + type="Ready", // the status condition's "Type" field +} 1 // value of 1 means a status condition of the labelled status and type exists, 0 if it doesn't exist +``` + +### **`achilles_trigger`** + +This metric is a counter that provides insight into the events triggering your controller's reconcilers. It allows operators to reason +about the frequency and types of events that are causing the controller to reconcile. This is typically examined when +looking to reduce the frequency of reconciliations or understand suspected extraneous reconciliations. + +For a given reconciler, it is emitted for each (triggering object, event type) pair. + +The "type" label indicates the type of triggering object: + +1. **"self"** triggers happen by nature of controller-runtime's reconciler model, where any event on the reconciled object +triggers a reconciliation. +2. **"relative"** triggers occur through events on related objects. Related object triggers are wired up +using the `.Watches()` [builder method](https://github.snooguts.net/reddit/achilles-sdk/blob/bd2f3522d9e38b513f3a0f206f9bb9b0532c8b50/pkg/fsm/controller.go#L136). +3. **"child"** triggers occur through events on managed child objects (objects whose `meta.ownerRef` refers to the reconciled object). Child triggers +are wired up using the `.Manages()` [builder method](https://github.snooguts.net/reddit/achilles-sdk/blob/bd2f3522d9e38b513f3a0f206f9bb9b0532c8b50/pkg/fsm/controller.go#L96) + +```c +achilles_trigger{ + controller="ASGRotatorClaim", // the name of the reconciler + group="component.infrared.reddit.com", // the Kubernetes group of the triggering object + version="v1alpha1", // the Kubernetes version of the triggering object + kind="ASGRotator", // the Kubernetes kind of the triggering object + event="create", // the event type, one of "create", "update", "delete" + reqName="asg-rotator-managed", // the name of the triggering object + reqNamespace="dpwfeni-test-usva-aws-1", // the namespace of the triggering object (empty for cluster-scoped objects) + type="relative", // the trigger type, one of "relative", "self", or "child" +} 13 // the number of observed trigger events +``` + +### **`achilles_object_suspended`** + +This metric is a gauge that indicates whether an object is suspended. It is emitted for each reconciled object. +This metric is useful for alerting on any long-suspended objects. + +```c +achilles_object_suspended{ + group="app.infrared.reddit.com", // the Kubernetes group of the reconciled object + version="v1alpha1", // the Kubernetes version of the reconciled object + kind="FederatedRedditNamespace", // the Kubernetes kind of the reconciled object + name="achilles-test-apps", // the name of the reconciled object + namespace="", // the namespace of the reconciled object (empty for cluster-scoped objects) +} 0 // value of 1 means the object is suspended, 0 if it is not +``` + +### **`achilles_state_duration_seconds`** + +This metric is a histogram that provides performance insight into the duration of each state in the FSM. It is emitted +for each (reconciler, state) pair. + +```c +achilles_state_duration_seconds_bucket{ + group="app.infrared.reddit.com", // the Kubernetes group of the reconciled object + version="v1alpha1", // the Kubernetes version of the reconciled object + kind="FederatedRedditNamespace", // the Kubernetes kind of the reconciled object + state="check-federation-refs", // the name of the FSM state + le="0.99", // the percentile of the histogram distribution +} 183 // the duration in milliseconds +``` + +The average durations are graphed over time in the [Achilles Detailed Reconciler Metrics dashboard](https://grafana.kubernetes.ue1.snooguts.net/d/0gaENrwVk/achilles-reconciler-detailed-metrics?orgId=1&from=1721983467755&to=1722005067755).