[CECO-1763] Controller for generic custom resource #1534

tbavelier · 2024-11-21T12:52:14Z

What does this PR do?

Introduces a generic controller relying on Datadog Go client to handle CRUD operations on multiple resources. Initial support includes notebooks and synthetics (api/browser tests).
Examples:

apiVersion: datadoghq.com/v1alpha1
kind: DatadogGenericCR
metadata:
  name: api-test
spec:
    type: synthetics_api_test
    jsonSpec: |-
        {
           [...]
        }

apiVersion: datadoghq.com/v1alpha1
kind: DatadogGenericCR
metadata:
  name: notebook
spec:
    type: notebook
    jsonSpec: |-
        {
           [...]
        }

Additional Notes

Next steps:

Document fully-fledged examples
Document onboarding strategy (e.g. export current resource from the API, create the associated YAML, deploy them)
Document internally how to add new resources
UI button to export to manifest
Add option to start the controller in the Helm chart

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

Agent: vX.Y.Z
Cluster Agent: vX.Y.Z

Describe your test plan

Include -datadogGenericCREnabled=true in args for the manager container to enable the controller
Deploy examples
Update their jsonSpec
Ensure the changes are reflected in the UI
Delete the resources
Ensure they are removed from the UI

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label

codecov-commenter · 2024-11-21T13:33:12Z

Codecov Report

Attention: Patch coverage is 40.28926% with 289 lines in your changes missing coverage. Please review.

Project coverage is 48.75%. Comparing base (0ca2c91) to head (53fa6d2).

Files with missing lines	Patch %	Lines
internal/controller/datadoggenericcr/controller.go	38.97%	74 Missing and 9 partials ⚠️
internal/controller/datadoggenericcr/utils.go	57.05%	67 Missing ⚠️
internal/controller/datadoggenericcr/synthetics.go	0.00%	48 Missing ⚠️
internal/controller/datadoggenericcr/notebooks.go	17.02%	39 Missing ⚠️
internal/controller/datadoggenericcr_controller.go	0.00%	15 Missing ⚠️
pkg/datadogclient/client.go	0.00%	14 Missing ⚠️
internal/controller/setup.go	31.25%	11 Missing ⚠️
internal/controller/datadoggenericcr/finalizer.go	74.35%	7 Missing and 3 partials ⚠️
cmd/main.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1534      +/-   ##
==========================================
- Coverage   48.94%   48.75%   -0.20%     
==========================================
  Files         227      235       +8     
  Lines       20636    21120     +484     
==========================================
+ Hits        10101    10296     +195     
- Misses      10010    10287     +277     
- Partials      525      537      +12

Flag	Coverage Δ
unittests	`48.75% <40.28%> (-0.20%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
api/datadoghq/v1alpha1/datadoggenericcr_types.go	`100.00% <100.00%> (ø)`
.../datadoghq/v1alpha1/datadoggenericcr_validation.go	`100.00% <100.00%> (ø)`
cmd/main.go	`0.00% <0.00%> (ø)`
internal/controller/datadoggenericcr/finalizer.go	`74.35% <74.35%> (ø)`
internal/controller/setup.go	`49.33% <31.25%> (-2.16%)`	⬇️
pkg/datadogclient/client.go	`0.00% <0.00%> (ø)`
internal/controller/datadoggenericcr_controller.go	`0.00% <0.00%> (ø)`
internal/controller/datadoggenericcr/notebooks.go	`17.02% <17.02%> (ø)`
internal/controller/datadoggenericcr/synthetics.go	`0.00% <0.00%> (ø)`
internal/controller/datadoggenericcr/utils.go	`57.05% <57.05%> (ø)`
... and 1 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ca2c91...53fa6d2. Read the comment docs.

levan-m · 2025-01-03T23:23:11Z

api/datadoghq/v1alpha1/datadoggenericcr_types.go

+// +k8s:openapi-gen=true
+type DatadogGenericCRSpec struct {
+	// Type is the type of the API object
+	// TODO: Add validation for the type (enum)


Suggested change

// TODO: Add validation for the type (enum)

// +kubebuilder:validation:Enum=notebook;synthetics_api_test;synthetics_browser_test

Will need to regenerate CRD. Worth adding a comment at the end of the constants about updating this line for new types.

levan-m · 2025-01-03T23:24:46Z

api/datadoghq/v1alpha1/datadoggenericcr_types.go

+// +kubebuilder:printcolumn:name="age",type="date",JSONPath=".metadata.creationTimestamp"
+// +k8s:openapi-gen=true
+// +genclient
+type DatadogGenericCR struct {


I'm not sure about this naming. CR is kubernetes-specific concept while this represents a generic resource in Datadog. Maybe DatadogResource, DatadogApiResource, DatadogGenericResource is a better fit?

levan-m · 2025-01-03T23:28:46Z

internal/controller/datadoggenericcr/utils.go

+type operation string
+
+const (
+	// mockSubresource is used to mock the subresource in tests


Any way to avoid this in the prod code?

levan-m · 2025-01-03T23:35:15Z

internal/controller/datadoggenericcr_controller.go

+//+kubebuilder:rbac:groups=datadoghq.com,resources=datadoggenericcrs/status,verbs=get;update;patch
+//+kubebuilder:rbac:groups=datadoghq.com,resources=datadoggenericcrs/finalizers,verbs=update
+
+// - https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/reconcile


nit, redundant comment

levan-m · 2025-01-04T02:16:37Z

Left some suggestions/nits. Overall looks good but handling of multiple resource APIs (in utils.go) can be refactored to more maintainable code.

tbavelier added 7 commits November 18, 2024 16:29

working browser tests and notebooks

d5a5a19

clean up a bit

6a9ac13

Use helper function

0be80c1

Start refactoring - not tested yet

f1cb11c

spec -> jsonSpec refactor

a07ba34

refactor & validation - working version (browser, notebook)

1afc052

revert devcontainer name + crd->cr reference

d76fa13

tbavelier added the do-not-merge label Nov 21, 2024

tbavelier added this to the v1.12.0 milestone Nov 21, 2024

remove redundant conversion

881b7b1

tbavelier added 13 commits November 25, 2024 13:43

Add validation that jsonSpec is not empty + unit test validation

0c72641

kustomize datadoggenericcr

aef1276

unit test finalizer - need to mock API to go further

40e3f98

finalizer test

5d660f3

utils unit test

5a5373c

controller unit test

a91d0f1

notebook utils unit tedt

c990340

more unit tests and support synthetics_api_test

9e9ad42

re-order alphabetically

3d0864b

resolve conflict before merging main

2677278

resolve kustomization conflict

7ae3827

Merge branch 'main' into tbavelier/generic_crd_controller

e63dc5d

empty

1f9164f

tbavelier added enhancement New feature or request and removed do-not-merge labels Nov 29, 2024

tbavelier marked this pull request as ready for review November 29, 2024 15:49

tbavelier requested a review from a team as a code owner November 29, 2024 15:49

Merge branch 'main' into tbavelier/generic_crd_controller

53fa6d2

levan-m approved these changes Jan 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CECO-1763] Controller for generic custom resource #1534

[CECO-1763] Controller for generic custom resource #1534

tbavelier commented Nov 21, 2024 •

edited

Loading

codecov-commenter commented Nov 21, 2024 •

edited

Loading

levan-m Jan 3, 2025

levan-m Jan 3, 2025

levan-m Jan 3, 2025

levan-m Jan 3, 2025

levan-m commented Jan 4, 2025

	// TODO: Add validation for the type (enum)
	// +kubebuilder:validation:Enum=notebook;synthetics_api_test;synthetics_browser_test

[CECO-1763] Controller for generic custom resource #1534

Are you sure you want to change the base?

[CECO-1763] Controller for generic custom resource #1534

Conversation

tbavelier commented Nov 21, 2024 • edited Loading

What does this PR do?

Additional Notes

Minimum Agent Versions

Describe your test plan

Checklist

codecov-commenter commented Nov 21, 2024 • edited Loading

Codecov Report

levan-m Jan 3, 2025

Choose a reason for hiding this comment

levan-m Jan 3, 2025

Choose a reason for hiding this comment

levan-m Jan 3, 2025

Choose a reason for hiding this comment

levan-m Jan 3, 2025

Choose a reason for hiding this comment

levan-m commented Jan 4, 2025

tbavelier commented Nov 21, 2024 •

edited

Loading

codecov-commenter commented Nov 21, 2024 •

edited

Loading