Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add classNamespace to topology #11352

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
14 changes: 13 additions & 1 deletion api/v1beta1/cluster_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ limitations under the License.
package v1beta1

import (
"cmp"
"fmt"
"net"
"strings"
Expand Down Expand Up @@ -509,6 +510,15 @@ type Topology struct {
// The name of the ClusterClass object to create the topology.
Class string `json:"class"`

// The namespace of the ClusterClass object to create the topology. Empty namespace assumes the namespace of the cluster object.
// Class namespace changes are not supported by the rebase procedure, as different CC namespace uses namespace-local templates.
// Cluster templates namespace modification is not allowed.
// +optional
fabriziopandini marked this conversation as resolved.
Show resolved Hide resolved
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=253
// +kubebuilder:validation:Pattern="^[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9](?:[-a-z0-9]*[a-z0-9])?)*$"
ClassNamespace string `json:"classNamespace,omitempty"`
fabriziopandini marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a minimum and maximum length to all raw string types, since this is a namespace, it follows DNS1123Subdomain and should also be validated as such (there's a regex for this). The maximum length is 253 and the minimum is 1.

Since you have omitempty, you can safely add the MinLength, it will prevent anyone using an unstructured client from persisting classNamespace: "" which would then otherwise not round trip through a structured request


// The Kubernetes version of the cluster.
Version string `json:"version"`

Expand Down Expand Up @@ -1037,7 +1047,9 @@ func (c *Cluster) GetClassKey() types.NamespacedName {
if c.Spec.Topology == nil {
return types.NamespacedName{}
}
return types.NamespacedName{Namespace: c.GetNamespace(), Name: c.Spec.Topology.Class}

namespace := cmp.Or(c.Spec.Topology.ClassNamespace, c.Namespace)
return types.NamespacedName{Namespace: namespace, Name: c.Spec.Topology.Class}
}

// GetConditions returns the set of conditions for this object.
Expand Down
23 changes: 23 additions & 0 deletions api/v1beta1/index/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ import (
const (
// ClusterClassNameField is used by the Cluster controller to index Clusters by ClusterClass name.
ClusterClassNameField = "spec.topology.class"
// ClusterClassNamespaceField is used by the Cluster controller to index Clusters by ClusterClass namespace.
ClusterClassNamespaceField = "spec.topology.classNamespace"
)

// ByClusterClassName adds the cluster class name index to the
Expand All @@ -44,6 +46,18 @@ func ByClusterClassName(ctx context.Context, mgr ctrl.Manager) error {
return nil
}

// ByClusterClassNamespace adds the cluster class namespace index to the
// managers cache.
func ByClusterClassNamespace(ctx context.Context, mgr ctrl.Manager) error {
if err := mgr.GetCache().IndexField(ctx, &clusterv1.Cluster{},
ClusterClassNamespaceField,
ClusterByClusterClassClassNamespace,
); err != nil {
return errors.Wrap(err, "error setting index field")
}
return nil
}

// ClusterByClusterClassClassName contains the logic to index Clusters by ClusterClass name.
func ClusterByClusterClassClassName(o client.Object) []string {
cluster, ok := o.(*clusterv1.Cluster)
Expand All @@ -55,3 +69,12 @@ func ClusterByClusterClassClassName(o client.Object) []string {
}
return nil
}

// ClusterByClusterClassClassNamespace contains the logic to index Clusters by ClusterClass namespace.
func ClusterByClusterClassClassNamespace(o client.Object) []string {
cluster, ok := o.(*clusterv1.Cluster)
if !ok {
panic(fmt.Sprintf("Expected Cluster but got a %T", o))
}
return []string{cluster.GetClassKey().Namespace}
}
4 changes: 4 additions & 0 deletions api/v1beta1/index/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ func AddDefaultIndexes(ctx context.Context, mgr ctrl.Manager) error {
if err := ByClusterClassName(ctx, mgr); err != nil {
return err
}

if err := ByClusterClassNamespace(ctx, mgr); err != nil {
return err
}
}

if feature.Gates.Enabled(feature.MachinePool) {
Expand Down
7 changes: 7 additions & 0 deletions api/v1beta1/zz_generated.openapi.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions config/crd/bases/cluster.x-k8s.io_clusters.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,89 @@ spec:
template: "{{ .cluster.name }}-{{ .machinePool.topologyName }}-{{ .random }}"
```

### Defining a custom namespace for ClusterClass object

As a user, I may need to create a `Cluster` from a `ClusterClass` object that exists only in a different namespace. To uniquely identify the `ClusterClass`, a `NamespacedName` ref is constructed from combination of:
* `cluster.spec.topology.classNamespace` - namespace of the `ClusterClass` object.
* `cluster.spec.topology.class` - name of the `ClusterClass` object.

Example of the `Cluster` object with the `name/namespace` reference:

```yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: my-docker-cluster
namespace: default
spec:
topology:
class: docker-clusterclass-v0.1.0
classNamespace: default
version: v1.22.4
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: default-worker
name: md-0
replicas: 4
failureDomain: region
```

<aside class="note warning">

<h1>Cluster rebase across namespaces</h1>

Changing `classNamespace` is not supported in rebase procedure, while changing `class` reference to a different `ClusterClass` from the same namespace performs regular `Cluster` rebase procedure.

</aside>

#### Securing cross-namespace reference to the ClusterClass

It is often desirable to restrict free cross-namespace `ClusterClass` access for the `Cluster` object. This can be implemented by defining a [`ValidatingAdmissionPolicy`](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) on the `Cluster` object.
Copy link
Member

@enxebre enxebre Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

besides the on admission check which is nice, do we have any rbac recommendation for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VAP supports secondary authz for added RBAC control, and VAP binding can be used with label selectors, which addresses #5673 (comment). From the proposal itself, it seems using the policy for added restriction on top of RBAC is within the scope.

Webhook allows to use paramRef of any kind, which can be potentially explored with specific CRD to further restrict access beyond admission, with a controller acting on that resource.

Currently, this is just an example of how an on top policy can be defined (if needed) in k8s 1.30+, where a user may decide to use different policy mechanisms to further restrict access, including a more granular RBAC. I’m thinking to showcase it as an option, but to not enforce any specific solution within this PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also an RBAC recommendation in https://kccncna2024.sched.com/event/1hoyX, if a talk considered to be one.


An example of such policy may be:

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: "cluster-class-ref.cluster.x-k8s.io"
spec:
failurePolicy: Fail
paramKind:
apiVersion: v1
kind: Secret
matchConstraints:
resourceRules:
- apiGroups: ["cluster.x-k8s.io"]
apiVersions: ["v1beta1"]
operations: ["CREATE", "UPDATE"]
resources: ["clusters"]
validations:
- expression: "!has(object.spec.topology.classNamespace) || object.spec.topology.classNamespace in params.data"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "cluster-class-ref-binding.cluster.x-k8s.io"
spec:
policyName: "cluster-class-ref.cluster.x-k8s.io"
validationActions: [Deny]
paramRef:
name: "allowed-namespaces.cluster-class-ref.cluster.x-k8s.io"
namespace: "default"
parameterNotFoundAction: Deny
---
apiVersion: v1
kind: Secret
metadata:
name: "allowed-namespaces.cluster-class-ref.cluster.x-k8s.io"
namespace: "default"
data:
default: ""
```

## Advanced features of ClusterClass with patches

This section will explain more advanced features of ClusterClass patches.
Expand Down
1 change: 1 addition & 0 deletions internal/apis/core/v1alpha4/conversion.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ func (src *Cluster) ConvertTo(dstRaw conversion.Hub) error {
if dst.Spec.Topology == nil {
dst.Spec.Topology = &clusterv1.Topology{}
}
dst.Spec.Topology.ClassNamespace = restored.Spec.Topology.ClassNamespace
dst.Spec.Topology.Variables = restored.Spec.Topology.Variables
dst.Spec.Topology.ControlPlane.Variables = restored.Spec.Topology.ControlPlane.Variables

Expand Down
1 change: 1 addition & 0 deletions internal/apis/core/v1alpha4/zz_generated.conversion.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 4 additions & 2 deletions internal/controllers/topology/cluster/cluster_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -372,8 +372,10 @@ func (r *Reconciler) clusterClassToCluster(ctx context.Context, o client.Object)
if err := r.Client.List(
ctx,
clusterList,
client.MatchingFields{index.ClusterClassNameField: clusterClass.Name},
client.InNamespace(clusterClass.Namespace),
client.MatchingFields{
index.ClusterClassNameField: clusterClass.Name,
index.ClusterClassNamespaceField: clusterClass.Namespace,
},
); err != nil {
return nil
}
Expand Down
68 changes: 68 additions & 0 deletions internal/controllers/topology/cluster/cluster_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ import (
var (
clusterName1 = "cluster1"
clusterName2 = "cluster2"
clusterName3 = "cluster3"
clusterClassName1 = "class1"
clusterClassName2 = "class2"
infrastructureMachineTemplateName1 = "inframachinetemplate1"
Expand Down Expand Up @@ -854,6 +855,19 @@ func setupTestEnvForIntegrationTests(ns *corev1.Namespace) (func() error, error)
Build()).
Build()

// Cross ns referencing cluster
cluster3 := builder.Cluster(ns.Name, clusterName3).
WithTopology(
builder.ClusterTopology().
WithClass(clusterClass.Name).
WithClassNamespace("other").
WithMachineDeployment(machineDeploymentTopology2).
WithMachinePool(machinePoolTopology2).
WithVersion("1.21.0").
WithControlPlaneReplicas(1).
Build()).
Build()

// Setup kubeconfig secrets for the clusters, so the ClusterCacheTracker works.
cluster1Secret := kubeconfig.GenerateSecret(cluster1, kubeconfig.FromEnvTestConfig(env.Config, cluster1))
cluster2Secret := kubeconfig.GenerateSecret(cluster2, kubeconfig.FromEnvTestConfig(env.Config, cluster2))
Expand All @@ -876,6 +890,7 @@ func setupTestEnvForIntegrationTests(ns *corev1.Namespace) (func() error, error)
clusterClassForRebase,
cluster1,
cluster2,
cluster3,
cluster1Secret,
cluster2Secret,
}
Expand Down Expand Up @@ -1577,3 +1592,56 @@ func TestReconciler_ValidateCluster(t *testing.T) {
})
}
}

func TestClusterClassToCluster(t *testing.T) {
utilfeature.SetFeatureGateDuringTest(t, feature.Gates, feature.ClusterTopology, true)
g := NewWithT(t)

ns, err := env.CreateNamespace(ctx, "cluster-reconcile-namespace")
g.Expect(err).ToNot(HaveOccurred())

// Create the objects needed for the integration test:
cleanup, err := setupTestEnvForIntegrationTests(ns)
g.Expect(err).ToNot(HaveOccurred())

// Defer a cleanup function that deletes each of the objects created during setupTestEnvForIntegrationTests.
defer func() {
g.Expect(cleanup()).To(Succeed())
}()

tests := []struct {
name string
clusterClass *clusterv1.ClusterClass
expected []reconcile.Request
}{
{
name: "ClusterClass change should request reconcile for the referenced class",
clusterClass: builder.ClusterClass(ns.Name, clusterClassName1).Build(),
expected: []reconcile.Request{
{NamespacedName: client.ObjectKeyFromObject(builder.Cluster(ns.Name, clusterName1).Build())},
{NamespacedName: client.ObjectKeyFromObject(builder.Cluster(ns.Name, clusterName2).Build())},
},
},
{
name: "ClusterClass with no matching name and namespace should not trigger reconcile",
clusterClass: builder.ClusterClass("other", clusterClassName2).Build(),
expected: []reconcile.Request{},
},
{
name: "Different ClusterClass with matching name and namespace should trigger reconcile",
clusterClass: builder.ClusterClass("other", clusterClassName1).Build(),
expected: []reconcile.Request{
{NamespacedName: client.ObjectKeyFromObject(builder.Cluster(ns.Name, clusterName3).Build())},
},
},
}

for _, tt := range tests {
t.Run(tt.name, func(*testing.T) {
r := &Reconciler{Client: env.GetClient()}

requests := r.clusterClassToCluster(ctx, tt.clusterClass)
g.Expect(requests).To(Equal(tt.expected))
})
}
}
5 changes: 4 additions & 1 deletion internal/controllers/topology/cluster/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,14 @@ func TestMain(m *testing.M) {
if err := index.AddDefaultIndexes(ctx, mgr); err != nil {
panic(fmt.Sprintf("unable to setup index: %v", err))
}
// Set up the ClusterClassName index explicitly here. This index is ordinarily created in
// Set up the ClusterClassName and ClusterClassNamespace index explicitly here. This index is ordinarily created in
// index.AddDefaultIndexes. That doesn't happen here because the ClusterClass feature flag is not set.
if err := index.ByClusterClassName(ctx, mgr); err != nil {
panic(fmt.Sprintf("unable to setup index: %v", err))
}
if err := index.ByClusterClassNamespace(ctx, mgr); err != nil {
panic(fmt.Sprintf("unable to setup index: %v", err))
}
}
setupReconcilers := func(ctx context.Context, mgr ctrl.Manager) {
clusterCache, err := clustercache.SetupWithManager(ctx, mgr, clustercache.Options{
Expand Down
7 changes: 5 additions & 2 deletions internal/webhooks/clusterclass.go
Original file line number Diff line number Diff line change
Expand Up @@ -379,12 +379,15 @@ func (webhook *ClusterClass) classNamesFromMPWorkerClass(w clusterv1.WorkersClas
func (webhook *ClusterClass) getClustersUsingClusterClass(ctx context.Context, clusterClass *clusterv1.ClusterClass) ([]clusterv1.Cluster, error) {
clusters := &clusterv1.ClusterList{}
err := webhook.Client.List(ctx, clusters,
client.MatchingFields{index.ClusterClassNameField: clusterClass.Name},
client.InNamespace(clusterClass.Namespace),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we should update the index, so it contains both namespace and name of the cluster class

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking a little bit more about this, I think we change this index so performance in the webhook are optimized and also the resulting code will be more readable (both here and in clusterClassToCluster)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me that in this method there is not enough information to know cluster namespace due to duality of the namespace reference.

If an index is added, there will be another Client.List call for client.MatchingFields{index.ClusterClassNamespaceField: clusterClass.Namespace}, and here I’m not sure how would an empty field be treated - matching an empty string or matching no selector at all?

If the first one the case, then the method will combine two list responses, and it will be more readable. If not, it will only increase the number of calls.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, seems it is not an issue, as the index can use GetClass() method which resolves ambiguity. Should be added now.

client.MatchingFields{
index.ClusterClassNameField: clusterClass.Name,
index.ClusterClassNamespaceField: clusterClass.Namespace,
},
)
if err != nil {
return nil, err
}

return clusters.Items, nil
}

Expand Down
Loading
Loading