Skip to content

Commit 73bbdf3

Browse files
authored
Merge pull request #13 from cozystack/feat/addressname-admission-policy
Ship an optional admission policy reserving IPAddressClaim.spec.addressName
2 parents e4484b3 + 3419374 commit 73bbdf3

10 files changed

Lines changed: 646 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,53 @@ The controller image is published as
124124
`ghcr.io/cozystack/address-controller:main` (plus `main-<sha>` and semver tags)
125125
by the release workflow on every push to main.
126126

127+
128+
## Admission policy
129+
130+
A tenant who can create claims can set `spec.addressName` and bind any
131+
`Available` address of a class by name. The chart can install a
132+
`ValidatingAdmissionPolicy` (`admissionPolicy.enabled`, off by default)
133+
that reserves setting, changing or removing the field to allowlisted
134+
principals; writes that leave it unchanged pass. Scope it with
135+
`admissionPolicy.namespaceSelector`, list admins and, in a brokered
136+
deployment, the machinery that moves addresses between namespaces in
137+
`allowedUsernames`/`allowedGroups`, and run it with
138+
`validationActions: [Warn, Audit]` before `Deny`. Which classes a namespace
139+
may claim from at all is not checked; that is the entitlement object the
140+
design proposal leaves open.
141+
142+
## Metrics
143+
144+
With `metrics.enabled`, the controller serves metrics over HTTPS on port
145+
8443 behind the Service `address-controller-metrics`. Scrapes are
146+
authenticated and authorized against the API server, so the scraper's
147+
identity needs the `address-controller-metrics-reader` ClusterRole, for
148+
example:
149+
150+
```sh
151+
kubectl create clusterrolebinding prometheus-address-controller-metrics \
152+
--clusterrole=address-controller-metrics-reader \
153+
--serviceaccount=<prometheus-namespace>:<prometheus-serviceaccount>
154+
```
155+
156+
Besides the controller-runtime metrics, two gauges count the ledger:
157+
158+
| metric | labels |
159+
|---|---|
160+
| `address_controller_ipaddresses` | `class`, `phase` |
161+
| `address_controller_ipaddressclaims` | `namespace`, `class` (resolved), `phase` |
162+
163+
Claims are the billable unit, as `PersistentVolumeClaim`s are for storage: a
164+
claim holds its addresses whether or not a Service uses them, so
165+
`address_controller_ipaddressclaims{phase="Bound"}` counts what each
166+
namespace holds.
167+
168+
`metrics.serviceMonitor.enabled` and `metrics.prometheusRule.enabled` add a
169+
`ServiceMonitor` and alerts where the `monitoring.coreos.com` API exists:
170+
addresses in `Conflict` or `Lost` for 5 minutes, and claims `Pending` for
171+
15 minutes. `metrics.serviceMonitor.labels` adds labels to the
172+
`ServiceMonitor`, for a Prometheus `serviceMonitorSelector`.
173+
127174
## Development
128175

129176
```sh
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
{{- /* Optional admission policy (proposal, Security: "spec.addressName cannot
2+
be checked in CEL"): only allowlisted principals may set, change or
3+
remove IPAddressClaim.spec.addressName, which binds one specific
4+
Available address and so lets its writer pick an address out of a
5+
class. Writes that leave the field as it was are always admitted. */ -}}
6+
{{- if .Values.admissionPolicy.enabled }}
7+
---
8+
apiVersion: admissionregistration.k8s.io/v1
9+
kind: ValidatingAdmissionPolicy
10+
metadata:
11+
name: address-controller-claim-address-name
12+
labels:
13+
{{- include "address-controller.labels" . | nindent 4 }}
14+
spec:
15+
failurePolicy: Fail
16+
matchConstraints:
17+
resourceRules:
18+
- apiGroups: ["local.sdn.cozystack.io"]
19+
apiVersions: ["v1alpha1"]
20+
operations: ["CREATE", "UPDATE"]
21+
resources: ["ipaddressclaims"]
22+
matchConditions:
23+
- name: not-allowlisted
24+
expression: >-
25+
!(request.userInfo.username in {{ toJson .Values.admissionPolicy.allowedUsernames }}) &&
26+
!(has(request.userInfo.groups) &&
27+
request.userInfo.groups.exists(g, g in {{ toJson .Values.admissionPolicy.allowedGroups }}))
28+
variables:
29+
- name: oldAddressName
30+
expression: >-
31+
request.operation == "UPDATE" && has(oldObject.spec.addressName)
32+
? oldObject.spec.addressName : ""
33+
- name: newAddressName
34+
expression: >-
35+
has(object.spec.addressName) ? object.spec.addressName : ""
36+
validations:
37+
- expression: variables.newAddressName == variables.oldAddressName
38+
message: >-
39+
spec.addressName is reserved to the platform; create the claim without it
40+
and the class assigns an address
41+
reason: Forbidden
42+
---
43+
apiVersion: admissionregistration.k8s.io/v1
44+
kind: ValidatingAdmissionPolicyBinding
45+
metadata:
46+
name: address-controller-claim-address-name
47+
labels:
48+
{{- include "address-controller.labels" . | nindent 4 }}
49+
spec:
50+
policyName: address-controller-claim-address-name
51+
validationActions: {{ toJson .Values.admissionPolicy.validationActions }}
52+
matchResources:
53+
namespaceSelector:
54+
{{- toYaml .Values.admissionPolicy.namespaceSelector | nindent 6 }}
55+
{{- end }}

‎chart/address-controller/templates/controller.yaml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,13 @@ spec:
5858
imagePullPolicy: {{ .Values.imagePullPolicy }}
5959
args:
6060
- --leader-elect={{ .Values.controller.leaderElect }}
61-
- --metrics-bind-address={{ .Values.controller.metricsBindAddress }}
61+
- --metrics-bind-address={{ if .Values.metrics.enabled }}:8443{{ else }}{{ .Values.controller.metricsBindAddress }}{{ end }}
6262
- --health-probe-bind-address=:8081
63+
{{- if .Values.metrics.enabled }}
64+
ports:
65+
- name: metrics
66+
containerPort: 8443
67+
{{- end }}
6368
securityContext:
6469
allowPrivilegeEscalation: false
6570
readOnlyRootFilesystem: true
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
{{- if .Values.metrics.enabled }}
2+
---
3+
apiVersion: v1
4+
kind: Service
5+
metadata:
6+
name: address-controller-metrics
7+
namespace: {{ .Release.Namespace }}
8+
labels:
9+
app: address-controller-metrics
10+
{{- include "address-controller.labels" . | nindent 4 }}
11+
spec:
12+
selector:
13+
app: address-controller
14+
ports:
15+
- name: metrics
16+
port: 8443
17+
targetPort: metrics
18+
---
19+
# Lets the controller authenticate and authorize scrape requests.
20+
apiVersion: rbac.authorization.k8s.io/v1
21+
kind: ClusterRole
22+
metadata:
23+
name: address-controller-metrics-auth
24+
labels:
25+
{{- include "address-controller.labels" . | nindent 4 }}
26+
rules:
27+
- apiGroups: ["authentication.k8s.io"]
28+
resources: ["tokenreviews"]
29+
verbs: ["create"]
30+
- apiGroups: ["authorization.k8s.io"]
31+
resources: ["subjectaccessreviews"]
32+
verbs: ["create"]
33+
---
34+
apiVersion: rbac.authorization.k8s.io/v1
35+
kind: ClusterRoleBinding
36+
metadata:
37+
name: address-controller-metrics-auth
38+
labels:
39+
{{- include "address-controller.labels" . | nindent 4 }}
40+
roleRef:
41+
apiGroup: rbac.authorization.k8s.io
42+
kind: ClusterRole
43+
name: address-controller-metrics-auth
44+
subjects:
45+
- kind: ServiceAccount
46+
name: address-controller
47+
namespace: {{ .Release.Namespace }}
48+
---
49+
# Bind a scraper's identity to this role to let it read the metrics.
50+
apiVersion: rbac.authorization.k8s.io/v1
51+
kind: ClusterRole
52+
metadata:
53+
name: address-controller-metrics-reader
54+
labels:
55+
{{- include "address-controller.labels" . | nindent 4 }}
56+
rules:
57+
- nonResourceURLs: ["/metrics"]
58+
verbs: ["get"]
59+
{{- if and .Values.metrics.serviceMonitor.enabled (.Capabilities.APIVersions.Has "monitoring.coreos.com/v1/ServiceMonitor") }}
60+
---
61+
apiVersion: monitoring.coreos.com/v1
62+
kind: ServiceMonitor
63+
metadata:
64+
name: address-controller
65+
namespace: {{ .Release.Namespace }}
66+
labels:
67+
{{- include "address-controller.labels" . | nindent 4 }}
68+
{{- with .Values.metrics.serviceMonitor.labels }}
69+
{{- toYaml . | nindent 4 }}
70+
{{- end }}
71+
spec:
72+
selector:
73+
matchLabels:
74+
app: address-controller-metrics
75+
endpoints:
76+
- port: metrics
77+
scheme: https
78+
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
79+
tlsConfig:
80+
# The controller serves a self-signed certificate.
81+
insecureSkipVerify: true
82+
{{- end }}
83+
{{- if and .Values.metrics.prometheusRule.enabled (.Capabilities.APIVersions.Has "monitoring.coreos.com/v1/PrometheusRule") }}
84+
---
85+
apiVersion: monitoring.coreos.com/v1
86+
kind: PrometheusRule
87+
metadata:
88+
name: address-controller
89+
namespace: {{ .Release.Namespace }}
90+
labels:
91+
{{- include "address-controller.labels" . | nindent 4 }}
92+
spec:
93+
groups:
94+
- name: address-controller
95+
rules:
96+
- alert: IPAddressConflict
97+
expr: sum by (class) (address_controller_ipaddresses{phase="Conflict"}) > 0
98+
for: 5m
99+
labels:
100+
severity: warning
101+
annotations:
102+
summary: "{{ "{{ $value }}" }} IPAddress(es) of class {{ "{{ $labels.class }}" }} in Conflict"
103+
description: A Service holds an address that its binding does not authorize. kubectl get ipaddresses names the offender.
104+
- alert: IPAddressLost
105+
expr: sum by (class) (address_controller_ipaddresses{phase="Lost"}) > 0
106+
for: 5m
107+
labels:
108+
severity: warning
109+
annotations:
110+
summary: "{{ "{{ $value }}" }} IPAddress(es) of class {{ "{{ $labels.class }}" }} Lost"
111+
description: The backend no longer holds a reserved address. Its claim reports Lost until the address is restored or deleted.
112+
- alert: IPAddressClaimPending
113+
expr: sum by (namespace, class) (address_controller_ipaddressclaims{phase="Pending"}) > 0
114+
for: 15m
115+
labels:
116+
severity: warning
117+
annotations:
118+
summary: "{{ "{{ $value }}" }} IPAddressClaim(s) in {{ "{{ $labels.namespace }}" }} Pending for 15m"
119+
description: The claim's class may be exhausted, missing or refused by its driver. kubectl describe on the claim shows why.
120+
{{- end }}
121+
{{- end }}

‎chart/address-controller/values.yaml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,44 @@ controller:
1818
memory: 64Mi
1919
limits:
2020
memory: 128Mi
21+
22+
metrics:
23+
# Serve controller-runtime and inventory metrics (address_controller_*)
24+
# over HTTPS on 8443, behind the Service address-controller-metrics.
25+
# Requests are authenticated and authorized against the API server, so a
26+
# scraper needs GET on the /metrics non-resource URL: bind its identity
27+
# to the address-controller-metrics-reader ClusterRole. Overrides
28+
# controller.metricsBindAddress.
29+
enabled: false
30+
serviceMonitor:
31+
# A monitoring.coreos.com ServiceMonitor for the Service; rendered only
32+
# where that API exists. The scraping Prometheus's ServiceAccount still
33+
# needs a ClusterRoleBinding to address-controller-metrics-reader, which
34+
# the chart cannot create without knowing that ServiceAccount.
35+
enabled: false
36+
# Extra labels on the ServiceMonitor, e.g. to match a Prometheus
37+
# serviceMonitorSelector.
38+
labels: {}
39+
prometheusRule:
40+
# Alerts on Conflict and Lost addresses and on long-Pending claims;
41+
# rendered only where monitoring.coreos.com exists.
42+
enabled: false
43+
44+
# Optional ValidatingAdmissionPolicy reserving IPAddressClaim.spec.addressName,
45+
# which binds one specific Available address of a class, to the principals
46+
# below. Writes that leave the field as it was are always admitted. In a
47+
# brokered deployment, allowlist the machinery that moves addresses between
48+
# namespaces.
49+
admissionPolicy:
50+
enabled: false
51+
# [Deny], or e.g. [Warn, Audit] to observe what would be denied first.
52+
validationActions: [Deny]
53+
# Namespaces the policy applies to; scope it to tenant namespaces.
54+
namespaceSelector: {}
55+
# Admission policies bind system:masters too, so cluster admins must be
56+
# listed; kubeadm since 1.29 issues admin credentials in
57+
# kubeadm:cluster-admins instead.
58+
allowedUsernames: []
59+
allowedGroups:
60+
- system:masters
61+
- kubeadm:cluster-admins

‎cmd/main.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ import (
3434
"sigs.k8s.io/controller-runtime/pkg/certwatcher"
3535
"sigs.k8s.io/controller-runtime/pkg/healthz"
3636
"sigs.k8s.io/controller-runtime/pkg/log/zap"
37+
"sigs.k8s.io/controller-runtime/pkg/metrics"
3738
"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
3839
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
3940
"sigs.k8s.io/controller-runtime/pkg/webhook"
@@ -207,6 +208,8 @@ func main() {
207208
os.Exit(1)
208209
}
209210

211+
metrics.Registry.MustRegister(&controller.InventoryCollector{Reader: mgr.GetClient()})
212+
210213
if err = controller.SetupIndexes(context.Background(), mgr); err != nil {
211214
setupLog.Error(err, "unable to set up field indexes")
212215
os.Exit(1)

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ go 1.26
55
godebug default=go1.23
66

77
require (
8+
github.com/prometheus/client_golang v1.19.1
89
k8s.io/api v0.32.0
910
k8s.io/apimachinery v0.32.0
1011
k8s.io/client-go v0.32.0
@@ -48,7 +49,6 @@ require (
4849
github.com/modern-go/reflect2 v1.0.2 // indirect
4950
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
5051
github.com/pkg/errors v0.9.1 // indirect
51-
github.com/prometheus/client_golang v1.19.1 // indirect
5252
github.com/prometheus/client_model v0.6.1 // indirect
5353
github.com/prometheus/common v0.55.0 // indirect
5454
github.com/prometheus/procfs v0.15.1 // indirect

0 commit comments

Comments
 (0)