Skip to content

Commit 9c34f1c

Browse files
siggysiggy
authored
Introduce Thanos autoscaling demo (#253)
This demo was given at Kubecon Barcelona 2019: - https://sched.co/MPbU - https://youtu.be/qTxunwzYO0g Signed-off-by: Andrew Seigner <[email protected]>
1 parent cc5a502 commit 9c34f1c

13 files changed

+9053
-3185
lines changed

‎thanos-demo/README.md‎

Lines changed: 61 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# Thanos Demo
22

33
This environment demonstrates deploying [Linkerd](https://linkerd.io) and sample
4-
apps across 4 cluster providers, and aggregating metrics into a single
5-
[Thanos](https://github.com/improbable-eng/thanos) Querier.
4+
apps across 4 cluster providers, aggregating metrics into a single
5+
[Thanos](https://github.com/improbable-eng/thanos) Querier, and autoscaling the
6+
sample apps when global latency increases.
67

78
These configs assume the following:
89
- Working Kubernetes clusters in 4 cloud providers:
@@ -12,6 +13,10 @@ These configs assume the following:
1213
- Google Kubernetes Engine (GKE)
1314
- Block storage configured and accessible from each Kubernetes cluster
1415

16+
This demo was given at Kubecon Barcelona in 2019:
17+
- https://sched.co/MPbU
18+
- https://youtu.be/qTxunwzYO0g
19+
1520
## Config files
1621

1722
- `cluster-*.yaml`: `Namespace`, `Service`, and `ConfigMap` objects
@@ -20,9 +25,14 @@ These configs assume the following:
2025
[Object Storage](https://github.com/improbable-eng/thanos/blob/master/docs/storage.md)
2126
in the [Thanos repo](https://github.com/improbable-eng/thanos).
2227
- `linkerd-install-*.yaml`: modified `linkerd install` configs to support Thanos
23-
integration.
28+
integration.
2429
- [`thanos-querier.yaml`](thanos-querier.yaml): The Thanos Querier and Grafana,
2530
to aggregate metrics from all clusters.
31+
- [`k8s-prometheus-adapter.yaml`](k8s-prometheus-adapter.yaml):
32+
[k8s-prometheus-adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter),
33+
to query metrics from Thanos and serve them to the HorizontalPodAutoscaler.
34+
- [`strest.yaml`](strest.yaml):
35+
[strest-grpc](https://github.com/BuoyantIO/strest-grpc) sample application.
2636

2737
### Linkerd / Thanos integration
2838

@@ -163,40 +173,24 @@ done
163173
linkerd --context $AKS check
164174
```
165175

166-
## Install sample apps
176+
## Install sample apps, inject Linkerd
167177

168178
```bash
169179
for CLUSTER in $AKS $DO $EKS $GKE
170180
do
171-
curl -s https://run.linkerd.io/emojivoto.yml |
181+
cat strest.yaml |
182+
linkerd --context $CLUSTER inject - |
172183
kubectl --context $CLUSTER apply -f -
173184
done
174185
```
175186

176-
### View sample app
177-
178-
```bash
179-
kubectl --context $AKS -n emojivoto port-forward svc/web-svc 8080:80
180-
```
181-
182-
### Inject sample apps with Linkerd
183-
184-
```bash
185-
for CLUSTER in $AKS $DO $EKS $GKE
186-
do
187-
curl -s https://run.linkerd.io/emojivoto.yml |
188-
linkerd inject - |
189-
kubectl --context $CLUSTER apply -f -
190-
done
191-
```
192-
193-
Validate proxy injected
187+
### Validate proxy injected
194188

195189
```bash
196190
for CLUSTER in $AKS $DO $EKS $GKE
197191
do
198192
printf "\n$CLUSTER\n"
199-
linkerd --context $CLUSTER -n emojivoto stat deploy
193+
linkerd --context $CLUSTER -n strest stat deploy
200194
done
201195
```
202196

@@ -278,3 +272,46 @@ kubectl --context $AKS -n thanos-demo port-forward svc/thanos-querier 10902
278272
```bash
279273
kubectl --context $AKS -n thanos-demo port-forward svc/grafana 3000
280274
```
275+
276+
#### Autoscaling
277+
278+
[`k8s-prometheus-adapter.yaml`](k8s-prometheus-adapter.yaml) runs in each
279+
cluster, queries Thanos for global metrics, and provides those metrics to a
280+
HorizontalPodAutoscaler. The adapter must be provided a Thanos IP address, which
281+
was created via a LoadBalancer in [`thanos-querier.yaml`](thanos-querier.yaml).
282+
283+
Obtain static addresses for Thanos Querier:
284+
285+
```bash
286+
kubectl --context $AKS -n thanos-demo get svc/thanos-querier -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
287+
```
288+
289+
Substitute IP address in
290+
[`k8s-prometheus-adapter.yaml`](k8s-prometheus-adapter.yaml):
291+
292+
```yaml
293+
containers:
294+
- name: custom-metrics-apiserver
295+
image: directxman12/k8s-prometheus-adapter-amd64:v0.5.0
296+
args:
297+
...
298+
- --prometheus-url=http://[THANOS_QUERIER_IP]:10902/
299+
```
300+
301+
```bash
302+
for CLUSTER in $AKS $DO $EKS $GKE
303+
do
304+
cat k8s-prometheus-adapter.yaml | kubectl --context $CLUSTER apply -f -
305+
done
306+
307+
# verify
308+
kubectl --context $AKS get --raw /apis/custom.metrics.k8s.io/v1beta1
309+
kubectl --context $AKS get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/strest/deployments/*/response_latency_ms_p99
310+
kubectl --context $AKS -n strest describe hpa/strest
311+
```
312+
313+
##### Scale up one strest-client
314+
315+
```bash
316+
kubectl --context $GKE -n strest scale --replicas=20 deploy/strest-client
317+
```

‎thanos-demo/cluster-aks.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ kind: Namespace
33
apiVersion: v1
44
metadata:
55
name: linkerd
6+
annotations:
7+
linkerd.io/inject: disabled
68
---
79
kind: Service
810
apiVersion: v1

‎thanos-demo/cluster-do.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ kind: Namespace
33
apiVersion: v1
44
metadata:
55
name: linkerd
6+
annotations:
7+
linkerd.io/inject: disabled
68
---
79
kind: Service
810
apiVersion: v1

‎thanos-demo/cluster-eks.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ kind: Namespace
33
apiVersion: v1
44
metadata:
55
name: linkerd
6+
annotations:
7+
linkerd.io/inject: disabled
68
---
79
kind: Service
810
apiVersion: v1

‎thanos-demo/cluster-gke.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ kind: Namespace
33
apiVersion: v1
44
metadata:
55
name: linkerd
6+
annotations:
7+
linkerd.io/inject: disabled
68
---
79
kind: Service
810
apiVersion: v1

‎thanos-demo/k8s-prometheus-adapter.yaml‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
---
2+
apiVersion: v1
3+
kind: Namespace
4+
metadata:
5+
name: custom-metrics
6+
---
7+
apiVersion: apiregistration.k8s.io/v1beta1
8+
kind: APIService
9+
metadata:
10+
name: v1beta1.custom.metrics.k8s.io
11+
spec:
12+
service:
13+
name: custom-metrics-apiserver
14+
namespace: custom-metrics
15+
group: custom.metrics.k8s.io
16+
version: v1beta1
17+
insecureSkipTLSVerify: true
18+
groupPriorityMinimum: 100
19+
versionPriority: 100
20+
---
21+
apiVersion: rbac.authorization.k8s.io/v1
22+
kind: ClusterRoleBinding
23+
metadata:
24+
name: custom-metrics:system:auth-delegator
25+
roleRef:
26+
apiGroup: rbac.authorization.k8s.io
27+
kind: ClusterRole
28+
name: system:auth-delegator
29+
subjects:
30+
- kind: ServiceAccount
31+
name: custom-metrics-apiserver
32+
namespace: custom-metrics
33+
---
34+
apiVersion: rbac.authorization.k8s.io/v1
35+
kind: RoleBinding
36+
metadata:
37+
name: custom-metrics-auth-reader
38+
namespace: kube-system
39+
roleRef:
40+
apiGroup: rbac.authorization.k8s.io
41+
kind: Role
42+
name: extension-apiserver-authentication-reader
43+
subjects:
44+
- kind: ServiceAccount
45+
name: custom-metrics-apiserver
46+
namespace: custom-metrics
47+
---
48+
apiVersion: rbac.authorization.k8s.io/v1
49+
kind: ClusterRoleBinding
50+
metadata:
51+
name: custom-metrics-resource-reader
52+
roleRef:
53+
apiGroup: rbac.authorization.k8s.io
54+
kind: ClusterRole
55+
name: custom-metrics-resource-reader
56+
subjects:
57+
- kind: ServiceAccount
58+
name: custom-metrics-apiserver
59+
namespace: custom-metrics
60+
---
61+
apiVersion: rbac.authorization.k8s.io/v1
62+
kind: ClusterRole
63+
metadata:
64+
name: custom-metrics-resource-reader
65+
rules:
66+
- apiGroups:
67+
- ""
68+
resources:
69+
- namespaces
70+
- pods
71+
- services
72+
verbs:
73+
- get
74+
- list
75+
- apiGroups:
76+
- "extensions"
77+
resources:
78+
- deployments
79+
verbs:
80+
- get
81+
- list
82+
---
83+
kind: ServiceAccount
84+
apiVersion: v1
85+
metadata:
86+
name: custom-metrics-apiserver
87+
namespace: custom-metrics
88+
---
89+
apiVersion: rbac.authorization.k8s.io/v1
90+
kind: ClusterRoleBinding
91+
metadata:
92+
name: hpa-controller-custom-metrics
93+
roleRef:
94+
apiGroup: rbac.authorization.k8s.io
95+
kind: ClusterRole
96+
name: custom-metrics-server-resources
97+
subjects:
98+
- kind: ServiceAccount
99+
name: horizontal-pod-autoscaler
100+
namespace: kube-system
101+
---
102+
apiVersion: rbac.authorization.k8s.io/v1
103+
kind: ClusterRole
104+
metadata:
105+
name: custom-metrics-server-resources
106+
rules:
107+
- apiGroups:
108+
- custom.metrics.k8s.io
109+
resources: ["*"]
110+
verbs: ["*"]
111+
---
112+
apiVersion: v1
113+
kind: Secret
114+
metadata:
115+
name: cm-adapter-serving-certs
116+
namespace: custom-metrics
117+
data:
118+
serving.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURZRENDQWtpZ0F3SUJBZ0lVZjg4a1cxNHp3cXNjR0d2TEpmMzV1N3BSTEhJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0RURUxNQWtHQTFVRUF3d0NZMkV3SGhjTk1Ua3dOVEE0TVRVeU56QXdXaGNOTWpRd05UQTJNVFV5TnpBdwpXakFqTVNFd0h3WURWUVFERXhoamRYTjBiMjB0YldWMGNtbGpjeTFoY0dselpYSjJaWEl3Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUROSzg3VVZLMGp6OEFqU25pbkJMcnQ2dy9xVFlnV1FuemYKcUtwT21OZzVhbEZnN3NJSks3czVBSWg3YkpIYmplaUdWSHczRm1GT3JsSFVzSmhyYUR2OUswbEZpSmVaQ2dTawpTTlIrQ0JhTWJxeTVQbXVLNlBHU0htUkFmeGRoa2U3UGExRnc0UTBsTmFGNitndW5XVFlGRFJycWg3Vlpud3V5CmxKNEg5a2xsa2YwbEtycGtiWWN0MVRhT2Y3MXhuNzFBd0I5OFFCZkpQcDNHZnprQklVS080aW5pVHNhMW02ck4KR3pmVXFrbCtKbko1ZlRvSHcwRmplSVJ0OHpMWlc5LzhwK0hEdW9xczVvUFZZOHY5OCtKZW9IaVFqM3prYS9iUwpQcndobUZsaDc1NEttbERucjVQZnBldjFBbkNxQW5rVDJ1Vk1jSDNpMXZUcnMwbGx0c1V2QWdNQkFBR2pnYUV3CmdaNHdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01Bd0dBMVVkRXdFQi93UUNNQUF3SFFZRFZSME9CQllFRkpUTDNDZVoKVDFaZ0pXUm1ldzNpc3REMnNvK09NRjhHQTFVZEVRUllNRmFDSjJOMWMzUnZiUzF0WlhSeWFXTnpMV0Z3YVhObApjblpsY2k1amRYTjBiMjB0YldWMGNtbGpjNElyWTNWemRHOXRMVzFsZEhKcFkzTXRZWEJwYzJWeWRtVnlMbU4xCmMzUnZiUzF0WlhSeWFXTnpMbk4yWXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUd2Wk9KNXo3amNvNVVnL1YKYzdlUUZ2OFVBNnh4RVhDVnFyVmgyS0dmYmViU3ZtQnNRaW1VZkVuOHAvT2FqZkNOVXRIZW50R3pMOFdINzAzNQpHd2NUckIzbTVZTVJkRU5UM2hDRnUxTTNnT1dGbURNaGFQRDhZdFY1MGsxbVBUamU1bGhvNkhUNEx6a1J1N3l5CmJsb0dQSEt3UFNMV082WktyTFBjRFk0a0hxRk5MaGhBNVFTNUUyd0J2Nm1XODlaQ0Rrd1dpeGxBa3JLOVU1MCsKV3M2eEQ4dUU4RnJVbEJ3RkY0SUYvSm5lOUJHYjVYSm9NZ0xickpRc0pwNCs5blk0WGxWK0Rwall4VFBmTDBoegpQd2ZYN1VKSENnV0lSc1puaU9xQnBjZUVFVlNpV21hN2RnSVhTZWl0cGFlREc4TVlnSWl2V0dja1BPS0RQT1FlCm5WMTRtUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
119+
serving.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBelN2TzFGU3RJOC9BSTBwNHB3UzY3ZXNQNmsySUZrSjgzNmlxVHBqWU9XcFJZTzdDCkNTdTdPUUNJZTJ5UjI0M29obFI4TnhaaFRxNVIxTENZYTJnNy9TdEpSWWlYbVFvRXBFalVmZ2dXakc2c3VUNXIKaXVqeGtoNWtRSDhYWVpIdXoydFJjT0VOSlRXaGV2b0xwMWsyQlEwYTZvZTFXWjhMc3BTZUIvWkpaWkg5SlNxNgpaRzJITGRVMmpuKzljWis5UU1BZmZFQVh5VDZkeG44NUFTRkNqdUlwNGs3R3RadXF6UnMzMUtwSmZpWnllWDA2CkI4TkJZM2lFYmZNeTJWdmYvS2ZodzdxS3JPYUQxV1BML2ZQaVhxQjRrSTk4NUd2MjBqNjhJWmhaWWUrZUNwcFEKNTYrVDM2WHI5UUp3cWdKNUU5cmxUSEI5NHRiMDY3TkpaYmJGTHdJREFRQUJBb0lCQUVQbkxjckVRNUZJbnJTUApYeU1YdzY0ZVQrUWh5TnBsSXVMNjlYS1J6MjRoSzlIQzgyRUpvaXNaYkJJOU9vREpsSjF3dExPZXFwSFp5NmR5CjB0OHBYa0ZKNURRcEl0TTVwNU9mcndRUE5UZEJJZFVsUFg1NTNnbVlHV0huTGh2U3FwRG5XY3JjRVBleHFrSUQKSTN3OVlFMkhxdExZRzBNUUNsVU9Mc2MvMGNiVlZFSk9kYUw2eUJ0c21aY0VNUXVZMkU5ZWhFdkpoc3VoZ3BpWgpsRzNESlNCL0k1RXZmbVBmUGthWmxIZ1drbG5vYmhxN3RyRFdhcWVjUGs2LzNBaTBISGdQSVRVQTlNQmMrcEtSClJ0cUN3cUtFektxMi9jVXBMZUJOM2xrUVJEV1Y3Z1Fob2JKR25lM2ZYL2VRYVlsQjlJMFFtWTFocjVIUlN1dVAKVHBUME1ERUNnWUVBOVExV2ZGZ0R3by9nWXByMTVDeDFsSVNIeUtMRXBMOXJjaDVxQzA3dXV4WFRIU3RDeGkzUgorU1g0bjFFSngvS1MwTFRPWEtKVnBlL1p4OWNxbHFjM0ZSZFdXaHhRR3laUkNpZlQ3cGtuZnV5ZEtlMkJEQzFpCkh1a1lFOHhFQm41UE43eEhySm1xT20zbFdKZ2hPU1d4ck14TkJtbG90eE1MOFRIc2twY0pKS3NDZ1lFQTFsWloKNVlMdVFJVGhEdXdJMkVIMHBacmhLeTBTSTVlOGs5bHFjOVhZczBua1pPQ09ZU0dJcU5PSjIwWWxlRU05blNiSgovb0lNWWY3enEzQ1dsTWJ6T3lrcVBGUGw2d2wwZ3QybDJTVzBzdUxJUjZIREhDS2UwLytVUDk1Zjhqb2RheExSCllZYTlxRlBnUjQ0Um5sSnhpblNzak9IUDJDVFZPaG96am9LS3VZMENnWUI4R2lNZ21kcGRvRXU4NUVsL0l0engKT0NVa0FQcWhqS2UzaFp1aTJuaWdtcTd0Nk1qS1lNNGo1WTBCOGlLSDlsdFFBeUUzWEtjUGdHYkVoRDEzU252awozSXJJMituZHRndjRucTcvK09ROXJVcnl2dXRKNVhuNXVZYU9FRGFyS3pDblExenhGbWxWYWU1cWJiRDFwaGRUCk1hd1lnUzJjK2JKa2xpbi9HeXB3UFFLQmdFNWxpVDRVZE5hdWc2ODBrRXVoWEQyU2Q5ZVdCYk4xRmFteWRXaG4KQ0hzVGhQR00zU2NUMyt0NGJFMEN6RU8vdkhLVnRINWR6Tyt2RkN0WjVBb0k5d0JsbnlndnU4SUxWRTFsaXZydworcllQa3hmNFIrZ3JsWktQUWdkV2JCdXFQQVprL0tiQ1pPNkR0MTRVemNidCtzK2N1MXhRWC9oOENHM1hGWktECmx3QWxBb0dCQUpwODFVTGF0b05OZzcrUVBPdmdSY09LWVRvUGs1MSt5YlBwQi9DTmR6S1BjczRlYXF0RStMakMKWUo0bmhhRFFGMEx6WjEvK1VhVmtmQ3hSQm9ORThkUkRDUnNOdVB3cW9oZkpFY3VVZ1FaYVpWQllIQ3h2ZWJmWAo1NVgwYmtsall3dVpVTkh5dDM5clJ2ZEUybWE3V1VOUkQ4M2dYYm1oTUtuUUFXbms1a3luCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
120+
---
121+
apiVersion: v1
122+
kind: ConfigMap
123+
metadata:
124+
name: adapter-config
125+
namespace: custom-metrics
126+
data:
127+
config.yaml: |
128+
rules:
129+
- seriesQuery: 'response_latency_ms_bucket{namespace="strest", deployment="strest-server", direction="inbound"}'
130+
resources:
131+
template: <<.Resource>>
132+
name:
133+
matches: "^(.*)_bucket"
134+
as: "${1}_p99"
135+
metricsQuery: 'histogram_quantile(0.99, sum(irate(<<.Series>>{namespace="strest", deployment="strest-server", direction="inbound"}[30s])) by (le, <<.GroupBy>>))'
136+
---
137+
apiVersion: v1
138+
kind: Service
139+
metadata:
140+
name: custom-metrics-apiserver
141+
namespace: custom-metrics
142+
spec:
143+
ports:
144+
- port: 443
145+
targetPort: 6443
146+
selector:
147+
app: custom-metrics-apiserver
148+
---
149+
apiVersion: apps/v1
150+
kind: Deployment
151+
metadata:
152+
labels:
153+
app: custom-metrics-apiserver
154+
name: custom-metrics-apiserver
155+
namespace: custom-metrics
156+
spec:
157+
replicas: 1
158+
selector:
159+
matchLabels:
160+
app: custom-metrics-apiserver
161+
template:
162+
metadata:
163+
labels:
164+
app: custom-metrics-apiserver
165+
name: custom-metrics-apiserver
166+
spec:
167+
serviceAccountName: custom-metrics-apiserver
168+
containers:
169+
- name: custom-metrics-apiserver
170+
image: directxman12/k8s-prometheus-adapter-amd64:v0.5.0
171+
args:
172+
- --secure-port=6443
173+
- --tls-cert-file=/var/run/serving-cert/serving.crt
174+
- --tls-private-key-file=/var/run/serving-cert/serving.key
175+
- --logtostderr=true
176+
- --prometheus-url=http://[THANOS_QUERIER_IP]:10902/
177+
- --metrics-relist-interval=1m
178+
- --config=/etc/adapter/config.yaml
179+
ports:
180+
- containerPort: 6443
181+
volumeMounts:
182+
- mountPath: /var/run/serving-cert
183+
name: volume-serving-cert
184+
readOnly: true
185+
- mountPath: /etc/adapter/
186+
name: config
187+
readOnly: true
188+
- mountPath: /tmp
189+
name: tmp-vol
190+
volumes:
191+
- name: volume-serving-cert
192+
secret:
193+
secretName: cm-adapter-serving-certs
194+
- name: config
195+
configMap:
196+
name: adapter-config
197+
- name: tmp-vol
198+
emptyDir: {}

0 commit comments

Comments
 (0)