容器集群-安全组件安装问题

[AceCoronnet9034]

问题描述：
我在Elkeid平台系统管理->容器集群->添加集群页面添加了一个k8s集群。添加完安装安全组件安装指引中提供的步骤(三个master节点均有操作)，完成了所有步骤。更改完kube-apiserver.yaml后，集群状态正常，无报错日志。但elkeid平台依然显示入侵&威胁检测状态：未安装。且确实未获取数据。

针对这个现象，我做了哪些操作：
1，删除集群重新在平台添加后再次尝试那些步骤。
2，检查核对了audit-policy.yaml和audit.kubeconfig文件内容，确定与平台生成的内容一致。
3，依次重启了k8s集群的三个master节点。
4，查看apiserver日志(无异常)。

环境信息：
OS：Ubuntu 20.04.6 LTS
K8S：v1.22.10
内核版本：1 5.4.0-189-generic

文件路径：
/etc/kubernetes/elkeid-audit/audit-policy.yaml
/etc/kubernetes/elkeid-audit/audit.kubeconfig

kube-apiserver.yaml内容：
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.20.1.1:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:

• command:
  • kube-apiserver
  • --advertise-address=172.20.1.1
  • --allow-privileged=true
  • --authorization-mode=Node,RBAC
  • --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
  • --audit-webhook-config-file=/etc/kubernetes/audit/audit.kubeconfig
  • --client-ca-file=/etc/kubernetes/pki/ca.crt
  • --enable-admission-plugins=NodeRestriction
  • --enable-bootstrap-token-auth=true
  • --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  • --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  • --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
  • --etcd-servers=https://127.0.0.1:2379
  • --feature-gates=TTLAfterFinished=true
  • --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
  • --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
  • --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  • --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
  • --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
  • --requestheader-allowed-names=front-proxy-client
  • --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
  • --requestheader-extra-headers-prefix=X-Remote-Extra-
  • --requestheader-group-headers=X-Remote-Group
  • --requestheader-username-headers=X-Remote-User
  • --secure-port=6443
  • --service-account-issuer=https://kubernetes.default.svc.cluster.local
  • --service-account-key-file=/etc/kubernetes/pki/sa.pub
  • --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
  • --service-cluster-ip-range=10.96.0.0/12
  • --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
  • --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.22.10
    imagePullPolicy: IfNotPresent
    livenessProbe:
    failureThreshold: 8
    httpGet:
    host: 172.20.1.1
    path: /livez
    port: 6443
    scheme: HTTPS
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
    failureThreshold: 3
    httpGet:
    host: 172.20.1.1
    path: /readyz
    port: 6443
    scheme: HTTPS
    periodSeconds: 1
    timeoutSeconds: 15
    resources:
    requests:
    cpu: 250m
    startupProbe:
    failureThreshold: 24
    httpGet:
    host: 172.20.1.1
    path: /livez
    port: 6443
    scheme: HTTPS
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 15
    volumeMounts:
  • mountPath: /etc/ssl/certs
    name: ca-certs
    readOnly: true
  • mountPath: /etc/kubernetes/audit/
    name: elkeid-audit
    readOnly: true
  • mountPath: /etc/ca-certificates
    name: etc-ca-certificates
    readOnly: true
  • mountPath: /etc/pki
    name: etc-pki
    readOnly: true
  • mountPath: /etc/kubernetes/pki
    name: k8s-certs
    readOnly: true
  • mountPath: /etc/localtime
    name: localtime
    readOnly: true
  • mountPath: /usr/local/share/ca-certificates
    name: usr-local-share-ca-certificates
    readOnly: true
  • mountPath: /usr/share/ca-certificates
    name: usr-share-ca-certificates
    readOnly: true
    hostNetwork: true
    priorityClassName: system-node-critical
    securityContext:
    seccompProfile:
    type: RuntimeDefault
    volumes:
• hostPath:
  path: /etc/ssl/certs
  type: DirectoryOrCreate
  name: ca-certs
• hostPath:
  path: /etc/kubernetes/elkeid-audit
  type: Directory
  name: elkeid-audit
• hostPath:
  path: /etc/ca-certificates
  type: DirectoryOrCreate
  name: etc-ca-certificates
• hostPath:
  path: /etc/pki
  type: DirectoryOrCreate
  name: etc-pki
• hostPath:
  path: /etc/kubernetes/pki
  type: DirectoryOrCreate
  name: k8s-certs
• hostPath:
  path: /etc/localtime
  type: File
  name: localtime
• hostPath:
  path: /usr/local/share/ca-certificates
  type: DirectoryOrCreate
  name: usr-local-share-ca-certificates
• hostPath:
  path: /usr/share/ca-certificates
  type: DirectoryOrCreate
  name: usr-share-ca-certificates
  status: {}

[UgOrange]

可以使用 curl -k 测试下 agent_center 审计日志是否正常监听，地址在下载到的 audit webhook config file 中 server 字段中（端口为 6754）

[UgOrange]

比如curl -k -X POST -H "Content-Type: application/json" --data '{"kind":"Event"}' https://{URL}

[AceCoronnet9034]

默认生成的文件中，server字段为server: https://127.0.0.1:6754/rawdata/audit。这里127.0.0.1:6754是表示我需要在k8s节点主机上安装agent_center服务吗？我elkeid部署是按照"单机docker快速部署 (单机测试环境推荐)"文档使用docker部署的，elkeid本身并没有监听6754端口。

[UgOrange]

需要确保 k8s 能够访问通 agent_center 所在的 6754 端口，该端口用于接收审计日志。docker 部署的 ac 应该也会监听对应端口，可以在docker run的时候指定端口映射

[AceCoronnet9034]

我给elkeid_community容器添加了6754端口映射(allinone部署方式只有这个容器)，然后使用命令curl -k -X POST -H "Content-Type: application/json" --data '{"kind":"Event"}' https://172.20.1.10:6754显示结果curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0。在容器内执行curl得到的结果为curl: (58) NSS: client certificate not found (nickname not specified)。正常的返回结果应该是什么？

[UgOrange]

可以重新配置一下 k8s 看看能不能正常接收到审计日志

[AceCoronnet9034]

重新配置了k8s依然是未安装，我查看master1节点的kube-apiserver日志如下：AUDIT: id="a5b2abcc-17b3-4d0e-967c-e8d8e0e23ffb" stage="ResponseComplete" ip="172.20.1.1" method="patch" user="system:node:master-1" groups=""system:nodes","system:authenticated"" as="" asgroups="" user-agent="kubelet/v1.22.10 (linux/amd64) kubernetes/eae22ba" namespace="" uri="/api/v1/nodes/master-1/status?timeout=10s" response="200"
看起来审计似乎配置是生效了的，k8s ping elkeid的6754端口也是通的。安全组件状态始终无法工作。

[UgOrange]

需要确保证书正确，重启 kube-apiserver（可以通过移动并恢复apiserver YAML文件的方式触发重启），然后建议通过 netstat 等工具查看是否有请求 6754 的网络连接

[DirtyPipe]

hello，这个问题有结果吗？我也遇到了。我使用的docker部署，6754端口我是通过iptables转发进去的，在容器里面看有收到容器集群发给6754端口的网络连接。但是找不到为啥安全组件一直不上线

[DirtyPipe]

已经解决，k8s会验证服务端证书的IP范围和证书是否一直，docker生成的证书为127.0.0.1
用脚本/elkeid/agent_center/k8s_cert_gen.sh重新生成证书覆盖原证书并重启服务elkeid_ac.service
然后再去界面重新生成audit.kubeconfig配置文件