/openapi/v3 api endpoint takes too long to respond

[L1ghtman2k]

What happened?

kubectl get --raw /openapi/v3 takes > 30 seconds to respond. This behavior is inconsistent, as sometimes it is instantaneous, and other times it takes too long

This seemed to be caused by a large number of CRDs on the cluster

Intern, terraform resources like helm_release end up timing out, causing errors like:

╷
│ Error: unable to build kubernetes object for pre-delete hook kyverno/templates/hooks/pre-delete.yaml: error validating "": error validating data: the server was unable to return a response in the time allotted, but may still be processing the request
│
│
╵

What did you expect to happen?

This is a multipart issue, that can be resolved either by fixing helm terraform provider, to allow larger timeouts.

I am creating this issue here because it seems to be in part related to vcluster only, as our EKS deployments are just fine. (On the other hand, we aren't running k8s 1.27 on our EKS clusters, which means we likely not leveraging /openapi/v3)

How can we reproduce it (as minimally and precisely as possible)?

Install vcluster with lots of CRDs (monitoring stack, kyverno), size may matter, here are our CRDs:

NAME                              SHORTNAMES              APIVERSION                                NAMESPACED   KIND
bindings                                                  v1                                        true         Binding
componentstatuses                 cs                      v1                                        false        ComponentStatus
configmaps                        cm                      v1                                        true         ConfigMap
endpoints                         ep                      v1                                        true         Endpoints
events                            ev                      v1                                        true         Event
limitranges                       limits                  v1                                        true         LimitRange
namespaces                        ns                      v1                                        false        Namespace
nodes                             no                      v1                                        false        Node
persistentvolumeclaims            pvc                     v1                                        true         PersistentVolumeClaim
persistentvolumes                 pv                      v1                                        false        PersistentVolume
pods                              po                      v1                                        true         Pod
podtemplates                                              v1                                        true         PodTemplate
replicationcontrollers            rc                      v1                                        true         ReplicationController
resourcequotas                    quota                   v1                                        true         ResourceQuota
secrets                                                   v1                                        true         Secret
serviceaccounts                   sa                      v1                                        true         ServiceAccount
services                          svc                     v1                                        true         Service
challenges                                                acme.cert-manager.io/v1                   true         Challenge
orders                                                    acme.cert-manager.io/v1                   true         Order
mutatingwebhookconfigurations                             admissionregistration.k8s.io/v1           false        MutatingWebhookConfiguration
validatingwebhookconfigurations                           admissionregistration.k8s.io/v1           false        ValidatingWebhookConfiguration
customresourcedefinitions         crd,crds                apiextensions.k8s.io/v1                   false        CustomResourceDefinition
apiservices                                               apiregistration.k8s.io/v1                 false        APIService
controllerrevisions                                       apps/v1                                   true         ControllerRevision
daemonsets                        ds                      apps/v1                                   true         DaemonSet
deployments                       deploy                  apps/v1                                   true         Deployment
replicasets                       rs                      apps/v1                                   true         ReplicaSet
statefulsets                      sts                     apps/v1                                   true         StatefulSet
selfsubjectreviews                                        authentication.k8s.io/v1                  false        SelfSubjectReview
tokenreviews                                              authentication.k8s.io/v1                  false        TokenReview
localsubjectaccessreviews                                 authorization.k8s.io/v1                   true         LocalSubjectAccessReview
selfsubjectaccessreviews                                  authorization.k8s.io/v1                   false        SelfSubjectAccessReview
selfsubjectrulesreviews                                   authorization.k8s.io/v1                   false        SelfSubjectRulesReview
subjectaccessreviews                                      authorization.k8s.io/v1                   false        SubjectAccessReview
horizontalpodautoscalers          hpa                     autoscaling/v2                            true         HorizontalPodAutoscaler
cronjobs                          cj                      batch/v1                                  true         CronJob
jobs                                                      batch/v1                                  true         Job
certificaterequests               cr,crs                  cert-manager.io/v1                        true         CertificateRequest
certificates                      cert,certs              cert-manager.io/v1                        true         Certificate
clusterissuers                                            cert-manager.io/v1                        false        ClusterIssuer
issuers                                                   cert-manager.io/v1                        true         Issuer
certificatesigningrequests        csr                     certificates.k8s.io/v1                    false        CertificateSigningRequest
controlplanerequestlimits                                 consul.hashicorp.com/v1alpha1             true         ControlPlaneRequestLimit
exportedservices                  exported-services       consul.hashicorp.com/v1alpha1             true         ExportedServices
gatewayclassconfigs                                       consul.hashicorp.com/v1alpha1             false        GatewayClassConfig
ingressgateways                   ingress-gateway         consul.hashicorp.com/v1alpha1             true         IngressGateway
jwtproviders                                              consul.hashicorp.com/v1alpha1             true         JWTProvider
meshes                                                    consul.hashicorp.com/v1alpha1             true         Mesh
meshservices                                              consul.hashicorp.com/v1alpha1             true         MeshService
proxydefaults                     proxy-defaults          consul.hashicorp.com/v1alpha1             true         ProxyDefaults
samenessgroups                    sameness-group          consul.hashicorp.com/v1alpha1             true         SamenessGroup
servicedefaults                   service-defaults        consul.hashicorp.com/v1alpha1             true         ServiceDefaults
serviceintentions                 service-intentions      consul.hashicorp.com/v1alpha1             true         ServiceIntentions
serviceresolvers                  service-resolver        consul.hashicorp.com/v1alpha1             true         ServiceResolver
servicerouters                    service-router          consul.hashicorp.com/v1alpha1             true         ServiceRouter
servicesplitters                  service-splitter        consul.hashicorp.com/v1alpha1             true         ServiceSplitter
terminatinggateways               terminating-gateway     consul.hashicorp.com/v1alpha1             true         TerminatingGateway
leases                                                    coordination.k8s.io/v1                    true         Lease
endpointslices                                            discovery.k8s.io/v1                       true         EndpointSlice
events                            ev                      events.k8s.io/v1                          true         Event
clusterexternalsecrets            ces                     external-secrets.io/v1beta1               false        ClusterExternalSecret
clustersecretstores               css                     external-secrets.io/v1beta1               false        ClusterSecretStore
externalsecrets                   es                      external-secrets.io/v1beta1               true         ExternalSecret
pushsecrets                                               external-secrets.io/v1alpha1              true         PushSecret
secretstores                      ss                      external-secrets.io/v1beta1               true         SecretStore
flowschemas                                               flowcontrol.apiserver.k8s.io/v1           false        FlowSchema
prioritylevelconfigurations                               flowcontrol.apiserver.k8s.io/v1           false        PriorityLevelConfiguration
clusterfilters                    cfbf                    fluentbit.fluent.io/v1alpha2              false        ClusterFilter
clusterfluentbitconfigs           cfbc                    fluentbit.fluent.io/v1alpha2              false        ClusterFluentBitConfig
clusterinputs                     cfbi                    fluentbit.fluent.io/v1alpha2              false        ClusterInput
clusteroutputs                    cfbo                    fluentbit.fluent.io/v1alpha2              false        ClusterOutput
clusterparsers                    cfbp                    fluentbit.fluent.io/v1alpha2              false        ClusterParser
collectors                        co                      fluentbit.fluent.io/v1alpha2              true         Collector
filters                           fbf                     fluentbit.fluent.io/v1alpha2              true         Filter
fluentbitconfigs                  fbc                     fluentbit.fluent.io/v1alpha2              true         FluentBitConfig
fluentbits                        fb                      fluentbit.fluent.io/v1alpha2              true         FluentBit
outputs                           fbo                     fluentbit.fluent.io/v1alpha2              true         Output
parsers                           fbp                     fluentbit.fluent.io/v1alpha2              true         Parser
clusterfilters                    cfdf                    fluentd.fluent.io/v1alpha1                false        ClusterFilter
clusterfluentdconfigs             cfdc                    fluentd.fluent.io/v1alpha1                false        ClusterFluentdConfig
clusterinputs                     cfdi                    fluentd.fluent.io/v1alpha1                false        ClusterInput
clusteroutputs                    cfdo                    fluentd.fluent.io/v1alpha1                false        ClusterOutput
filters                           fdf                     fluentd.fluent.io/v1alpha1                true         Filter
fluentdconfigs                    fdc                     fluentd.fluent.io/v1alpha1                true         FluentdConfig
fluentds                          fd                      fluentd.fluent.io/v1alpha1                true         Fluentd
inputs                            fdi                     fluentd.fluent.io/v1alpha1                true         Input
outputs                           fdo                     fluentd.fluent.io/v1alpha1                true         Output
gatewayclasses                    gc                      gateway.networking.k8s.io/v1beta1         false        GatewayClass
gateways                          gtw                     gateway.networking.k8s.io/v1beta1         true         Gateway
grpcroutes                                                gateway.networking.k8s.io/v1alpha2        true         GRPCRoute
httproutes                                                gateway.networking.k8s.io/v1beta1         true         HTTPRoute
referencegrants                   refgrant                gateway.networking.k8s.io/v1beta1         true         ReferenceGrant
tcproutes                                                 gateway.networking.k8s.io/v1alpha2        true         TCPRoute
tlsroutes                                                 gateway.networking.k8s.io/v1alpha2        true         TLSRoute
udproutes                                                 gateway.networking.k8s.io/v1alpha2        true         UDPRoute
acraccesstokens                   acraccesstoken          generators.external-secrets.io/v1alpha1   true         ACRAccessToken
ecrauthorizationtokens            ecrauthorizationtoken   generators.external-secrets.io/v1alpha1   true         ECRAuthorizationToken
fakes                             fake                    generators.external-secrets.io/v1alpha1   true         Fake
gcraccesstokens                   gcraccesstoken          generators.external-secrets.io/v1alpha1   true         GCRAccessToken
passwords                         password                generators.external-secrets.io/v1alpha1   true         Password
vaultdynamicsecrets               vaultdynamicsecret      generators.external-secrets.io/v1alpha1   true         VaultDynamicSecret
helmchartconfigs                                          helm.cattle.io/v1                         true         HelmChartConfig
helmcharts                                                helm.cattle.io/v1                         true         HelmChart
miniojobs                         miniojob                job.min.io/v1alpha1                       true         MinIOJob
addons                                                    k3s.cattle.io/v1                          true         Addon
etcdsnapshotfiles                                         k3s.cattle.io/v1                          false        ETCDSnapshotFile
admissionreports                  admr                    kyverno.io/v1alpha2                       true         AdmissionReport
backgroundscanreports             bgscanr                 kyverno.io/v1alpha2                       true         BackgroundScanReport
cleanuppolicies                   cleanpol                kyverno.io/v2beta1                        true         CleanupPolicy
clusteradmissionreports           cadmr                   kyverno.io/v1alpha2                       false        ClusterAdmissionReport
clusterbackgroundscanreports      cbgscanr                kyverno.io/v1alpha2                       false        ClusterBackgroundScanReport
clustercleanuppolicies            ccleanpol               kyverno.io/v2beta1                        false        ClusterCleanupPolicy
clusterpolicies                   cpol                    kyverno.io/v1                             false        ClusterPolicy
policies                          pol                     kyverno.io/v1                             true         Policy
policyexceptions                  polex                   kyverno.io/v2beta1                        true         PolicyException
updaterequests                    ur                      kyverno.io/v1beta1                        true         UpdateRequest
nodes                                                     metrics.k8s.io/v1beta1                    false        NodeMetrics
pods                                                      metrics.k8s.io/v1beta1                    true         PodMetrics
tenants                           tenant                  minio.min.io/v2                           true         Tenant
alertmanagerconfigs               amcfg                   monitoring.coreos.com/v1alpha1            true         AlertmanagerConfig
alertmanagers                     am                      monitoring.coreos.com/v1                  true         Alertmanager
podmonitors                       pmon                    monitoring.coreos.com/v1                  true         PodMonitor
probes                            prb                     monitoring.coreos.com/v1                  true         Probe
prometheusagents                  promagent               monitoring.coreos.com/v1alpha1            true         PrometheusAgent
prometheuses                      prom                    monitoring.coreos.com/v1                  true         Prometheus
prometheusrules                   promrule                monitoring.coreos.com/v1                  true         PrometheusRule
scrapeconfigs                     scfg                    monitoring.coreos.com/v1alpha1            true         ScrapeConfig
servicemonitors                   smon                    monitoring.coreos.com/v1                  true         ServiceMonitor
thanosrulers                      ruler                   monitoring.coreos.com/v1                  true         ThanosRuler
ingressclasses                                            networking.k8s.io/v1                      false        IngressClass
ingresses                         ing                     networking.k8s.io/v1                      true         Ingress
networkpolicies                   netpol                  networking.k8s.io/v1                      true         NetworkPolicy
runtimeclasses                                            node.k8s.io/v1                            false        RuntimeClass
poddisruptionbudgets              pdb                     policy/v1                                 true         PodDisruptionBudget
clusterrolebindings                                       rbac.authorization.k8s.io/v1              false        ClusterRoleBinding
clusterroles                                              rbac.authorization.k8s.io/v1              false        ClusterRole
rolebindings                                              rbac.authorization.k8s.io/v1              true         RoleBinding
roles                                                     rbac.authorization.k8s.io/v1              true         Role
priorityclasses                   pc                      scheduling.k8s.io/v1                      false        PriorityClass
csidrivers                                                storage.k8s.io/v1                         false        CSIDriver
csinodes                                                  storage.k8s.io/v1                         false        CSINode
csistoragecapacities                                      storage.k8s.io/v1                         true         CSIStorageCapacity
storageclasses                    sc                      storage.k8s.io/v1                         false        StorageClass
volumeattachments                                         storage.k8s.io/v1                         false        VolumeAttachment
policybindings                    policybinding           sts.min.io/v1alpha1                       true         PolicyBinding
bundles                                                   trust.cert-manager.io/v1alpha1            false        Bundle
clusterpolicyreports              cpolr                   wgpolicyk8s.io/v1alpha2                   false        ClusterPolicyReport
policyreports                     polr                    wgpolicyk8s.io/v1alpha2                   true         PolicyReport

run kubectl get --raw /openapi/v3 a few times, to see how long it takes to respond

Anything else we need to know?

No response

Host cluster Kubernetes version

$ kubectl version
# paste output here

Host cluster Kubernetes distribution

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

vlcuster version

vcluster version 0.19.4

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k3s

OS and Arch

amd64

[FabianKramm]

Hey @L1ghtman2k ! Thanks for creating this issue! Does disabling the proxy metrics server help? Also is it faster within a smaller vCluster?

[L1ghtman2k]

@FabianKramm.
max time to complete the call:
With metrics-server: ~60s
Without metrics server: ~0.61 seconds

I should also mention that our vcluster runs on top of microk8s

[everflux]

Could this be the cause for #1589 as well?

[FabianKramm]

@L1ghtman2k thanks for the info, with what helm values are you deploying vClsuter with and without metrics server?

[johannesfrey]

@L1ghtman2k Following the request from Fabian above, which backing store are you using for your virtual cluster? The amount of CRDs and potential expensive discovery requests resulting from this might strain the API server and its underlying backing store (See: #1589 (comment))

[heiko-braun]

Quick heads up, I think there a changes in 0.20.0-beta.12 that should improve the performance significantly. @L1ghtman2k once you get a chance to upgrade would you mind letting me know if you see a difference?

[L1ghtman2k]

Quick heads up, I think there a changes in 0.20.0-beta.12 that should improve the performance significantly. @L1ghtman2k once you get a chance to upgrade would you mind letting me know if you see a difference?

Thanks, I will take a look when we do the upgrade