"Object 'Kind' is missing in 'null'"

[alex-berger]

What steps did you take and what happened:

After upgrading to Gatekeeper 3.18.0 we are observing below error message (exactly 4 messages every 2 minutes) and now I am wondering what might cause this.

{
    "level": "error",
    "ts": 1734272869.0688941,
    "logger": "webhook",
    "msg": "error while excluding namespace",
    "hookType": "validation",
    "error": "Object 'Kind' is missing in 'null'",
    "stacktrace": "github.com/open-policy-agent/gatekeeper/v3/pkg/webhook.(*validationHandler).Handle\n\t/go/src/github.com/open-policy-agent/gatekeeper/pkg/webhook/policy.go:172\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:119\nsigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2220\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:147\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2220\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:109\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2220\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2747\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3210\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2092"
}

Here a formatted version of the error message's stack trace:

github.com/open-policy-agent/gatekeeper/v3/pkg/webhook.(*validationHandler).Handle
    /go/src/github.com/open-policy-agent/gatekeeper/pkg/webhook/policy.go:172\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle
    /go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
    /go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:119\nsigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1
    /go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:60\nnet/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2220\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
    /go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:147\nnet/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2220\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
    /go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:109\nnet/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2220\nnet/http.(*ServeMux).ServeHTTP
    /usr/local/go/src/net/http/server.go:2747\nnet/http.serverHandler.ServeHTTP
    /usr/local/go/src/net/http/server.go:3210\nnet/http.(*conn).serve
    /usr/local/go/src/net/http/server.go:2092

It looks like this is because skipExcludedNamespace is calling deserializer.Decode(nil, ...) as req.Object.Raw seems to be nil.

func (h *webhookHandler) skipExcludedNamespace(req *admissionv1.AdmissionRequest, excludedProcess process.Process) (bool, error) {
	obj := &unstructured.Unstructured{}
	if _, _, err := deserializer.Decode(req.Object.Raw, nil, obj); err != nil {
		return false, err
	}
        ...
	return isNamespaceExcluded, err
}

What did you expect to happen:

No errors :-)

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Gatekeeper version: 3.18.0
• Kubernetes version: v1.30.6-eks-7f9249a

[ritazh]

Thanks for the report @alex-berger!
To help us reproduce this issue, how did you install Gatekeeper v3.18.0? via helm chart or deploy/gatekeeper.yaml?
There is a known issue: #3738 (comment) such that --operation=generate needs to be added to the audit deployment.

There are few reasons why req.Object.Raw could be nil:

• operation type is DELETE or CONNECT
• if a CRD does not have a proper schema or preserveUnknownFields: true, the object might fail to serialize
• bad webhook configuration
• if request does not modify or need the entire object data (like GET or LIST requests), the Object.Raw field might not be populated

To help further debug the issue, please check the k8s API server logs to inspect the request object that ended in error e.g. operation and if it's a CRD does it have a proper schema. And if you can share the resource to help us reproduce that would be great.

[alex-berger]

To help us reproduce this issue, how did you install Gatekeeper v3.18.0? via helm chart or deploy/gatekeeper.yaml?

@ritazh In our case Gatekeeper v3.18.0 was installed via Helm Chart, actually it was upgrade from version v3.15.1 (via FluxCD / GitOps).

To help further debug the issue, please check the k8s API server logs to inspect the request object that ended in error e.g. operation and if it's a CRD does it have a proper schema.

Interestingly, we have not found anything in the k8s API server logs that would indicated failing webhook calls. But, that might well be, because the webhook's Handler function is not failing but just logging the error and then continues (i.e. does not immediately return).

	isExcludedNamespace, err := h.skipExcludedNamespace(&req.AdmissionRequest, process.Webhook)
	if err != nil {
		h.log.Error(err, "error while excluding namespace")
	}

[JaydipGabani]

@alex-berger can you provide the steps to reproduce? I am not able to reproduce the same on my end.

Did you update to 3.18.1? if so, after 3.18.1 are you still facing the same issue?