test: harden schema-registry mSSL test error matching - #36232
Merged
Conversation
These two `!` expected-failure assertions have been flaky for over a year because the exact TLS error text varies with the TLS backend, reqwest version, and OpenSSL build in use. The tests only care that the connection fails with a disallowed client certificate — the wording is "not under our control" (as the existing comment already notes). Precedent: PR #30501 (Nov 2024) loosened the same test family for the same reason, citing 16 CI failures over an inconsistent error string. Recent investigation on the reqwest 0.13 bump surfaced yet another variant (OpenSSL `PEM routines:get_name:no start line` on some runs). Rather than re-pin to a specific wording each time, switch to `contains:error sending request for url` — the outer reqwest wrapper that all failure modes share. Affected assertions (both in test/kafka-auth/): - test-schema-registry-mssl-basic.td: line 44 `!` — disallowed client cert against mssl-basic schema registry. - test-schema-registry-mssl.td: line 51 `!` — same scenario against mssl schema registry. No production code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jasonhernandez
marked this pull request as ready for review
April 23, 2026 22:24
def-
approved these changes
Apr 24, 2026
jasonhernandez
added a commit
that referenced
this pull request
Jul 7, 2026
The suite was disabled entirely in #36405 after unexplained PEM flakes (SS-115). While it was dark, the Confluent Platform 7.9.4 -> 8.2.0 bump (#36683) broke its setup in two ways that nothing caught: - CP 8.x ships Jetty 12, which moved the JAAS login modules from org.eclipse.jetty.jaas.spi to org.eclipse.jetty.security.jaas.spi. The schema registry basic-auth realm silently failed to load its login module and returned 401 for every request, correct credentials included. Point the jaas.config at the new class path. - CP 8.x brokers negotiate TLS 1.3, where a missing or untrusted client certificate fails with different alert text (certificate required, alert 116) than TLS 1.2 (bad certificate, alert 42). The kafka mssl tests asserted the exact TLS 1.2 wording. Assert only that the broker rejected the connection with a TLS alert, since the exact text varies with TLS version and OpenSSL build. This is the third exact-text loosening in this suite (#30501, #36232). With both fixes the full suite passes locally against current main images. The original SS-115 PEM flake did not reproduce in local runs on either reqwest 0.12 (main) or the 0.13.4 branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 tasks
jasonhernandez
added a commit
that referenced
this pull request
Jul 7, 2026
… 8.x) (#37470) ## Summary Re-enables the Kafka auth test suite, disabled since #36405 (SS-115), and fixes the two breakages that accumulated while it was dark. ## What broke while the suite was disabled The Confluent Platform 7.9.4 → 8.2.0 bump (#36683) landed three weeks after the suite was skipped, and broke its setup in two ways nothing could catch: 1. **Schema registry basic auth returned 401 for everyone.** CP 8.x ships Jetty 12, which moved the JAAS login modules from `org.eclipse.jetty.jaas.spi` to `org.eclipse.jetty.security.jaas.spi`. Our `schema-registry.jaas.config` referenced the old class, so the login module never loaded and every request, correct credentials included, got 401. Verified directly: `curl -u materialize:sekurity` against the container → 401 before, 200 after the one-line class-path fix (wrong/missing creds still 401). 2. **Kafka mssl negative tests asserted TLS 1.2 alert text.** CP 8.x brokers negotiate TLS 1.3, where a missing/untrusted client certificate fails with `tlsv13 alert certificate required` (alert 116) instead of `ssl/tls alert bad certificate` (alert 42). The assertions now check only that the broker rejected the connection with a TLS alert. The exact wording is a function of TLS version and OpenSSL build that we do not control. This is the third exact-text loosening in this suite's history (#30501, #36232), so the loosening deliberately goes all the way rather than chasing the new strings. ## On the original SS-115 flake The `PEM routines:get_name:no start line` flake that got the suite disabled did not reproduce in local full-suite runs against current main images (reqwest 0.12) or against the reqwest 0.13.4 branch (#37469). Timeline evidence from SS-115 itself shows the flake continued after the reqwest 0.13 revert (#36241), so reqwest was likely misattributed. If it resurfaces, the `ci-regexp` in SS-115 will link it; the suite being enabled is what gives us the data to root-cause it. ## Why now Two in-flight workstreams change exactly the surface this suite covers: the reqwest 0.13 re-application (#37469) and the rdkafka → AWS-LC switch (#35941). Landing those with this suite dark is how auth/TLS regressions reach production unobserved. ## Test plan - [x] Full suite green locally against main images (all fixes applied) - [x] Repeated local runs to probe for the SS-115 flake: 8 consecutive local full-suite passes (5-run loop + 3 individual), plus 2 clean runs on the reqwest-0.12 main baseline. The PEM flake did not reproduce. - [ ] Kafka auth 1–3 green in this PR's CI (the un-skip makes them run here) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Jason Hernandez <7144515+jasonhernandez@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two
!assertions intest/kafka-auth/test-schema-registry-mssl{-basic,}.tdhave been flaky for over a year because the exact TLS error text varies with the TLS backend, reqwest version, and OpenSSL build in use. The tests only verify that a disallowed-client-cert connection fails — the wording is "not under our control" (as the comment in the tests already says).Switch both
contains:alert certificate unknowntocontains:error sending request for url— the outer reqwest wrapper that every failure mode we've observed shares.Precedent
Variants observed
ssl/tls alert certificate unknown(native-tls + OpenSSL, current main)alert bad certificate(pre-2024 wording, fixed by kafka auth test: Handle different error #30501)CertificateUnknown(rustls; from the crypto migration branch)PEM routines:get_name:no start line:crypto/pem/pem_lib.c:773:Expecting: CERTIFICATE(reqwest 0.13 on Linux, intermittent)Why separate from the reqwest bump
The reqwest 0.13 bump (#36215) surfaces the OpenSSL PEM variant more often, which is how I noticed this — but it's not the root cause. The flakiness is latent on
mainand will bite any subsequent TLS-stack change (rustls migration, OpenSSL updates). Hardening the assertion is independently justifiable.Test plan
bin/mzcompose --find kafka-auth run defaultpasses.alert certificate unknownassertions exist against these endpoints (verified viagrep).🤖 Generated with Claude Code