[CORE-15019] Consider group membership for authz checks #29157

michael-redpanda · 2026-01-06T18:16:04Z

Summary

This PR implements OIDC group-based authorization for Redpanda, allowing administrators to grant topic access permissions based on group membership claims from identity providers like Keycloak. Users in specific IdP groups can now be authorized to access topics via Group:<group-name> ACL principals, without needing to configure per-user ACLs.

Changes

Core Security Changes

Group claim extraction: Added processing of the groups claim from OIDC/JWT tokens, configurable via oidc_group_claim_path
Nested group handling: Added nested_group_behavior config option (none or suffix) to control how hierarchical group paths are processed
Authorization integration: Extended the authorizer to evaluate Group:* ACL principals against a user's group membership
Proto update: Added groups field to ResolveOidcIdentityResponse to expose resolved group memberships via the admin API

Kafka Connection Flow

Extended sasl_mechanism with a groups() method to retrieve group memberships
Added groups to connection_context and request_auth to flow group info through authorization checks
Updated authorized_user method to accept groups parameter for ACL evaluation

Ducktape Test Infrastructure

Added new helper methods to the Keycloak test service:

create_group_mapper() - Creates an OIDC protocol mapper to include groups in tokens
create_group() - Creates groups in Keycloak
add_service_user_to_group() - Adds a service account to a group
remove_service_user_from_group() - Removes a service account from a group

New Tests

test_group_claim: Matrix test validating group claim mapping with different full_group and nested_group_mode combinations
test_group_membership_change: End-to-end test verifying that dynamic group membership changes correctly affect topic visibility:
- Phase 1: User in group1 → can see topic1 only
- Phase 2: User in group2 → can see topic2 only
- Phase 3: User in group3 (no permissions) → cannot see any topics
- Phase 4: User in all groups → can see both topics

Backports Required

Release Notes

none

Copilot

Pull request overview

This PR implements OIDC group-based authorization for Redpanda, enabling administrators to grant topic access permissions based on OIDC group membership claims. The implementation adds support for extracting group information from OIDC tokens, configuring how nested groups are handled, and authorizing requests based on Group:* ACL principals. The changes span both core security logic and test infrastructure.

Key Changes

Extended OIDC authentication to extract and propagate group membership claims throughout the authorization pipeline
Modified the authorizer to evaluate Group:* ACL principals alongside user and role principals, with proper precedence handling (deny ACLs take priority)
Added comprehensive C++ unit tests and ducktape integration tests to validate group-based authorization behavior

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`tests/rptest/tests/redpanda_oauth_test.py`	Added two ducktape tests validating group claim mapping and dynamic group membership changes
`tests/rptest/services/keycloak.py`	Added helper methods for creating group mappers and managing service user group membership
`tests/rptest/clients/admin/proto/redpanda/core/admin/v2/security_pb2.pyi`	Updated protobuf type stubs to include groups field in OIDC identity response
`tests/rptest/clients/admin/proto/redpanda/core/admin/v2/security_pb2.py`	Updated generated protobuf code for groups field
`tests/rptest/clients/admin/proto/redpanda/core/admin/v2/__init__.pyi`	Added internal module import
`src/v/security/tests/role_store_bench.cc`	Updated benchmark tests to pass empty groups parameter
`src/v/security/tests/authorizer_test.cc`	Added 15 new test cases for group authorization and updated existing tests to pass groups parameter
`src/v/security/sasl_authentication.h`	Added virtual `groups()` method to `sasl_mechanism` base class
`src/v/security/request_auth.h`	Added groups field to `request_auth_result` class
`src/v/security/request_auth.cc`	Updated authentication to extract and propagate groups from OIDC tokens
`src/v/security/oidc_authenticator.h`	Implemented `groups()` override in OIDC SASL authenticator
`src/v/security/authorizer.h`	Added group field to `auth_result` and groups parameter to authorization methods
`src/v/security/authorizer.cc`	Refactored authorization logic to evaluate groups alongside users and roles
`src/v/security/audit/schemas/tests/ocsf_schemas_test.cc`	Updated audit schema tests to pass empty groups
`src/v/redpanda/admin/services/security.cc`	Added groups to OIDC identity resolution admin API response
`src/v/pandaproxy/schema_registry/authorization.cc`	Updated schema registry authorization to pass groups
`src/v/kafka/server/connection_context.h`	Added groups parameter to authorization methods
`src/v/kafka/server/connection_context.cc`	Implemented `get_groups()` and passed groups to authorizer
`proto/redpanda/core/admin/v2/security.proto`	Added groups field to `ResolveOidcIdentityResponse` proto definition

Comments suppressed due to low confidence (1)

tests/rptest/tests/redpanda_oauth_test.py:1

The next() call will raise StopIteration if no group matches the name, but the code then checks if group is None. This logic is unreachable - if no group is found, the exception occurs before the None check. Use next(g for g in groups if g["name"] == group_name, None) with a default value to properly handle the case when no group is found.

# Copyright 2023 Redpanda Data, Inc.

tests/rptest/tests/redpanda_oauth_test.py

src/v/security/authorizer.cc

src/v/security/request_auth.h

tests/rptest/tests/redpanda_oauth_test.py

michael-redpanda · 2026-01-06T18:23:27Z

Force push:

Update test to verify we see the appropriate groups in the resolve response
Update comments per bot

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Copilot · 2026-01-06T18:24:31Z

tests/rptest/services/keycloak.py

+
+    def _get_group_id(self, group_name: str) -> str | None:
+        groups = self.kc_admin.get_groups()
+        group = next(g for g in groups if g["name"] == group_name)


The next() call on line 203 will raise StopIteration if no group with the given name is found, but line 204 checks if group is None, which is unreachable. Use next((...), None) with a default value to properly handle the case when no group is found.

Suggested change

group = next(g for g in groups if g["name"] == group_name)

group = next((g for g in groups if g["name"] == group_name), None)

src/v/security/authorizer.cc

src/v/security/tests/authorizer_test.cc

michael-redpanda · 2026-01-06T21:05:33Z

Force push:

Addressed assert issue

vbotbuildovich · 2026-01-06T22:58:08Z

CI test results

test results on build#78606

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
JavaCompressionTest	test_upgrade_java_compression	{"compression_type": "snappy"}	integration	https://buildkite.com/redpanda/redpanda/builds/78606#019b9541-ac43-433f-8daa-9ff2c87b7407	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=JavaCompressionTest&test_method=test_upgrade_java_compression
PandaproxyAuthSecurityReportTest	test_security_report	{"auto_auth": false, "enable_auth": true}	integration	https://buildkite.com/redpanda/redpanda/builds/78606#019b9541-ac46-4997-92b7-0ee46fae15fc	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=PandaproxyAuthSecurityReportTest&test_method=test_security_report
WriteCachingFailureInjectionE2ETest	test_crash_all	{"use_transactions": false}	integration	https://buildkite.com/redpanda/redpanda/builds/78606#019b9541-ac45-4e77-b6be-d6b8a7b5fcde	FLAKY	8/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.1198, p0=0.3408, reject_threshold=0.0100. adj_baseline=0.3180, p1=0.3363, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=WriteCachingFailureInjectionE2ETest&test_method=test_crash_all

BenPope

I didn't look heavily at the tests (but coverage is great), or keycloak stuff. LGTM.

BenPope · 2026-01-07T15:32:58Z

src/v/kafka/server/connection_context.cc

+      name,
+      quiet,
+      superuser_required,
+      get_groups());


Not a huge fan of the ordering here, I feel like it should be principal, groups, ... , but this overload set is tricky.

src/v/security/authorizer.cc

src/v/redpanda/admin/services/security.cc

michael-redpanda · 2026-01-07T17:26:37Z

Force push:

Addressed PR comments

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Copilot · 2026-01-07T17:27:20Z

tests/rptest/services/keycloak.py

        self.logger.debug(f"client_id: {id}")
        return id

+    def create_group_mapper(self, client_id: str, use_full_path: bool = True):


Add a docstring explaining the purpose of this method, including what the use_full_path parameter controls and how it affects the group claims in tokens.

Suggested change

def create_group_mapper(self, client_id: str, use_full_path: bool = True):

def create_group_mapper(self, client_id: str, use_full_path: bool = True):

"""

Ensure that the client has an OIDC group membership protocol mapper.

The mapper adds the client's group memberships to the ``groups`` claim

in ID, access, and userinfo tokens. When ``use_full_path`` is ``True``,

groups are emitted as their full path (for example ``/parent/child``);

when ``False``, only the simple group name is used. This flag is

propagated to the Keycloak mapper's ``full.path`` configuration.

"""

Copilot · 2026-01-07T17:27:21Z

tests/rptest/services/keycloak.py

+        parent: str | None = None,
+        skip_exists: bool = True,
+        **kwargs,
+    ) -> str:


Add a docstring explaining what this method does, especially the meaning of skip_exists and parent parameters, and what the return value represents.

Suggested change

) -> str:

) -> str:

"""

Create a Keycloak group and return its identifier.

This method creates a group with the given ``group_name`` and any additional

attributes provided via ``kwargs``.

:param group_name: Name of the group to create.

:param parent: Optional identifier of the parent group under which the new

group should be created. If ``None``, the group is created at the root

level of the realm.

:param skip_exists: If ``True``, treat an already existing group with the

same name as success and return its ID. If ``False``, a failure to

create the group (for example, because it already exists) will cause a

``RuntimeError`` to be raised.

:param kwargs: Additional fields to include in the group representation

passed to Keycloak.

:return: The Keycloak identifier of the created group, or of the existing

group if ``skip_exists`` is ``True`` and the group already exists.

:raises RuntimeError: If the group cannot be created and ``skip_exists`` is

``False``.

"""

To be used by OIDC authenticator to return the list of groups present in the OIDC token. Signed-off-by: Michael Boquard <[email protected]>

Signed-off-by: Michael Boquard <[email protected]>

Consolidate the authorization logic by building an effective_principals list that includes the user and their roles. Simplify the check_access function to work with principal views directly, eliminating the separate check_role_access function. This reduces duplication and clarifies the ACL matching flow across both allow and deny permissions. Signed-off-by: Michael Boquard <[email protected]>

Signed-off-by: Michael Boquard <[email protected]>

This mapper is used to insert group claims into OIDC tokens Signed-off-by: Michael Boquard <[email protected]>

Signed-off-by: Michael Boquard <[email protected]>

michael-redpanda · 2026-01-07T17:54:49Z

Force push:

Rebased off of dev to fix merge conflict

nguyen-andrew

Nice tests and refactor of authorizer::do_authorized!

nguyen-andrew · 2026-01-07T22:55:19Z

src/v/security/tests/authorizer_test.cc

+    // Create many groups (simulate realistic scenario)
+    chunked_vector<acl_principal> many_groups;
+    std::vector<acl_binding> bindings;
+
+    for (int i = 0; i < 50; ++i) {
+        acl_principal group(principal_type::group, fmt::format("group{}", i));
+        many_groups.push_back(group);
+
+        // Only group42 has permissions
+        if (i == 42) {
+            acl_entry allow_read(
+              group,
+              acl_host::wildcard_host(),
+              acl_operation::read,
+              acl_permission::allow);
+
+            resource_pattern resource(
+              resource_type::topic, default_topic(), pattern_type::literal);
+            bindings.emplace_back(resource, allow_read);
+        }
+    }


nit tweak

Suggested change

// Create many groups (simulate realistic scenario)

chunked_vector<acl_principal> many_groups;

std::vector<acl_binding> bindings;

for (int i = 0; i < 50; ++i) {

acl_principal group(principal_type::group, fmt::format("group{}", i));

many_groups.push_back(group);

// Only group42 has permissions

if (i == 42) {

acl_entry allow_read(

group,

acl_host::wildcard_host(),

acl_operation::read,

acl_permission::allow);

resource_pattern resource(

resource_type::topic, default_topic(), pattern_type::literal);

bindings.emplace_back(resource, allow_read);

}

}

constexpr auto num_groups = 50;

constexpr auto selected_group_idx = 42;

static_assert(selected_group_idx < num_groups, "selected_group_idx must be less than num_groups");

// Create many groups (simulate realistic scenario)

chunked_vector<acl_principal> many_groups;

std::vector<acl_binding> bindings;

for (auto i : std::views::iota(0, num_groups)) {

many_groups.emplace_back(

principal_type::group,

ssx::sformat("group{}", i));

}

// Only selected group has permissions

acl_entry allow_read(

many_groups[selected_group_idx],

acl_host::wildcard_host(),

acl_operation::read,

acl_permission::allow);

resource_pattern resource(

resource_type::topic, default_topic(), pattern_type::literal);

bindings.emplace_back(resource, allow_read);

BenPope

LGTM iff:

Call append_roles for groups is intended for a future PR
The perf regression is acceptable

BenPope · 2026-01-08T11:50:47Z

src/v/security/authorizer.cc


+    std::ranges::for_each(groups, [&effective_principals](const auto& g) {
+        effective_principals.emplace_back(g);
+        // TODO(gbac) Call append_roles for groups


Is this intended for another PR?

correct - we still don't support groups in roles but that's coming very soon

BenPope · 2026-01-08T11:57:31Z

src/v/security/authorizer.cc

+
+    // Only users can be a member of roles, not ephemeral_users
+    if (principal.type() == principal_type::user) {
+        append_roles(principal);


Note: There is a potential performance regression here: The collection of roles is now performed eagerly, even if the match would have been on the principal rather than a role.

hmm that's a fair point... maybe the principal needs to be checked first, then groups, and then build the roles list... wdyt?

I've gone ahead and filed CORE-15238 to come back to this - trying to get this into the QE teams hands asap for system testing and this PR goes a long way to getting them started.

michael-redpanda requested review from BenPope and nguyen-andrew January 6, 2026 18:16

michael-redpanda self-assigned this Jan 6, 2026

michael-redpanda requested a review from rockwotj as a code owner January 6, 2026 18:16

Copilot AI review requested due to automatic review settings January 6, 2026 18:16

michael-redpanda requested a review from a team as a code owner January 6, 2026 18:16

github-actions bot added the area/redpanda label Jan 6, 2026

Copilot AI reviewed Jan 6, 2026

View reviewed changes

tests/rptest/tests/redpanda_oauth_test.py Show resolved Hide resolved

src/v/security/authorizer.cc Show resolved Hide resolved

src/v/security/request_auth.h Show resolved Hide resolved

tests/rptest/tests/redpanda_oauth_test.py Outdated Show resolved Hide resolved

michael-redpanda force-pushed the gbac/core-14894/core-15019-group-authorizer branch from df0f13b to f07ac25 Compare January 6, 2026 18:23

michael-redpanda requested a review from Copilot January 6, 2026 18:23

Copilot AI reviewed Jan 6, 2026

View reviewed changes

michael-redpanda force-pushed the gbac/core-14894/core-15019-group-authorizer branch from f07ac25 to 91a8bba Compare January 6, 2026 21:05

BenPope reviewed Jan 7, 2026

View reviewed changes

michael-redpanda force-pushed the gbac/core-14894/core-15019-group-authorizer branch from 91a8bba to b4ec01d Compare January 7, 2026 17:26

michael-redpanda requested review from BenPope and Copilot January 7, 2026 17:26

Copilot AI reviewed Jan 7, 2026

View reviewed changes

michael-redpanda added 9 commits January 7, 2026 12:53

security: Add groups() method to sasl_mechanism

20d1a0b

To be used by OIDC authenticator to return the list of groups present in the OIDC token. Signed-off-by: Michael Boquard <[email protected]>

security/oidc: Implement groups method

cbc7c84

Signed-off-by: Michael Boquard <[email protected]>

k/server: Add get_groups method to connection_context

ee219e8

Signed-off-by: Michael Boquard <[email protected]>

k/server: Add groups paramter to authorized_user method

82d24e3

Signed-off-by: Michael Boquard <[email protected]>

security: Add groups to request_auth result

cf5bcea

Signed-off-by: Michael Boquard <[email protected]>

security: Add groups to authorized method

eb27943

Signed-off-by: Michael Boquard <[email protected]>

security: Created authz result for groups

d03d1c7

Signed-off-by: Michael Boquard <[email protected]>

security: Handle groups in authz checks

f73689c

Signed-off-by: Michael Boquard <[email protected]>

michael-redpanda added 6 commits January 7, 2026 12:53

dt: Add method to create group_mapper

8d5b860

This mapper is used to insert group claims into OIDC tokens Signed-off-by: Michael Boquard <[email protected]>

dt: Add method to create groups in keycloak

f3fb23e

Signed-off-by: Michael Boquard <[email protected]>

dt: Add method to add a service account to a group

372ba79

Signed-off-by: Michael Boquard <[email protected]>

proto: Add groups to ResolveOidcIdentityResponse

fd3ac57

Signed-off-by: Michael Boquard <[email protected]>

admin/security: Implement groups

728e57d

Signed-off-by: Michael Boquard <[email protected]>

dt: Added basic group authz test

53d94c1

Signed-off-by: Michael Boquard <[email protected]>

michael-redpanda force-pushed the gbac/core-14894/core-15019-group-authorizer branch from b4ec01d to 53d94c1 Compare January 7, 2026 17:54

nguyen-andrew reviewed Jan 7, 2026

View reviewed changes

BenPope approved these changes Jan 8, 2026

View reviewed changes

michael-redpanda merged commit 7c08c4a into redpanda-data:dev Jan 8, 2026
30 checks passed

	group = next(g for g in groups if g["name"] == group_name)
	group = next((g for g in groups if g["name"] == group_name), None)

-    def create_group_mapper(self, client_id: str, use_full_path: bool = True):
+    def create_group_mapper(self, client_id: str, use_full_path: bool = True):
+        """
+        Ensure that the client has an OIDC group membership protocol mapper.
+        The mapper adds the client's group memberships to the ``groups`` claim
+        in ID, access, and userinfo tokens. When ``use_full_path`` is ``True``,
+        groups are emitted as their full path (for example ``/parent/child``);
+        when ``False``, only the simple group name is used. This flag is
+        propagated to the Keycloak mapper's ``full.path`` configuration.
+        """

-    ) -> str:
+    ) -> str:
+        """
+        Create a Keycloak group and return its identifier.
+        This method creates a group with the given ``group_name`` and any additional
+        attributes provided via ``kwargs``.
+        :param group_name: Name of the group to create.
+        :param parent: Optional identifier of the parent group under which the new
+            group should be created. If ``None``, the group is created at the root
+            level of the realm.
+        :param skip_exists: If ``True``, treat an already existing group with the
+            same name as success and return its ID. If ``False``, a failure to
+            create the group (for example, because it already exists) will cause a
+            ``RuntimeError`` to be raised.
+        :param kwargs: Additional fields to include in the group representation
+            passed to Keycloak.
+        :return: The Keycloak identifier of the created group, or of the existing
+            group if ``skip_exists`` is ``True`` and the group already exists.
+        :raises RuntimeError: If the group cannot be created and ``skip_exists`` is
+            ``False``.
+        """

[CORE-15019] Consider group membership for authz checks #29157

[CORE-15019] Consider group membership for authz checks #29157

Uh oh!

Conversation

michael-redpanda commented Jan 6, 2026

Summary

Changes

Core Security Changes

Kafka Connection Flow

Ducktape Test Infrastructure

New Tests

Backports Required

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-redpanda commented Jan 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

michael-redpanda commented Jan 6, 2026

Uh oh!

vbotbuildovich commented Jan 6, 2026

CI test results

Uh oh!

BenPope left a comment

Choose a reason for hiding this comment

Uh oh!

BenPope Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

michael-redpanda commented Jan 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

michael-redpanda commented Jan 7, 2026

Uh oh!

nguyen-andrew left a comment

Choose a reason for hiding this comment

Uh oh!

nguyen-andrew Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

BenPope left a comment

Choose a reason for hiding this comment

Uh oh!

BenPope Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

michael-redpanda Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

BenPope Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!