CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation #1550

kasiakoziol · 2025-07-28T08:35:28Z

Description

This PR implements a separation between ingestion and indexing services within the Splunk Operator for Kubernetes. The goal is to enable the operator to independently manage the ingestion service while maintaining seamless integration with the indexing service.

Key Changes

Introduction of new Custom Resource Definition for IngestorCluster
Support for Remote Queue and Pipeline inputs in IngestorCluster and IndexerCluster

Testing and Verification

These changed were tested manually and as a part of the further enhancements, automated tests will be implemented.

Related Issues

Jira Epic: https://splunk.atlassian.net/browse/CSPL-3549

Notes

PR Checklist

Code changes adhere to the project's coding standards.
Relevant unit and integration tests are included.
Documentation has been updated accordingly.
All tests pass locally.
The PR description follows the project's guidelines.

coveralls · 2025-07-29T10:52:58Z

Pull Request Test Coverage Report for Build 17610018166

Details

332 of 467 (71.09%) changed or added relevant lines in 9 files are covered.
6 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-0.5%) to 86.098%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/splunk/client/enterprise.go	25	29	86.21%
pkg/splunk/enterprise/util.go	10	14	71.43%
internal/controller/ingestorcluster_controller.go	70	76	92.11%
pkg/splunk/enterprise/indexercluster.go	62	75	82.67%
pkg/splunk/enterprise/ingestorcluster.go	146	254	57.48%

Files with Coverage Reduction	New Missed Lines	%
pkg/splunk/enterprise/afwscheduler.go	1	92.93%
pkg/splunk/enterprise/cp.go	1	33.33%
pkg/splunk/enterprise/indexercluster.go	4	73.4%

Totals
Change from base Build 17296474155:	-0.5%
Covered Lines:	11049
Relevant Lines:	12833

💛 - Coveralls

rlieberman-splunk · 2025-08-20T14:13:58Z

api/v4/ingestorcluster_types.go

+	Replicas int32 `json:"replicas"`
+
+	// Splunk Enterprise app repository that specifies remote app location and scope for Splunk app management
+	AppFrameworkConfig AppFrameworkSpec `json:"appRepo,omitempty"`


This doesn't have to happen right now, but when we do more productization, we should update the AppFramework and CustomResources docs to include this CR.

rlieberman-splunk · 2025-08-20T14:23:44Z

config/crd/patches/additional_supported_versions_patch_ingestorclusters.yaml

+- op: add
+  path: "/spec/versions/-"
+  value:
+    name: v1


Will this be supported for v1 and v2? I don't see these files for clustermanager and licensemanager for example, which were later CRs.

It got generated by make command as far as I remember. Will revisit it later.

config/rbac/ingestorcluster_editor_role.yaml

rlieberman-splunk · 2025-08-20T14:25:12Z

config/rbac/ingestorcluster_viewer_role.yaml

+# without permissions to modify them. It is ideal for monitoring purposes and limited-access viewing.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole


We will probably need to add this to the helm chart as well: https://github.com/splunk/splunk-operator/tree/main/helm-chart/splunk-operator/templates/rbac

rlieberman-splunk · 2025-08-20T14:25:19Z

config/rbac/role.yaml

  - clustermanagers
  - clustermasters
  - indexerclusters
+  - ingestorclusters


We will probably need to add this to the helm chart as well: https://github.com/splunk/splunk-operator/tree/main/helm-chart/splunk-operator/templates/rbac

pkg/splunk/enterprise/ingestorcluster.go

vivekr-splunk

few changes are requested

vivekr-splunk · 2025-10-15T06:01:25Z

cmd/main.go

 	utilruntime.Must(clientgoscheme.AddToScheme(scheme))
 	utilruntime.Must(enterpriseApi.AddToScheme(scheme))
 	utilruntime.Must(enterpriseApiV3.AddToScheme(scheme))
+	utilruntime.Must(enterprisev4.AddToScheme(scheme))


we already have utilruntime.Must(enterpriseApi.AddToScheme(scheme)) which points to v4

vivekr-splunk · 2025-10-15T06:02:50Z

config/rbac/ingestorcluster_viewer_role.yaml

+# without permissions to modify them. It is ideal for monitoring purposes and limited-access viewing.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole


vivekr-splunk · 2025-10-15T06:03:17Z

config/rbac/role.yaml

  - clustermanagers
  - clustermasters
  - indexerclusters
+  - ingestorclusters


vivekr-splunk · 2025-10-15T06:05:45Z

pkg/splunk/client/enterprise.go

+
+	request, err := http.NewRequest("POST", endpoint, strings.NewReader(body))
+	if err != nil {
+		return err


log error here, also we need to make sure to add events which shows stages of CR status.

vivekr-splunk · 2025-10-15T06:07:18Z

pkg/splunk/enterprise/indexercluster.go

+		if cr.Spec.PullBus.Type != "" {
+			err = mgr.handlePullBusOrPipelineConfigChange(ctx, cr, client)
+			if err != nil {
+				scopedLog.Error(err, "Failed to update conf file for PullBus/Pipeline config change after pod creation")


add an k8s event here ( or at least one failure) and stages of Ingestor state

vivekr-splunk · 2025-10-15T06:10:35Z

pkg/splunk/enterprise/indexercluster.go

+var newSplunkClientForPullBusPipeline = splclient.NewSplunkClient
+
+// Checks if only PullBus or Pipeline config changed, and updates the conf file if so
+func (mgr *indexerClusterPodManager) handlePullBusOrPipelineConfigChange(ctx context.Context, newCR *enterpriseApi.IndexerCluster, k8s client.Client) error {


Note: Not related to this feature , but on longer run we need to find a way to bring these changes to SOK https://github.com/splunk/splunk-operator/pull/1193/files. it just simplifies how we manage Rest calls

vivekr-splunk · 2025-10-15T06:10:59Z

pkg/splunk/enterprise/indexercluster.go

+	return updateErr
+}
+
+func getChangedPullBusAndPipelineFieldsIndexer(newCR *enterpriseApi.IndexerCluster) (pullBusChangedFieldsInputs, pullBusChangedFieldsOutputs, pipelineChangedFields [][]string) {


add description to the function

vivekr-splunk · 2025-10-15T06:12:04Z

pkg/splunk/enterprise/types.go

 	case SplunkMonitoringConsole:
 		role = "splunk_monitor"
+	case SplunkIngestor:
+		role = "splunk_standalone" // TODO: change this to a new role when we have one (splunk_ingestor)


I thought @Igor-splunk added role to ansible

vivekr-splunk · 2025-10-15T06:12:25Z

pkg/splunk/enterprise/types_test.go

 		SplunkLicenseMaster:     splcommon.LicenseManagerRole,
 		SplunkLicenseManager:    splcommon.LicenseManagerRole,
 		SplunkMonitoringConsole: "splunk_monitor",
+		SplunkIngestor:          "splunk_standalone", // TODO: change this to a new role when we have one (splunk_ingestor)


same here , have we added new role to ansible

CSPL-3551 Init IngestorCluster CR implementation

0e36568

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 6 times, most recently from ca230a9 to eb21049 Compare July 28, 2025 15:15

CSPL-3551 Enhancing Ingestor inputs

2cca0d7

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from eb21049 to 2cca0d7 Compare July 29, 2025 08:24

kasiakoziol added 3 commits July 29, 2025 10:25

Merge branch 'develop' into CSPL-3551-ingestion-cr

f747ab4

fix

9a7503c

fix

7134350

CSPL-3551 Adding tests

44349fa

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from d3779b0 to 44349fa Compare July 29, 2025 14:54

kasiakoziol added 2 commits July 31, 2025 13:56

CSPL-3551 Mound defaults and update them with no Splunk restart

e71ea9c

Merge branch 'develop' into CSPL-3551-ingestion-cr

14ccef4

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 13 times, most recently from 3b2cf1c to 5cad7f9 Compare August 5, 2025 08:49

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 3 times, most recently from 8e32bd1 to 4df319e Compare August 19, 2025 15:39

rlieberman-splunk reviewed Aug 20, 2025

View reviewed changes

config/rbac/ingestorcluster_editor_role.yaml Show resolved Hide resolved

rlieberman-splunk reviewed Aug 20, 2025

View reviewed changes

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 3 times, most recently from f2d467a to 713eac7 Compare August 21, 2025 13:45

CSPL-3551 Fixes

1c77634

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from 713eac7 to 1c77634 Compare August 21, 2025 13:46

kasiakoziol requested review from Igor-splunk, gabrielm-splunk, patrykw-splunk and vivekr-splunk August 21, 2025 14:07

kasiakoziol changed the title ~~Cspl 3551 ingestion cr~~ CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation Aug 21, 2025

CSPL-3560 Initial docs for I&I separation

fb4e87f

kasiakoziol commented Aug 29, 2025

View reviewed changes

pkg/splunk/enterprise/ingestorcluster.go Outdated Show resolved Hide resolved

Merge branch 'develop' into CSPL-3551-ingestion-cr

3e376d8

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 2 times, most recently from b12dbb4 to 68e8c7c Compare September 1, 2025 12:06

CSPL-3551 Update documentation to reflect on Grafana

e62f381

kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from 68e8c7c to e62f381 Compare September 1, 2025 12:15

kasiakoziol added 2 commits September 10, 2025 11:52

CSPL-3556 Unit tests

84d8491

CSPL-3972 Addressing TODOs

c3b3aa6

kasiakoziol mentioned this pull request Sep 19, 2025

CSPL-3972 Addressing TODOs #1576

Open

5 tasks

Merge branch 'develop' into CSPL-3551-ingestion-cr

d3b2ad6

vivekr-splunk reviewed Oct 15, 2025

View reviewed changes

CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation #1550

Are you sure you want to change the base?

CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation #1550

Uh oh!

Conversation

kasiakoziol commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Testing and Verification

Related Issues

Notes

PR Checklist

Uh oh!

coveralls commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 17610018166

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vivekr-splunk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kasiakoziol commented Jul 28, 2025 •

edited

Loading

coveralls commented Jul 29, 2025 •

edited

Loading