docs: add Federated Search setup guide for SVA-C3 with NGINX Ingress using Splunk Operator for Kubernetes

[vivekr-splunk]

Adds a step-by-step Federated Search setup guide for SVA-C3 (Search Head Cluster + Indexer Cluster) using the Splunk Operator for Kubernetes, with NGINX Ingress between LOCAL and REMOTE SHCs. The guide is generalized from a path validated with the NCDOC customer. It covers prerequisites, REMOTE preparation, LOCAL configuration via a lightweight app and AppFramework, manual install, validation, troubleshooting, security, performance notes, and maintenance.

Key Changes

• New guide: docs/federated-search/federated-search-setup.md

• Optional, if your repo uses a docs index or nav

  • Add a link in docs/README.md
  • Update mkdocs.yml navigation

Testing and Verification

• Verified Markdown renders in GitHub preview

• Ran link checks on internal anchors and external references

• Linted Markdown locally

• Validated YAML snippets for basic syntax in fenced blocks

• Smoke tested representative commands on an SVA-C3 dev environment

  • Created role and service account on REMOTE
  • Confirmed management API reachability through NGINX Ingress
  • Verified federated provider and federated index resolution on LOCAL
  • Confirmed example searches return results from federated:r_audit

• No automated tests added, documentation only

Related Issues

• Jira, CSPL-4134: SOK-#### Document Federated Search setup for SVA-C3 using SOK with NGINX Ingress
• Add related GitHub issue IDs if applicable

PR Checklist

• Code changes adhere to the project's coding standards.
  N/A, documentation only.
• Relevant unit and integration tests are included.
  N/A, documentation only.
• Documentation has been updated accordingly.
• All tests pass locally.
  Markdown lint and link checks pass.
• The PR description follows the project's guidelines.

[Copilot]

Pull Request Overview

This PR introduces a comprehensive guide for configuring Federated Search between LOCAL and REMOTE Splunk Search Head Clusters (SHCs) in a Kubernetes environment using the Splunk Operator and NGINX Ingress. The guide enables users to query data from a remote cluster as if it were local, facilitating cross-cluster search capabilities.

Key Changes:

• Step-by-step setup instructions for REMOTE cluster preparation (service accounts, roles, NGINX Ingress configuration)
• LOCAL cluster configuration using AppFramework with federated provider and index definitions
• Validation procedures, troubleshooting guidance, and maintenance workflows

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Multiple -d flags with the same parameter name srchIndexesAllowed may not work as intended. Consider using comma-separated values in a single parameter: -d 'srchIndexesAllowed=_audit,demo' or verify that the Splunk API supports this pattern.

Suggested change

	-d "srchIndexesAllowed=_audit" \
	-d "srchIndexesAllowed=demo" \
	-d "srchIndexesAllowed=_audit,demo" \

[Copilot]

Using -d 'capabilities=search' will replace all existing capabilities on the admin role with only 'search', potentially breaking admin functionality. Use add_capabilities or append with += if the API supports it, or document that this overwrites all capabilities.

Suggested change

	Fix the permission issue by adding the `search` capability:
	```bash
	kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
	-u "admin:$REMOTE_ADMIN" \
	-X POST "https://localhost:8089/services/authorization/roles/admin" \
	-d "capabilities=search"
	Fix the permission issue by adding the `search` capability (without overwriting existing admin capabilities):
	> **Note:** Using `add_capabilities=search` will add the `search` capability to the admin role without removing existing capabilities. Do **not** use `capabilities=search`, as that will overwrite all current capabilities and may break admin functionality.
	```bash
	kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
	-u "admin:$REMOTE_ADMIN" \
	-X POST "https://localhost:8089/services/authorization/roles/admin" \
	-d "add_capabilities=search"

[Copilot]

Using -d 'capabilities=search' will replace all existing capabilities on the admin role with only 'search', potentially breaking admin functionality. Use add_capabilities or append with += if the API supports it, or document that this overwrites all capabilities.

Suggested change

	-d "capabilities=search"
	-d "add_capabilities=search"

[Copilot]

Using -d 'capabilities=search' will replace all existing capabilities on the admin role with only 'search', potentially breaking admin functionality. Use add_capabilities or append with += if the API supports it, or document that this overwrites all capabilities.

Suggested change

	**Solution:**
	```bash
	# On BOTH LOCAL and REMOTE clusters
	kubectl -n $NAMESPACE exec <pod-name> -c splunk -- curl -sk \
	-u "admin:<password>" \
	-X POST "https://localhost:8089/services/authorization/roles/admin" \
	**Solution:**
	> **Note:** Using `-d "capabilities=search"` will overwrite all existing capabilities for the admin role, which may break admin functionality.
	> If your Splunk API supports appending capabilities, use `add_capabilities=search` instead to safely add the `search` capability without removing others:
	```bash
	# On BOTH LOCAL and REMOTE clusters
	kubectl -n $NAMESPACE exec <pod-name> -c splunk -- curl -sk \
	-u "admin:<password>" \
	-X POST "https://localhost:8089/services/authorization/roles/admin" \
	-d "add_capabilities=search"

If add_capabilities is not supported, back up your current admin capabilities first and be aware that the following will replace all capabilities with only search:

# On BOTH LOCAL and REMOTE clusters
kubectl -n $NAMESPACE exec <pod-name> -c splunk -- curl -sk \
  -u "admin:<password>" \
  -X POST "https://localhost:8089/services/authorization/roles/admin" \

[Copilot]

This command will replace all existing srchIndexesAllowed values with only 'new_index', removing access to previously configured indexes like '_audit' and 'demo'. Use an HTTP GET to retrieve current values first, or document that users must include all desired indexes in the command.

[coveralls]

Pull Request Test Coverage Report for Build 18706609002

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.008%) to 86.544%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/splunk/enterprise/afwscheduler.go	1	92.9%

Totals
Change from base Build 18653942794:	-0.008%
Covered Lines:	10709
Relevant Lines:	12374

---

💛 - Coveralls

[kasiakoziol]

Does this document apply only to Azure? If so, would be good to mention it above at the beginning of the doc.

[kasiakoziol]

Same question about nginx

[kasiakoziol]

It returns 404

[rlieberman-splunk]

Is this guaranteed to be search-head-0, or should we update the documentation to find the captain search head?