Skip to content

CLAUDE.md

Latest commit

 

History

History
294 lines (231 loc) · 20.3 KB

CLAUDE.md

File metadata and controls

294 lines (231 loc) · 20.3 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is the main repository for Red Hat OpenShift on Azure (ARO) using the Hosted Control Planes (HCP) architecture. It contains some of the code for the required microservices along with most of the configuration and pipeline definition necessary to deploy it.

Common Commands

Build, Test, Lint (whole repo)

make test              # Run all unit tests across the Go workspace (requires envtest setup, done automatically)
make lint              # Run golangci-lint across all modules
make lint-fix          # Run lint with --fix
make fmt               # Run goimports across all modules
make verify            # Run all verification checks (deepcopy, json-format, yamlfmt, materialize, gomega, schema) — this is what CI runs
make verify-kql        # Validate KQL (Kusto Query Language) files
make generate          # Regenerate deepcopy, mocks, format, tidy
make generate-kiota    # Regenerate the ARM SDK (under test/sdk/), then run licenses + fmt
make tidy              # Run go mod tidy across all modules + go work sync
make install-tools     # Install all dev tools via bingo

Running a single test

There is no workspace-wide single-test shortcut. cd into the module directory and run:

cd frontend && go test -run TestMyFunction -v ./...
cd internal && go test -run TestSomething -v ./path/to/package/...

For integration tests (require Cosmos emulator):

cd test-integration && make test    # Runs all integration tests with Cosmos emulator
go test ./test-integration/frontend/...  # Frontend integration tests only
go test ./test-integration/backend/...   # Backend integration tests only

E2E tests (local)

make e2e/local                                  # Full local E2E suite
make e2e-local/run-test TEST_NAME="TestName"    # Single E2E test

Building individual services

make -C frontend build   # Build frontend binary
make -C backend build    # Build backend binary
make build-services      # Build and push all in-repo service images in parallel (requires ACR login)

Config changes

After modifying config/config*.yaml or schema files:

cd config && make materialize   # Re-render configs and update helm fixtures

Rendered config changes under config/rendered/ must be committed with your PR. Run make verify-materialize to check.

Personal dev environment

make personal-dev-env   # Build, push images, deploy infrastructure (DEPLOY_ENV=pers or swft required)

Other useful commands

make rebase                      # Rebase on upstream and re-materialize config (hack/rebase-n-materialize.sh)
make envtest-setup               # Download kubebuilder binaries (etcd, kube-apiserver) for controller tests
make record-nonlocal-e2e         # Regenerate nonlocal-e2e-specs.txt after adding/removing/renaming E2E tests
make validate-config-pipelines   # Validate all pipeline.yaml files against topology and config

Note: make test runs envtest-setup automatically, but if you run go test directly in a module with controller tests (e.g. kube-applier), you need KUBEBUILDER_ASSETS set. Use: export KUBEBUILDER_ASSETS=$(make -s envtest-setup)

Target Environments

There's a multi-layered build and deploy system supporting multiple target environments with configs under config/):

  • Personal dev
    • cloud: "dev"; environment: "pers" (DEPLOY_ENV=pers)
    • Local development using make and properly-configured az command
    • Changes must be manually applied by the user by running proper make commands
    • Note: The integrated/shared dev environment (DEPLOY_ENV=dev) has been decommissioned. Only global infrastructure (shared ACRs, DNS zones) is still deployed to the dev environment via the global-pipeline-postsubmit Prow job.
  • Production systems
    • cloud: "public"; environments: "int", "stg", "prod"
    • Deployed via Microsoft ADO and EV2 into Microsoft Int, Stage and Prod environments (hosted in Microsoft Azure tenants)
    • For changes to be applied, a PR must be created for sdp-pipelines repo and updated pipelines run manually
    • After making changes, remind user about this and point at aka.ms/arohcp-pipelines for next steps

Access

  • "Dev/*" environments can be accessed with a @redhat.com account and are largely read/write. It's likely that az is currently logged into it.
  • "Public/Int" environment can be accessed with a @microsoft account and is mostly read-only. It's possible to az login into, but you need to ask the user explicitly to do that.
  • "Public/Stage" and "Public/Prod" require SRE-level access.

Deployments

Loosely defined categories of deployed objects:

  • Native Azure objects (using bicep files in dev-infrastructure/).
    • This includes databases, roles, access definitions, automations, monitoring stack "plumbing", etc. (e.g. global from dev-infrastructure/global-pipeline.yaml, regional from dev-infrastructure/region-pipeline.yaml, etc.)
    • But also two types of AKS clusters: Service Clusters and Management Clusters (dev-infrastructure/mgmt-pipeline.yaml)
  • Service Clusters – One per supported Azure region
    • Hosts the Resource Provider (frontend and backend) and Cluster Service
    • services_svc_pipelines in Makefile has a list of deployed services (on top of what's in dev-infrastructure/svc-pipeline.yaml)
  • Management Clusters
    • Multiple per Azure region (number depends on how many HCPs are running in that region)
    • Run all the things required to actually run HCPs
    • services_mgmt_pipelines in Makefile has a list of deployed services (on top of what's in dev-infrastructure/mgmt-pipeline.yaml)

Components

Loose categorization:

  • Deployment pipelines (dirs have *pipeline.yaml files present)
    • dev-infrastructure/ has all the bicep files
    • The rest deploy services to management and service clusters. Most of these do not host the code of the service, just reference already-released images.
  • ARO-HCP-specific microservices, like the Resource Provider's frontend/ and backend/:
    • These contain Go code in addition to pipelines configs.
  • Additional/helper (test/, hack/, tooling/, etc.)

Incomplete list:

  • Frontend: ARM REST API endpoint (frontend/) - Go service handling Azure ARM API calls
  • Backend: Internal processing service (backend/) - Go service for async operations
  • Cluster Service: Core cluster management (cluster-service/) - Wraps OCM's Clusters Service with ARO-HCP-specific configuration; manages HCP cluster lifecycle
  • Maestro: Multi-cluster orchestration (maestro/) - Handles communication between service and management clusters
  • Kube-Applier: Kubernetes resource applier (kube-applier/) - Reconciles desired-state manifests on management clusters
  • Fleet: Fleet controller (fleet/) - Service cluster controller for fleet-level operations
  • Mgmt-Agent: Management cluster agent (mgmt-agent/) - Agent running on each management cluster
  • SessionGate: Session management (sessiongate/) - Handles session gating for ARM requests
  • Admin: Admin API (admin/) - Internal admin interface (split into admin/client and admin/server modules)
  • Infrastructure: Azure infrastructure as code (dev-infrastructure/) - Bicep templates for all Azure resources
  • Internal: Shared libraries (internal/) - Common APIs, database, OCM client, tracing utilities
  • demo: helper scripts to quickly spin up and tear down an HCP cluster (see demo/README.md for usage)

The github.com/Azure/ARO-Tools repo is also a dependency and changes can be suggested for it.

Architectural Boundaries

  • The frontend must never access kube-applier, ReadDesireLister, or management cluster APIs directly. All management-cluster state needed by the frontend or admission code must be mirrored through ServiceProviderCluster by the backend. This architectural boundary prevents the frontend from having credentials or network access to management clusters. Concretely: the backend (which legitimately watches management clusters) observes the needed state and writes a distilled form onto the ServiceProviderCluster document in Cosmos; the frontend prefetches ServiceProviderCluster and admission reads it from the admission context — never from a live management-cluster client. For example, admitClusterVersionID validates a version change against ServiceProviderCluster.Status.DesiredVersionChannels, which the backend ControlPlaneActiveVersions controller mirrors from the observed HostedCluster. See internal/admission/CLAUDE.md for the full rule.

Additional Build, Configuration and Deployment Info

Go Workspace

The project uses Go workspaces (see go.work for the Go version and full module list):

  • Services: admin/client, admin/server, backend, frontend, fleet, kube-applier, mgmt-agent, sessiongate
  • Libraries: internal, test, test-integration
  • Tooling: tooling/hcpctl, tooling/templatize, tooling/secret-sync, etc.
  • Infrastructure scripts: dev-infrastructure/scripts/

After adding a module or changing cross-module deps, run make tidy (which includes go work sync).

Environment Configuration

  • Main config: config/config.yaml and overlays
  • Environment-specific settings in dev-infrastructure/configurations/
  • Service configs use Helm charts in each deploy/ directory

Pipeline System

Uses a custom templatize system (tooling/templatize/) that processes:

  • pipeline.yaml files define deployment pipelines
  • Bicep templates for Azure resource deployment
  • Helm charts for Kubernetes service deployment

Deployment Topology

topology.yaml defines the entire deployment graph as a tree of service groups, each referencing a pipeline.yaml. Entrypoints (Global, Region, Monitoring) are the top-level deployment targets. Run make entrypoint/Region or make pipeline/RP.Frontend to execute specific parts of the tree locally.

Configuration System

  • config/config.yaml is the main config with Go template variables ({{ .ctx.region }}, etc.)
  • config/config.schema.json validates the config — update the schema when adding new parameters. make verify-schema checks that objects use additionalProperties: false (with explicit exceptions like extraVars and features.items), so new fields must be added to the schema or validation will fail
  • config/config.msft.clouds-overlay.yaml provides overrides for Microsoft cloud environments
  • After any config change: cd config && make materialize to re-render, then commit config/rendered/ changes
  • See docs/configuration.md for full details

Code Organization

Service Structure

Each service follows consistent patterns:

  • Makefile - Build, deploy, and run targets
  • deploy/ - Helm charts and Kubernetes manifests
  • pipeline.yaml - Deployment pipeline definition
  • Go services include standard main.go, go.mod

Build Tags

  • Lint tags: LINT_GOTAGS='${GOTAGS},E2Etests'

Go conventions

  • Every go func(...) spawned in non-test code must defer utilruntime.HandleCrash() (alias k8s.io/apimachinery/pkg/util/runtime) as its first deferred call, so an unhandled panic in the goroutine respects utilruntime.ReallyCrash and the binary's crash-on-panic policy instead of silently taking down the process. Goroutines whose body is passed to a kube wait helper that already wraps the call (e.g. wait.Until, wait.UntilWithContext, wait.JitterUntil) do not need it — those helpers call HandleCrash internally. When the goroutine invokes a named function via go fn(...), put the defer utilruntime.HandleCrash() at the top of fn rather than wrapping the call site in a closure. Test code (*_test.go, test/, test-integration/, generated SDK fakes) is exempt.

  • Import aliases are enforced by linter. The .golangci.yml importas config requires specific aliases. Key ones:

    • arohcpv1alpha1 for github.com/openshift-online/ocm-sdk-go/aro_hcp/v1alpha1
    • cmv1 for github.com/openshift-online/ocm-sdk-go/clustersmgmt/v1
    • azcorearm for github.com/Azure/azure-sdk-for-go/sdk/azcore/arm
    • hcpsdk20240610preview for github.com/Azure/ARO-HCP/test/sdk/v20240610preview/...
    • azureclient for github.com/Azure/ARO-HCP/backend/pkg/azure/client
    • OpenShift API packages use {group}{version} pattern (e.g. configv1)

  • Import ordering is enforced by gci (configured in .golangci.yml): standard, blank, dot, default, k8s.io, sigs.k8s.io, github.com/Azure, github.com/openshift, github.com/Azure/ARO-HCP. Run make lint-fix to fix.

  • Gomega assertions must have descriptive annotations. CI runs verify-gomega-assertions on test/e2e/ and test/util/, which rejects bare Expect().To()/NotTo() calls. Always add a context string:

    • Expect(err).NotTo(HaveOccurred(), "failed to create HCP cluster") — not Expect(err).NotTo(HaveOccurred())
    • Expect(resp.Properties).NotTo(BeNil(), "cluster response Properties was nil") — not Expect(resp.Properties).NotTo(BeNil())

  • Every controller has a name constant and seeds its logger from it. Each controller package defines const XxxControllerName = "..." and uses that single value for: the name field on the controller struct, the workqueue Name (which surfaces as a Prometheus label), utils.ContextWithControllerName(ctx, name), and utils.LogValues{}.AddControllerName(name). Hardcoded string literals in those four call sites are a review-blocker — they cause silent drift between metrics labels, ctx values, and log fields.

  • Workqueue keys implement utils.LoggableKey. Any struct used as a controller workqueue key must implement AddLoggerValues(logger logr.Logger) logr.Logger (declared in internal/utils/context.go alongside LogValues). The standard implementation builds the key's *azcorearm.ResourceID and calls utils.LogValues{}.AddLogValuesForResourceID(...) so every log line from a reconcile carries the same subscription_id / resource_group / resource_id / hcp_cluster_name set — uniform across backend and kube-applier, and consistent with the Kusto indexes. Backend examples: HCPClusterKey, HCPNodePoolKey, OperationKey in backend/pkg/controllers/controllerutils/util.go. Kube-applier examples: ApplyDesireKey, DeleteDesireKey, ReadDesireKey in kube-applier/pkg/controllers/keys/keys.go.

  • Worker loops seed per-key logger via utils.AddLoggerValues. Every controller's processNext (or equivalent) must call logger := utils.AddLoggerValues(utils.LoggerFromContext(ctx), key) and ctx = utils.ContextWithLogger(ctx, logger) after pulling a key off the queue but before invoking SyncOnce. The backend's genericWatchingController does this once; kube-applier controllers, which run their own worker loops, must do it themselves.

API Versioning

ARM API versions live under internal/api/v<YYYYMMDD>preview/ (e.g. v20240610preview, v20251223preview). Each version directory has a generated/ subdirectory with auto-generated types and a register.go that wires the version into the API registry. Conversion between API versions and internal types happens in the *_methods.go files. The internal (versionless) types live in internal/api/.

E2E Tests

E2E tests live in test/e2e/ and use Ginkgo/Gomega. Key conventions:

  • E2E test files must not use the _test.go suffix (except e2e_test.go entry point) — the Ginkgo OpenShift extension excludes _test.go files during imports.
  • Every It() block must accept context.Context as its first parameter.
  • Every test must have labels: environment (labels.RequireNothing or labels.RequireHappyPathInfra), importance (labels.Critical/High/Medium/Low), and positivity (labels.Positive/labels.Negative).
  • Framework helpers use explicit API version suffixes: CreateHCPClusterFromParam20240610, GetHCPCluster20251223, etc.
  • See test/AGENTS.md for the full E2E design guide and code review standards.

Integration Tests

The test-integration/ directory uses a declarative artifact-driven test framework. Tests are defined as numbered step directories (00-load-initial-state/, 01-httpCreate-resource/, etc.) under artifacts/ trees — no Go code changes needed to add a new test case. See test-integration/claude.md for the full step type reference.

Helm Fixture Testing

Services have zz_fixture_TestHelmTemplate_*.yaml files that are auto-generated by cd config && make materialize. These are test fixtures for Helm chart validation — they must be committed with your PR. If you change a service's Helm chart or config values, re-run materialize to update them.

License Headers

All .go files require Apache 2.0 license headers. Run make licenses to add them via the addlicense tool. CI will fail if headers are missing.

Infrastructure as Code

  • Bicep templates in dev-infrastructure/templates/
  • Reusable modules in dev-infrastructure/modules/
  • Parameter files: *.bicepparam (with .tmpl.bicepparam templates)

PR Standards

  • PR titles must use Conventional Commits format: feat:, fix:, docs:, refactor:, test:, chore:, etc.
  • PR descriptions must explain the "Why", not just the "What", and link to a Jira/GitHub issue.
  • Tide is a merge-automation bot — its status is not a CI check; ignore it when evaluating CI.
  • See CONTRIBUTING.md for the full PR standards.

Tooling

Key development tools (installed via make install-tools):

  • golangci-lint - Go linting
  • mockgen - Mock generation
  • addlicense - License header management
  • yamlfmt - YAML formatting
  • oras - OCI registry interaction

Custom tools in tooling/:

  • hcpctl - CLI for HCP management and access
  • templatize - Pipeline template processing
  • secret-sync - Secret management utilities
  • prometheus-rules - Monitoring rule generation

Cosmos Data Flow Documentation

docs/generated/service-provider-control-flow.md documents every controller, endpoint writes, external resource effects, and cluster/node-pool/external-auth lifecycles. The ARO Ops bot keeps the reference and its Graphviz sources/PNGs in sync with the code.

When to regenerate

Everything under docs/generated/ is generated output, regenerated and committed separately by the ARO Ops bot on an automated hourly cadence. Do not regenerate these files inside feature PRs unless specifically requested, to avoid merge conflicts.

The separate regeneration uses the generation prompt to update docs/generated/service-provider-control-flow.md when source changes touch:

  • frontend/pkg/frontend/ — any handler that writes to Cosmos
  • Controller implementations or startup registration in backend, fleet, kube-applier, mgmt-agent, sessiongate, or shared informer management — including controllers without Cosmos writes
  • Azure/Cluster Service/Kubernetes effects, reconciliation gates, or resource lifecycle dependencies
  • internal/api/ — any struct field that is stored in Cosmos
  • internal/database/ — any change to CRUD operations or precondition logic

Field-level writer annotations

Every leaf struct field in internal/api/types_*.go files (the types stored in Cosmos) must have a // Written by: comment listing the actors (frontend endpoints and/or backend controllers) that set it. For example:

// Written by: Frontend PUT Cluster (Create), ClusterClusterServiceCreate
ClusterServiceID *InternalID `json:"clusterServiceID,omitempty"`

When adding or modifying a field, update this comment. When adding or modifying a controller or frontend endpoint that writes a field, update the comment on the field it writes.

Subdirectory Guidance

Several subdirectories have their own AI guidance files with domain-specific detail:

  • test/AGENTS.md — E2E test design principles, review standards, label reference, and anti-patterns
  • test-integration/claude.md — Step type reference for the declarative integration test framework
  • config/CLAUDE.md — Config rendering workflow and example commits

Documentation

Key documentation files: