chore(rest-api): Add ingress based TLS support for REST API

docs/getting-started/installation-options/reference-install.md

@@ -12,7 +12,7 @@ This page collects the maintained, manifest-level references for operators who n
 
 `setup.sh` is a thin wrapper over the Helm charts and kustomize manifests in the repository. The source-of-truth guides document each step, the order of operations, the PKI and secrets model, and troubleshooting:
 
-- **Prerequisites and NICo Core** — [`helm-prereqs/README.md`](https://github.com/NVIDIA/infra-controller/blob/main/helm-prereqs/README.md) covers the prerequisite stack (local-path-provisioner, postgres-operator, MetalLB, cert-manager, Vault, External Secrets), the NICo Core deployment, the per-site values files, the `.forge` compatibility DNS, and the `health-check.sh` verification.
+- **Prerequisites and NICo Core** — [`helm-prereqs/README.md`](https://github.com/NVIDIA/infra-controller/blob/main/helm-prereqs/README.md) covers the prerequisite stack (local-path-provisioner, postgres-operator, MetalLB, optional ingress-nginx, cert-manager, Vault, External Secrets), the NICo Core deployment, the per-site values files, the `.forge` compatibility DNS, and the `health-check.sh` verification.
 - **NICo REST** — [`rest-api/deploy/INSTALLATION.md`](https://github.com/NVIDIA/infra-controller/blob/main/rest-api/deploy/INSTALLATION.md) is the prescriptive, manifest-by-manifest bring-up for the REST control plane (PostgreSQL, Keycloak, Temporal, the internal cert-manager, site-manager, API, workflow workers, and site-agent).
 
 > **NICo REST is in-tree.** The REST stack lives in this repository under `rest-api/`; it is no longer a separate repository. `setup.sh` resolves it automatically, so no `NCX_REPO` clone is required.

docs/getting-started/quick-start.md

@@ -193,6 +193,41 @@ nico-rest-api:
 
 When `keycloak.enabled: false`, the Keycloak deployment is still created by `setup.sh`, but `nico-rest-api` will not use it for token validation.
 
+#### Optional: Expose NICo REST over TLS with Ingress
+
+By default, `setup.sh` exposes `nico-rest-api` through a NodePort in `helm-prereqs/values/nico-rest.yaml`. To expose it through a fully qualified domain name with TLS, enable the `nico-rest-api` ingress and configure its certificate settings in that file:
+
+```yaml
+nico-rest-api:
+  ingress:
+    enabled: true
+    className: nginx
+    hosts:
+      - host: rest-api.mysite.example.com
+        paths:
+          - path: /
+            pathType: Prefix
+    tls:
+      - secretName: rest-api-mysite-example-com-tls
+        hosts:
+          - rest-api.mysite.example.com
+    certificate:
+      enabled: true
+      secretName: rest-api-mysite-example-com-tls
+      commonName: rest-api.mysite.example.com
+```
+
+The chart creates a cert-manager `Certificate` for the ingress TLS Secret when `ingress.certificate.enabled: true`. The certificate is issued by the REST stack's `nico-rest-ca-issuer`, so this is a quick self-signed/private-CA setup suitable for lab and site-local deployments. If your cluster already has a TLS Secret for the domain, set `ingress.certificate.enabled: false` and keep `ingress.tls[].secretName` pointed at that existing Secret.
+
+The ingress routes to the chart-managed `nico-rest-api` Service on `service.port`; the API pod still serves plain HTTP internally and does not need TLS-specific configuration. If your cluster does not already have an ingress controller, run setup with the optional nginx controller:
+
+```bash
+./setup.sh --install-ingress-nginx
+# or
+export NICO_INSTALL_INGRESS_NGINX=true
+./setup.sh -y
+```
+
 ### 3f. Review site-agent Config
 
 The defaults in `helm-prereqs/values/nico-site-agent.yaml` point at the Zalando-managed `nico-pg-cluster` (`DB_ADDR: nico-pg-cluster.postgres.svc.cluster.local`, `DB_DATABASE: nico_rest`), which is the same cluster used by `nico-rest-api`. No changes are needed for a standard deployment.
@@ -313,6 +348,7 @@ The `setup.sh` script installs all prerequisites and NICo components in sequenti
 | 1 | local-path-provisioner + StorageClasses |
 | 1b | postgres-operator (Zalando) |
 | 1c | MetalLB + site BGP/L2 config |
+| 1d | Optional ingress-nginx controller (`--install-ingress-nginx`) |
 | 2 | cert-manager + Vault TLS bootstrap (PKI chain) |
 | 3 | HashiCorp Vault (3-node HA Raft) |
 | 4 | Vault init + unseal + SSH host key |
@@ -325,6 +361,7 @@ The following components are deployed:
 ```
 local-path-provisioner     (raw manifest - StorageClasses for Vault + PostgreSQL PVCs)
 metallb                    (metallb/metallb 0.14.5 - LoadBalancer IPs via BGP or L2)
+ingress-nginx              (optional - Ingress controller for REST API FQDN/TLS)
 postgres-operator          (zalando/postgres-operator 1.10.1 - manages nico-pg-cluster)
 cert-manager               (jetstack/cert-manager v1.17.1)
 vault                      (hashicorp/vault 0.25.0, 3-node HA Raft, TLS)

helm-prereqs/README.md

@@ -43,7 +43,7 @@ helm-prereqs/
 │   ├── nico-site-agent.yaml     # Site-agent deployment values (DB config, gRPC settings)
 │   └── metallb-config.yaml     # MetalLB IP pools, BGP peers, and advertisements
 ├── templates/                  # nico-prereqs Helm chart templates (PKI, ESO, PostgreSQL)
-├── operators/                  # Raw manifests and operator values (local-path, MetalLB, cert-manager, Vault, ESO)
+├── operators/                  # Raw manifests and operator values (local-path, MetalLB, ingress-nginx, cert-manager, Vault, ESO)
 └── keycloak/                   # Dev Keycloak deployment and token helper scripts
 ```
 
@@ -104,6 +104,7 @@ The tables below summarize the keys that must be set per site.
 | `NICO_SITE_UUID` | No | Stable UUID for this site. If unset, `setup.sh` tries to reuse the UUID from a prior install (site-agent ConfigMap). If that fails, it adopts an existing REST site with the same name, or mints a UUID and seeds the site record itself. |
 | `NICO_MANAGE_DEFAULT_STORAGE_CLASS` | No | Whether `setup.sh` marks `local-path` as the default StorageClass. Defaults to `true`. Set to `false` when the cluster already has an operator-managed default StorageClass. |
 | `NICO_STORAGE_CLASS` | No | StorageClass used by Vault data/audit PVCs. Defaults to `local-path-persistent`. |
+| `NICO_INSTALL_INGRESS_NGINX` | No | Install the optional ingress-nginx controller after MetalLB. Defaults to `false`; set to `true` only when the cluster does not already provide an ingress controller. |
 | `PREFLIGHT_CHECK_IMAGE` | No | Image used for preflight per-node checks. Defaults to `busybox:1.36`; set to a local mirror for air-gapped clusters. |
 
 ### `values.yaml`
@@ -185,6 +186,7 @@ It supports these common deployment modes:
 | `--skip-core --skip-rest` | Infrastructure-only run; image tags, image registry, and REST repo are not required. |
 | `--core-values <file>` | Use site-specific Core values instead of `helm-prereqs/values/nico-core.yaml`. |
 | `--metallb-config <path>` | Use a site-specific MetalLB manifest file or kustomize directory. |
+| `--install-ingress-nginx` | Install the optional ingress-nginx controller. The controller Service is `LoadBalancer` and receives its external IP from MetalLB. |
 | `--site-overlay <dir>` | Apply a site kustomize overlay after Core deploys. |
 | `--debug` | Enable bash tracing. This can print secrets, so avoid it in shared logs. |
 

helm-prereqs/clean.sh

@@ -21,10 +21,11 @@
 #   0. NCX stack           (nico-rest helm, temporal, keycloak, ncx postgres)
 #   1. nico core        (separate helm release, if installed)
 #   2. helmfile releases   (nico-prereqs, external-secrets, vault, cert-manager,
-#                           postgres-operator)
+#                           postgres-operator, ingress-nginx)
 #   3. cluster-scoped hook resources (ClusterIssuers, ClusterSecretStore, etc.)
 #   4. vault init secrets  (vault-cluster-keys, vaultunsealkeys, vaultroottoken)
-#   5. namespaces          (nico-system, cert-manager, vault, external-secrets, postgres)
+#   5. namespaces          (nico-system, cert-manager, vault, external-secrets,
+#                           postgres, ingress-nginx)
 #   6. local-path-persistent PVs owned by this stack (Retain policy — not deleted with namespace)
 #   7. local-path-provisioner + StorageClass (applied via kubectl, not helm-managed)
 # =============================================================================
@@ -68,7 +69,7 @@ helm uninstall nico -n nico-system 2>/dev/null || true
 
 # ---------------------------------------------------------------------------
 # 2. All helmfile releases in reverse dependency order:
-#    nico-prereqs → external-secrets → vault → cert-manager → metallb
+#    nico-prereqs → external-secrets → vault → cert-manager → ingress-nginx → metallb
 # ---------------------------------------------------------------------------
 echo "=== [2/8] Destroying helmfile releases ==="
 
@@ -153,6 +154,14 @@ kubectl get clusterrole,clusterrolebinding -o name \
     | grep nico-rest \
     | xargs kubectl delete --ignore-not-found 2>/dev/null || true
 
+# ingress-nginx cluster-scoped resources are normally removed by helm uninstall,
+# but remove stragglers so clean.sh can recover from partial installs.
+echo "Removing ingress-nginx cluster-scoped resources..."
+kubectl delete ingressclass nginx --ignore-not-found 2>/dev/null || true
+kubectl get clusterrole,clusterrolebinding -o name \
+    | grep ingress-nginx \
+    | xargs kubectl delete --ignore-not-found 2>/dev/null || true
+
 # ---------------------------------------------------------------------------
 # 3. Cluster-scoped resources created by helm hooks.
 #    These survive helm/helmfile uninstall because hook-delete-policy is
@@ -193,12 +202,12 @@ kubectl delete secret vault-cluster-keys vaultunsealkeys vaultroottoken \
 #    conflict with setup.sh's helmfile install into the external-secrets ns.
 # ---------------------------------------------------------------------------
 echo "=== [5/8] Deleting namespaces ==="
-kubectl delete ns nico-system cert-manager vault external-secrets postgres metallb-system \
+kubectl delete ns nico-system cert-manager vault external-secrets postgres ingress-nginx metallb-system \
     --wait=false --ignore-not-found 2>/dev/null || true
 
 echo "Waiting for namespaces to terminate..."
 kubectl wait --for=delete \
-    ns/nico-system ns/cert-manager ns/vault ns/external-secrets ns/postgres ns/metallb-system \
+    ns/nico-system ns/cert-manager ns/vault ns/external-secrets ns/postgres ns/ingress-nginx ns/metallb-system \
     --timeout=180s 2>/dev/null || true
 
 echo "Purging default namespace (ESO and other non-kubespray resources)..."
@@ -233,7 +242,7 @@ echo "=== [6/8] Removing Released PersistentVolumes owned by this stack ==="
 kubectl get pv -o json 2>/dev/null \
     | jq -r '.items[] | select(
         .spec.storageClassName == "local-path-persistent" and
-        (.spec.claimRef.namespace // "" | test("^(nico-system|cert-manager|vault|external-secrets|postgres|metallb-system|nico-rest|temporal)$"))
+        (.spec.claimRef.namespace // "" | test("^(nico-system|cert-manager|vault|external-secrets|postgres|ingress-nginx|metallb-system|nico-rest|temporal)$"))
       ) | .metadata.name' \
     | xargs -r kubectl delete pv --ignore-not-found 2>/dev/null || true
 

helm-prereqs/health-check.sh

@@ -95,17 +95,25 @@ fi
 
 # cert-manager, ESO, MetalLB: discover by known deployment names
 if [[ -z "${CERT_MANAGER_NS:-}" ]]; then
-  CERT_MANAGER_NS=$(kubectl get deployment cert-manager -A \
+  CERT_MANAGER_NS=$(kubectl get deployment -A \
+    --field-selector metadata.name=cert-manager \
     -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || printf 'cert-manager')
 fi
 if [[ -z "${ESO_NS:-}" ]]; then
-  ESO_NS=$(kubectl get deployment external-secrets -A \
+  ESO_NS=$(kubectl get deployment -A \
+    --field-selector metadata.name=external-secrets \
     -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || printf 'external-secrets')
 fi
 if [[ -z "${METALLB_NS:-}" ]]; then
-  METALLB_NS=$(kubectl get deployment metallb-controller -A \
+  METALLB_NS=$(kubectl get deployment -A \
+    --field-selector metadata.name=metallb-controller \
     -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || printf 'metallb-system')
 fi
+if [[ -z "${INGRESS_NGINX_NS:-}" ]]; then
+  INGRESS_NGINX_NS=$(kubectl get deployment -A \
+    --field-selector metadata.name=ingress-nginx-controller \
+    -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || printf 'ingress-nginx')
+fi
 
 printf "  %-26s %s\n" "NICo namespace:"     "${NICO_NS}"
 printf "  %-26s %s\n" "vault namespace:"       "${VAULT_NS}"
@@ -114,6 +122,7 @@ printf "  %-26s %s\n" "postgres namespace:"    "${POSTGRES_NS}"
 printf "  %-26s %s\n" "cert-manager ns:"       "${CERT_MANAGER_NS}"
 printf "  %-26s %s\n" "external-secrets ns:"   "${ESO_NS}"
 printf "  %-26s %s\n" "metallb ns:"            "${METALLB_NS}"
+printf "  %-26s %s\n" "ingress-nginx ns:"      "${INGRESS_NGINX_NS:-not installed}"
 
 # --------------------------------------------------------------------------
 # Test helpers
@@ -241,6 +250,14 @@ else
   fail "daemonset/metallb-speaker: ${_ready:-0}/${_desired} ready"
 fi
 
+if [[ -n "${INGRESS_NGINX_NS:-}" ]]; then
+  section "ingress-nginx"
+  _check_deployment "${INGRESS_NGINX_NS}" ingress-nginx-controller
+else
+  section "ingress-nginx"
+  skip "not installed"
+fi
+
 # --------------------------------------------------------------------------
 # 2. Vault
 # --------------------------------------------------------------------------

helm-prereqs/helmfile.yaml

@@ -35,6 +35,8 @@ repositories:
     url: https://opensource.zalando.com/postgres-operator/charts/postgres-operator
   - name: metallb
     url: https://metallb.github.io/metallb
+  - name: ingress-nginx
+    url: https://kubernetes.github.io/ingress-nginx
 
 releases:
   # ---------------------------------------------------------------------------
@@ -55,6 +57,21 @@ releases:
     values:
       - operators/values/metallb.yaml
 
+  # ---------------------------------------------------------------------------
+  # ingress-nginx — optional Ingress controller for clusters that do not already
+  # provide one. setup.sh installs it only when requested; the controller Service
+  # uses type LoadBalancer and receives an address from MetalLB.
+  # ---------------------------------------------------------------------------
+  - name: ingress-nginx
+    namespace: ingress-nginx
+    createNamespace: true
+    chart: ingress-nginx/ingress-nginx
+    version: "4.12.1"
+    wait: true
+    timeout: 600
+    values:
+      - operators/values/ingress-nginx.yaml
+
   # ---------------------------------------------------------------------------
   # Zalando postgres-operator — manages nico-pg-cluster HA Patroni cluster.
   # Installed before cert-manager (no TLS dependency). Must be up before

helm-prereqs/operators/values/ingress-nginx.yaml

@@ -0,0 +1,23 @@
+# =============================================================================
+# operators/values/ingress-nginx.yaml — ingress-nginx Helm chart static values
+#
+# Optional cluster-level Ingress controller for NICo REST ingress exposure.
+# The controller Service is type LoadBalancer and is expected to receive an
+# external IP from MetalLB.
+#
+# Chart: ingress-nginx/ingress-nginx 4.12.1
+# =============================================================================
+
+controller:
+  ingressClassResource:
+    name: nginx
+    enabled: true
+    default: false
+    controllerValue: k8s.io/ingress-nginx
+  ingressClass: nginx
+  watchIngressWithoutClass: false
+  service:
+    type: LoadBalancer
+    externalTrafficPolicy: Cluster
+  admissionWebhooks:
+    enabled: true

helm-prereqs/setup.sh

@@ -44,6 +44,8 @@
 #                          StorageClass. Default: true.
 #   NICO_STORAGE_CLASS     StorageClass for Postgres and Vault data and audit PVCs.
 #                          Default: local-path-persistent.
+#   NICO_INSTALL_INGRESS_NGINX
+#                          Install ingress-nginx after MetalLB. Default: false.
 #   VAULT_NS               Vault namespace. Default: vault
 #   CERT_MANAGER_NS        cert-manager namespace. Default: cert-manager
 #   PREFLIGHT_CHECK_IMAGE  Image for preflight per-node checks.
@@ -65,6 +67,7 @@
 #   ./setup.sh --skip-core --skip-rest  # fully non-interactive infra-only run
 #   ./setup.sh --core-values /path/to/values.yaml      # use site-specific values for Phase 6
 #   ./setup.sh --metallb-config /path/to/metallb.yaml  # use site-specific MetalLB config (file or kustomize dir)
+#   ./setup.sh --install-ingress-nginx # install optional nginx Ingress controller
 #   ./setup.sh --site-overlay /path/to/kustomize-dir   # kubectl apply -k after Phase 6 (NTP services, etc.)
 #   ./setup.sh --debug                  # enable bash -x trace (or run: bash -x ./setup.sh)
 #
@@ -83,6 +86,7 @@ AUTO_YES=false
 SKIP_CORE=false
 SKIP_REST=false
 SKIP_FLOW=false
+INSTALL_INGRESS_NGINX="${NICO_INSTALL_INGRESS_NGINX:-false}"
 CORE_VALUES=""
 METALLB_CONFIG=""
 SITE_OVERLAY=""
@@ -92,6 +96,7 @@ while [[ $# -gt 0 ]]; do
         --skip-core)    SKIP_CORE=true ;;
         --skip-rest)    SKIP_REST=true ;;
         --skip-flow)    SKIP_FLOW=true ;;
+        --install-ingress-nginx) INSTALL_INGRESS_NGINX=true ;;
         --debug)        set -x         ;;
         --core-values)
             [[ -z "${2:-}" ]] && { echo "Error: --core-values requires a file path"; exit 1; }
@@ -108,7 +113,7 @@ while [[ $# -gt 0 ]]; do
             SITE_OVERLAY="$(cd "$(dirname "$2")" && pwd)/$(basename "$2")"
             [[ ! -d "${SITE_OVERLAY}" ]] && { echo "Error: --site-overlay directory not found: $2"; exit 1; }
             shift ;;
-        *) echo "Usage: $0 [-y] [--skip-core] [--skip-rest] [--skip-flow] [--core-values <file>] [--metallb-config <file-or-dir>] [--site-overlay <dir>] [--debug]"; exit 1 ;;
+        *) echo "Usage: $0 [-y] [--skip-core] [--skip-rest] [--skip-flow] [--install-ingress-nginx] [--core-values <file>] [--metallb-config <file-or-dir>] [--site-overlay <dir>] [--debug]"; exit 1 ;;
     esac
     shift
 done
@@ -126,6 +131,10 @@ VAULT_NS="${VAULT_NS:-vault}"
 CERT_MANAGER_NS="${CERT_MANAGER_NS:-cert-manager}"
 NICO_MANAGE_DEFAULT_STORAGE_CLASS="${NICO_MANAGE_DEFAULT_STORAGE_CLASS:-true}"
 NICO_STORAGE_CLASS="${NICO_STORAGE_CLASS:-local-path-persistent}"
+case "${INSTALL_INGRESS_NGINX}" in
+    true|false) ;;
+    *) echo "Error: NICO_INSTALL_INGRESS_NGINX must be true or false"; exit 1 ;;
+esac
 
 # ---------------------------------------------------------------------------
 # Failure handler — offer to run clean.sh if setup exits with an error.
@@ -311,6 +320,24 @@ else
 fi
 echo "MetalLB ready"
 
+# ---------------------------------------------------------------------------
+# 1d. ingress-nginx — optional Ingress controller.
+#     Install after MetalLB so the controller LoadBalancer Service can receive
+#     an external address. Skip this when the cluster already has an Ingress
+#     controller or uses a different implementation.
+# ---------------------------------------------------------------------------
+if [[ "${INSTALL_INGRESS_NGINX}" == "true" ]]; then
+    _SETUP_PHASE="[1d] ingress-nginx"
+    echo "=== [1d] ingress-nginx ==="
+    helmfile sync -l name=ingress-nginx
+    echo "Waiting for ingress-nginx controller to be ready..."
+    kubectl wait --for=condition=Available deployment/ingress-nginx-controller \
+        -n ingress-nginx --timeout=120s
+    echo "ingress-nginx ready"
+else
+    echo "Skipping ingress-nginx (set NICO_INSTALL_INGRESS_NGINX=true or pass --install-ingress-nginx to install it)"
+fi
+
 # ---------------------------------------------------------------------------
 # 2. cert-manager + Prometheus CRDs + Vault TLS bootstrap
 #    cert-manager must be up before we can issue certs for vault.

helm-prereqs/values/nico-rest.yaml

@@ -18,6 +18,25 @@ nico-rest-api:
   nodePort:
     enabled: true
     port: 30388
+  # To expose REST through the optional ingress-nginx controller, run setup.sh
+  # with --install-ingress-nginx and configure a DNS name here. The ingress
+  # routes to the chart-managed ClusterIP Service on service.port.
+  # ingress:
+  #   enabled: true
+  #   className: nginx
+  #   hosts:
+  #     - host: api.example.com
+  #       paths:
+  #         - path: /
+  #           pathType: Prefix
+  #   tls:
+  #     - secretName: api-example-com-tls
+  #       hosts:
+  #         - api.example.com
+  #   certificate:
+  #     enabled: true
+  #     secretName: api-example-com-tls
+  #     commonName: api.example.com
   config:
     keycloak:
       enabled: true