Skip to content
17 changes: 15 additions & 2 deletions tests/bdd/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,23 @@ logic into `dsl/`.
namespace, and intended Gateway parent plus the shared context and timeout.
The step requires `Accepted=True` and `ResolvedRefs=True` for that parent but
does not allowlist route kinds or duplicate Gateway API validation.
- File-mutating steps (`I copy the file`, `I update yaml file`,
`I prepare self-managed secrets file`, `I substitute a block`)
- File-mutating steps (`I copy the file`, `I write yaml file`,
`I update yaml file`, `I prepare self-managed secrets file`,
`I substitute a block`)
snapshot the destination through `Suite.Ledger` before the first write.
Suite teardown restores every snapshotted path.
- `CommandRunner` tokenizes command text with shlex and execs the argv
directly. There is no shell, so heredocs, pipes, and redirections only
work inside an explicit `/bin/bash -c '...'` wrapper. Apply inline YAML
through `Kubernetes manifest ... is:` plus `I successfully apply
Kubernetes manifest ... using contexts:` instead of `kubectl apply -f -`
with a heredoc.
- `Kubernetes manifest ... is:` stores the raw docstring in
`ScenarioContext.Manifests`; the apply step interpolates `${VAR}` and
writes the rendered body under `out/<run-id>/` before each
explicit-context `kubectl apply`. Declare manifests with the scenario's
Givens even when they reference an env var exported later, and never
interpolate at declaration time.
- `Given command has succeeded:` keys on the fully resolved command
text. Two scenarios whose pre-interpolation text matches but whose
env vars differ must miss the cache. The cache lives in
Expand Down
15 changes: 14 additions & 1 deletion tests/bdd/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,23 @@ refactor in every consumer; that is a feature.
| Step | Notes |
|------|-------|
| `And I copy the file {string} to {string}` | Both paths are repo-relative. |
| `And I write yaml file {string} with values:` (two-column table of dotted-path and value) | Creates a new YAML file from the visible table. The destination must not already exist; the step fails instead of overwriting so an authored file is never silently replaced. Parent directories are created. Path syntax and `${VAR}` expansion match `I update yaml file`. Boolean literals and collection literals such as `[]` are written as native YAML types, not quoted strings, because Helm treats the string `"false"` as truthy. Numbers stay as strings. The destination is ledger-backed and removed at teardown. |
| `And I update yaml file {string} with keys:` (two-column table of dotted-path and value) | Path supports dotted notation and `[n]` indices (e.g. `global.imagePullSecrets[0].name`). Missing intermediate maps and missing list indices are upserted: writing `global.imagePullSecrets[0].name` against a file that has neither `global.imagePullSecrets` nor any list entry creates both. Existing scalars at intermediate positions cause the step to fail rather than silently overwrite a non-map. Value cells expand `${VAR}` from `os.Environ`. |
| `And I prepare Helmfile environment {string} for stack {string} from fixture {string} with values:` (two-column table of dotted-path and value) | Validates the stack and environment names, derives `deploy/stacks/<stack>/environments/<environment>.yaml` from the absolute repository root, copies the explicit fixture, and applies the visible values table with the same YAML update and `${VAR}` interpolation behavior. Supported stacks are `self-managed`, `observability`, and `nvcf-compute-plane`. The destination is ledger-backed. |
| `And I prepare self-managed secrets file {string} from template {string} using the current NGC registry credential` | The destination and template are explicit repo-relative paths with `${VAR}` interpolation. Replaces the template's registry credential placeholder with base64 of the current `$oauthtoken:<NGC_API_KEY>` credential and writes the destination with mode `0600`. The destination is ledger-backed, and secret material never enters Gherkin, command logs, or failure messages. |
| `And I substitute a block in file {string}:` (docstring) | The docstring contains an old block and replacement block separated by exactly one `---` line. `${VAR}` interpolation applies before an exact, ledger-backed replacement. Missing or malformed old blocks fail. |

### Kubernetes manifests (Given / When)

The Given keeps every manifest field visible in the feature file. The When
hides only the repeated `kubectl --context <ctx> apply -f <file>` mechanics
and the target contexts stay visible as table rows.

| Step | Notes |
|------|-------|
| `Given Kubernetes manifest {string} is:` (YAML docstring) | Stores the raw docstring under the visible name in scenario state. The name must be non-empty and declared at most once per scenario. No interpolation, validation, or I/O happens here, so the manifest may reference an env var that a later step exports. |
| `When I successfully apply Kubernetes manifest {string} using contexts:` (table) | Requires a `context` header and one or more contexts. Interpolates `${VAR}` in the named manifest at apply time, writes the rendered body once to a file under the run's `out/<run-id>/` directory, and runs one explicit-context `kubectl apply -f <file>` per row in order. Each apply must exit 0 and is recorded like any successful command. Failures name the row, manifest, and context. The rendered file is a run artifact, not a ledger-backed working-tree path. |

### Command execution (When)

| Step | Notes |
Expand Down Expand Up @@ -194,6 +206,7 @@ original order. Repeated options and empty values are preserved.
| `Then these Kubernetes resources should exist in namespace {string} using context {string}:` (table) | Requires `kind` and `name` headers. Gets each named resource with the explicit namespace and context, and reports the row whose resource is missing. |
| `Then these Kubernetes resources should not exist in namespace {string} using context {string}:` (table) | Requires `kind` and `name` headers. Gets each named resource with `--ignore-not-found` and requires empty name output, so absence does not depend on human-readable error text. |
| `Then deployment {string} in namespace {string} using context {string} should complete rollout within {string}` | Runs `kubectl rollout status` for the named deployment with the explicit namespace, context, and timeout. Failure messages name the deployment without printing command output. |
| `Then these Kubernetes workloads should complete rollout using context {string} within {string}:` (table) | Requires `kind`, `name`, and `namespace` headers. Runs one explicit-context `kubectl rollout status <kind>/<name>` per row in order with the shared context and timeout. The kind is lowercased and passed through without an allowlist, so Deployment, StatefulSet, and DaemonSet rows all work. Failures name the row, kind, name, and namespace without printing command output. Prefer this over a raw `kubectl rollout status` command step. |
| `Then NVCFBackend {string} in namespace {string} using context {string} should report agent status {string} within {string}` | Waits for the named backend's `status.agentStatus` to equal the visible value using the explicit namespace, context, and timeout. Failure messages name the backend without printing resource output. |
| `Then these Gateway API routes should be accepted and resolved using context {string} within {string}:` (table) | Requires `kind`, `name`, `namespace`, and `parent` headers. Waits for every named route to report both `Accepted=True` and `ResolvedRefs=True` for the named Gateway parent using the explicit context and timeout. The route kind is passed through without an allowlist. Failures name the table row, route, namespace, parent, and unmet condition without printing resource output. |
| `Then every Pylon for function {string} using container {string} and context {string} should report metrics within {string}:` (table) | Requires `metric`, `comparison`, and `count` headers. Polls every running pod selected by the visible `function-name` annotation and container name. Each pod must expose non-empty metrics, and each metric row counts connected series whose sample value is `1`; `comparison` is `exactly` or `at least`, and the expected non-negative count remains visible. Discovery, parsing, and scrape failures remain failures rather than zero metric counts. |
Expand Down Expand Up @@ -288,7 +301,7 @@ contract verified in `src/clis/nvcf-cli/cmd/`):
## File restoration

Every step that writes into a path under the repo working tree
(`I copy the file ... to ...`, `I update yaml file ...`,
(`I copy the file ... to ...`, `I write yaml file ...`, `I update yaml file ...`,
`I prepare self-managed secrets file ...`, `I substitute a block ...`)
registers that path with the runner's
restoration ledger:
Expand Down
27 changes: 25 additions & 2 deletions tests/bdd/dsl/kubectl.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,15 +118,38 @@ func KubernetesResourceAbsent(raw string, resource KubernetesResource) error {
return nil
}

// KubernetesWorkload identifies one rollout-capable workload by kind, name,
// and namespace.
type KubernetesWorkload struct {
Kind string
Name string
Namespace string
}

// KubernetesDeploymentRolloutCommand builds an explicit-context rollout wait
// for one deployment.
func KubernetesDeploymentRolloutCommand(name, namespace, kubeContext, timeout string) (string, error) {
target, err := resolveKubernetesWaitTarget("deployment", name, namespace, kubeContext, timeout)
return KubernetesWorkloadRolloutCommand(
KubernetesWorkload{Kind: "deployment", Name: name, Namespace: namespace},
kubeContext,
timeout,
)
}

// KubernetesWorkloadRolloutCommand builds an explicit-context rollout wait
// for one workload. The kind is lowercased and passed through without an
// allowlist, so every kind kubectl rollout status accepts works.
func KubernetesWorkloadRolloutCommand(workload KubernetesWorkload, kubeContext, timeout string) (string, error) {
kind := strings.ToLower(strings.TrimSpace(Interpolate(workload.Kind)))
if kind == "" {
return "", fmt.Errorf("kubernetes workload kind is empty")
}
target, err := resolveKubernetesWaitTarget(kind, workload.Name, workload.Namespace, kubeContext, timeout)
if err != nil {
return "", err
}
return strings.Join([]string{
"kubectl", "rollout", "status", quoteCommandArg("deployment/" + target.name),
"kubectl", "rollout", "status", quoteCommandArg(kind + "/" + target.name),
"-n", quoteCommandArg(target.namespace),
"--context", quoteCommandArg(target.kubeContext),
quoteCommandArg("--timeout=" + target.timeout),
Expand Down
38 changes: 38 additions & 0 deletions tests/bdd/dsl/kubectl_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,44 @@ func TestKubernetesDeploymentRolloutCommandBuildsExplicitWait(t *testing.T) {
}
}

// TestKubernetesWorkloadRolloutCommandLowercasesKind verifies that a
// mixed-case kind is lowercased for kubectl and that ${VAR} in the
// workload name is interpolated before the argv is built.
func TestKubernetesWorkloadRolloutCommandLowercasesKind(t *testing.T) {
t.Setenv("BDD_ROUTER_NAME", "llm-request-router-region-b")
got, err := KubernetesWorkloadRolloutCommand(
KubernetesWorkload{Kind: "StatefulSet", Name: "${BDD_ROUTER_NAME}", Namespace: "nvcf"},
"k3d-ncp-local-cp",
"10m",
)
if err != nil {
t.Fatalf("build command: %v", err)
}
want := "kubectl rollout status statefulset/llm-request-router-region-b -n nvcf --context k3d-ncp-local-cp --timeout=10m"
if got != want {
t.Fatalf("command = %q, want %q", got, want)
}
}

// TestKubernetesWorkloadRolloutCommandRejectsMissingInputs confirms that
// an empty kind, name, namespace, or kube context is rejected instead of
// producing a rollout wait against an implicit target.
func TestKubernetesWorkloadRolloutCommandRejectsMissingInputs(t *testing.T) {
cases := map[string]KubernetesWorkload{
"kind": {Name: "router", Namespace: "nvcf"},
"name": {Kind: "StatefulSet", Namespace: "nvcf"},
"namespace": {Kind: "StatefulSet", Name: "router"},
}
for missing, workload := range cases {
if _, err := KubernetesWorkloadRolloutCommand(workload, "k3d-ncp-local-cp", "10m"); err == nil {
t.Fatalf("expected empty %s error", missing)
}
}
if _, err := KubernetesWorkloadRolloutCommand(KubernetesWorkload{Kind: "Deployment", Name: "router", Namespace: "nvcf"}, "", "10m"); err == nil {
t.Fatal("expected empty context error")
}
}

func TestNVCFBackendAgentStatusCommandBuildsExplicitWait(t *testing.T) {
t.Setenv("BDD_BACKEND_NAME", "ncp-local-compute-1")
got, err := NVCFBackendAgentStatusCommand("${BDD_BACKEND_NAME}", "nvca-operator", "k3d-ncp-local-compute-1", "healthy", "10m")
Expand Down
50 changes: 50 additions & 0 deletions tests/bdd/dsl/yamledit.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,32 @@ const (
MatchSubset
)

// RenderYAMLFromKeys builds a YAML document from the supplied
// dotted-path/value pairs and returns its serialized bytes. It is the
// pure counterpart of UpdateYAMLKeys for a file that does not exist
// yet: the caller owns the destination path, the existence check, and
// the write. Path syntax matches UpdateYAMLKeys. Value cells run
// through Interpolate and then decodeTypedValue so booleans and
// collection literals reach Helm as native YAML types rather than
// quoted strings.
func RenderYAMLFromKeys(keys [][2]string) ([]byte, error) {
rootMap := map[string]any{}
for _, kv := range keys {
segments, err := parsePath(kv[0])
if err != nil {
return nil, fmt.Errorf("render yaml: %w", err)
}
if err := setNested(rootMap, segments, decodeTypedValue(Interpolate(kv[1]))); err != nil {
return nil, fmt.Errorf("render yaml: %w", err)
}
}
body, err := yaml.Marshal(rootMap)
if err != nil {
return nil, fmt.Errorf("render yaml: marshal: %w", err)
}
return body, nil
}

// UpdateYAMLKeys reads the YAML file at path, applies each (dotted-path,
// value) pair as an upsert, and writes the file back. Path syntax uses
// "." between segments and "[n]" for list indices; missing intermediate
Expand Down Expand Up @@ -210,6 +236,30 @@ func SubstituteFileBlock(path, spec string) error {
return SubstituteFile(path, oldBlock, newBlock)
}

// decodeTypedValue converts the YAML-significant literals that Helm
// evaluates differently when quoted. Booleans are decoded because
// Go templates treat the string "false" as truthy. Collection
// literals like "[]" are decoded so Helm sees an empty list instead
// of a non-empty string. Numbers are left as strings: Helm coerces
// them in template expressions, and eagerly parsing "1.0" as a float
// would lose the trailing zero on round-trip.
func decodeTypedValue(s string) any {
switch s {
case "true":
return true
case "false":
return false
}
var decoded any
if err := yaml.Unmarshal([]byte(s), &decoded); err == nil {
switch decoded.(type) {
case []any, map[string]any:
return decoded
}
}
return s
}

// readYAMLAny reads path and unmarshals into a generic any value.
// An empty document parses to nil.
func readYAMLAny(path string) (any, error) {
Expand Down
101 changes: 101 additions & 0 deletions tests/bdd/dsl/yamledit_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,107 @@ func TestSubstituteFileBlockRejectsMissingOldBlock(t *testing.T) {
}
}

// TestRenderYAMLFromKeysBuildsNestedDocument verifies that
// RenderYAMLFromKeys produces a nested YAML structure from dotted-path
// key/value pairs without touching the filesystem.
func TestRenderYAMLFromKeysBuildsNestedDocument(t *testing.T) {
keys := [][2]string{
{"llmRequestRouter.fullnameOverride", "llm-request-router-region-b"},
{"llmRequestRouter.replicaCount", "2"},
{"llmRequestRouter.workload.kind", "StatefulSet"},
}
body, err := RenderYAMLFromKeys(keys)
if err != nil {
t.Fatalf("render: %v", err)
}

out := string(body)
for _, want := range []string{
"fullnameOverride: llm-request-router-region-b",
"replicaCount: \"2\"",
"kind: StatefulSet",
} {
if !strings.Contains(out, want) {
t.Fatalf("output missing %q:\n%s", want, out)
}
}
}

// TestRenderYAMLFromKeysPreservesBoolsAndCollections verifies that
// booleans and collection literals are decoded to native YAML types
// while numbers remain as quoted strings.
func TestRenderYAMLFromKeysPreservesBoolsAndCollections(t *testing.T) {
keys := [][2]string{
{"router.enabled", "true"},
{"router.pki.enabled", "false"},
{"router.replicaCount", "2"},
{"router.discovery.remoteWatchUrls", "[]"},
{"router.name", "region-b"},
}
body, err := RenderYAMLFromKeys(keys)
if err != nil {
t.Fatalf("render: %v", err)
}
out := string(body)

for _, want := range []string{
"enabled: true",
"enabled: false",
"remoteWatchUrls: []",
"name: region-b",
} {
if !strings.Contains(out, want) {
t.Fatalf("output missing %q:\n%s", want, out)
}
}
for _, unwanted := range []string{
`enabled: "true"`,
`enabled: "false"`,
`remoteWatchUrls: "[]"`,
} {
if strings.Contains(out, unwanted) {
t.Fatalf("value emitted as quoted string %q:\n%s", unwanted, out)
}
}
// Numbers stay as quoted strings; Helm coerces them in templates.
if !strings.Contains(out, `replicaCount: "2"`) {
t.Fatalf("replicaCount should remain a quoted string:\n%s", out)
}
}

// TestRenderYAMLFromKeysRejectsInvalidPath confirms that a malformed
// dotted path surfaces as an error instead of a partial document.
func TestRenderYAMLFromKeysRejectsInvalidPath(t *testing.T) {
keys := [][2]string{
{"router.name", "region-b"},
{"router..enabled", "true"},
}
body, err := RenderYAMLFromKeys(keys)
if err == nil || !strings.Contains(err.Error(), "empty segment") {
t.Fatalf("err = %v, want invalid-path error", err)
}
if body != nil {
t.Fatalf("body should be nil on error, got:\n%s", body)
}
}

// TestRenderYAMLFromKeysInterpolatesValues confirms that ${VAR}
// references in value cells are expanded before serialization.
func TestRenderYAMLFromKeysInterpolatesValues(t *testing.T) {
t.Setenv("BDD_TEST_HOST", "region-b.example.invalid")

keys := [][2]string{
{"service.host", "${BDD_TEST_HOST}"},
}
body, err := RenderYAMLFromKeys(keys)
if err != nil {
t.Fatalf("render: %v", err)
}
if !strings.Contains(string(body), "host: region-b.example.invalid") {
t.Fatalf("interpolation failed:\n%s", body)
}
}

func TestParsePathInvalidShapes(t *testing.T) {
bads := []string{
"a..b",
Expand Down
Loading