Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions tests/bdd/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ logic into `dsl/`.
- Function lifecycle CLI option tables preserve row order, repeated options,
empty values, and product-invalid values. Only the `option | value` table
structure is validated before the command runs.
- The Helm function lifecycle uses `SAMPLE_HELM_FUNCTION_CHART`. The chart
must be publicly reachable by ReVal, expose an `entrypoint` Service on port
8000, answer `/health`, and echo the request message from `/echo`. Keep the
chart reference outside the feature so release versions are not hard-coded.
- Gateway API route readiness tables expose each route's kind, name,
namespace, and intended Gateway parent plus the shared context and timeout.
The step requires `Accepted=True` and `ResolvedRefs=True` for that parent but
Expand Down
9 changes: 6 additions & 3 deletions tests/bdd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,10 @@ go test -short ./...

Live runs build nvcf-cli, bring up a real k3d cluster, and exercise the
feature end to end. They require an NGC API key and sample registry
coordinates.
coordinates. The single-cluster Helmfile run also requires
`SAMPLE_HELM_FUNCTION_CHART`. It must identify a chart that ReVal can fetch
without credentials. The chart must expose an `entrypoint` Service on port
8000, answer `/health`, and echo the request message from `/echo`.

Each `-run` argument is anchored with `^...$` so the live entry point
runs without also matching its `...FeatureFileWiresToSteps` wiring
Expand All @@ -67,9 +70,9 @@ go test -run '^TestSingleClusterUpOneClick$' -timeout 30m -v
# Multi-cluster CLI feature
go test -run '^TestMultiClusterUp$' -timeout 60m -v

# Single-cluster Helmfile feature (requires NGC_API_KEY, SAMPLE_NGC_ORG,
# SAMPLE_NGC_TEAM)
# Single-cluster Helmfile feature (also requires a public sample Helm chart)
NGC_API_KEY=<key> SAMPLE_NGC_ORG=<org> SAMPLE_NGC_TEAM=<team> \
SAMPLE_HELM_FUNCTION_CHART=<chart-url-or-oci-reference> \
go test -run '^TestSingleClusterHelmfile$' -timeout 90m -v

# Focused single-cluster Helmfile feature for the documented public Docker Hub
Expand Down
35 changes: 35 additions & 0 deletions tests/bdd/features/single-cluster-helmfile.feature
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,41 @@ Feature: Install a local single-cluster NVCF stack with Helmfile
# scenario's deployment at once.
And I successfully undeploy the function selected by NVCF CLI

@function-lifecycle @helm-function
Scenario: Operator creates, deploys, and invokes a Helm chart function
Given environment variable "SAMPLE_HELM_FUNCTION_CHART" is set
And I use NVCF CLI config "${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml"

# Function creation and deployment both cross the ReVal chart-rendering
# boundary. A deployed ReVal pod alone does not prove this path works.
When I successfully run command:
"""
${NVCF_CLI} --config ${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml function create --name bdd-helm-function --helm-chart ${SAMPLE_HELM_FUNCTION_CHART} --helm-chart-service entrypoint --inference-url /echo --inference-port 8000 --health-uri /health --health-port 8000 --health-timeout PT30S
"""

And I successfully deploy the function selected by NVCF CLI with options:
| option | value |
| --gpu | H100 |
| --instance-type | NCP.GPU.H100_1x |
| --backend | ncp-local |
| --regions | us-west-1 |
| --min-instances | 1 |
| --max-instances | 1 |
| --timeout | 900 |

And I successfully generate a function API key with CLI options:
| option | value |
| --description | bdd-helm-function |
| --scopes | invoke_function,list_functions,queue_details,list_functions_details |

When I successfully invoke the function selected by NVCF CLI over HTTP with timeout "120" seconds and poll duration "5" seconds:
"""
{"message":"bdd-helm-echo","repeats":1}
"""
Then the command output should contain "bdd-helm-echo"

And I successfully undeploy the function selected by NVCF CLI

@function-lifecycle @grpc
Scenario: Operator creates, deploys, and invokes the gRPC Load Tester Supreme sample function
Given I use NVCF CLI config "${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml"
Expand Down
19 changes: 18 additions & 1 deletion tests/bdd/godog_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -438,11 +438,14 @@ func TestSingleClusterHelmfileFeatureFileWiresToSteps(t *testing.T) {
const vanityInvokeCommand = "/usr/bin/nvcf-cli --config /repo-root-placeholder/tests/bdd/fixtures/nvcf-cli-local.yaml" +
" function invoke --vanity-host vanity.localhost --path /bdd/echo --timeout 120" +
" --request-body '{\"message\":\"bdd-vanity-echo\",\"repeats\":1}'"
const helmInvokeCommand = "/usr/bin/nvcf-cli --config /repo-root-placeholder/tests/bdd/fixtures/nvcf-cli-local.yaml" +
" function invoke --request-body '{\"message\":\"bdd-helm-echo\",\"repeats\":1}' --timeout 120 --poll-duration 5"
home := t.TempDir()
t.Setenv("HOME", home)
t.Setenv("NGC_API_KEY", "test-key")
t.Setenv("SAMPLE_NGC_ORG", "test-org")
t.Setenv("SAMPLE_NGC_TEAM", "test-team")
t.Setenv("SAMPLE_HELM_FUNCTION_CHART", "https://charts.example.test/inference-test-0.1.0.tgz")
t.Setenv("NVCF_CLI", "/usr/bin/nvcf-cli")
t.Setenv("REPO_ROOT", "/repo-root-placeholder")
runner := newFakeRunner(map[string]harness.Result{
Expand All @@ -463,6 +466,10 @@ func TestSingleClusterHelmfileFeatureFileWiresToSteps(t *testing.T) {
ExitCode: 0,
Stdout: "Function invocation completed!\n\nResponse:\n{\"rawResponse\":\"bdd-vanity-echo\"}\n",
},
helmInvokeCommand: {
ExitCode: 0,
Stdout: "Function invocation completed!\n\nResponse:\n{\"rawResponse\":\"bdd-helm-echo\"}\n",
},
"/usr/bin/nvcf-cli --config /repo-root-placeholder/tests/bdd/fixtures/nvcf-cli-local.yaml function invoke" +
" --grpc --grpc-plaintext --grpc-service Echo --grpc-method EchoMessage" +
" --request-body '{\"message\":\"bdd-grpc-echo\"}' --timeout 120 --poll-duration 5": {
Expand Down Expand Up @@ -563,6 +570,16 @@ func TestSingleClusterHelmfileFeatureFileWiresToSteps(t *testing.T) {
"--description bdd-grpc-load-tester-supreme") {
t.Fatal("gRPC sample function API key was not generated for the function service")
}
if !commandRanThatContainsAll(suite.Runner.(*fakeRunner).runs,
"function create --name bdd-helm-function",
"--helm-chart https://charts.example.test/inference-test-0.1.0.tgz",
"--helm-chart-service entrypoint",
"--inference-url /echo --inference-port 8000") {
Comment thread
sbaum1994 marked this conversation as resolved.
t.Fatal("Helm sample function was not created through the chart-rendering path")
}
if !commandRanExactly(suite.Runner.(*fakeRunner).runs, helmInvokeCommand) {
t.Fatal("Helm sample function was not invoked")
}
if !commandRanThatContainsAll(suite.Runner.(*fakeRunner).runs,
"function create --name bdd-openai-compatible-sample",
"nvcf-openai-compatible-sample:local",
Expand All @@ -576,7 +593,7 @@ func TestSingleClusterHelmfileFeatureFileWiresToSteps(t *testing.T) {
if !commandRanThatContains(suite.Runner.(*fakeRunner).runs, "function delete --deployment-only") {
t.Fatal("function deployment cleanup was never invoked")
}
assertFunctionDeploymentsUseInstanceType(t, suite.Runner.(*fakeRunner).runs, "NCP.GPU.H100_1x", 3)
assertFunctionDeploymentsUseInstanceType(t, suite.Runner.(*fakeRunner).runs, "NCP.GPU.H100_1x", 4)
if !commandRanThatContains(suite.Runner.(*fakeRunner).runs, "http://llm.localhost:8080/v1/chat/completions") {
t.Fatal("unauthenticated LLM gateway check was never invoked")
}
Expand Down
Loading