feat(e2e): Run e2e tests in kube - #470
Merged
matthewgrossman merged 21 commits intoJun 26, 2026
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a composite Kind setup action, rewires Kind CPU CI jobs to use it, updates E2E storage and timeout behavior, and changes jobs-launcher exit handling. ChangesKind CI setup
E2E harness updates
Jobs launcher OTEL and exec flow
Sequence Diagram(s)sequenceDiagram
participant "setup-kind-cluster action" as SetupKind
participant "setup_local_kind_cpu.sh" as SetupLocalKind
participant kubectl as Kubectl
participant "prepull_kind_images.sh" as Prepull
participant "install_helm_e2e.sh" as InstallHelm
participant "wait_for_api.sh" as WaitForApi
SetupKind->>SetupLocalKind: start Kind cluster
SetupKind->>Kubectl: set namespace and verify Gateway API
SetupKind->>Prepull: pre-pull NMP images
SetupKind->>InstallHelm: install NeMo Platform
SetupKind->>WaitForApi: wait for /cluster-info
Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@e2e/test_jobs.py`:
- Around line 380-381: The subprocess skip marker in test_jobs.py is using
NMP_BASE_URL as a proxy for backend mode, but that setting is only for
external-vs-local platform selection. Update the _is_subprocess_mode /
_skip_subprocess logic to check the real backend or cluster configuration used
by the test setup in e2e/conftest.py, so container-backed Kind/Docker runs are
not incorrectly skipped. Use the existing test configuration symbols around the
subprocess/container backend selection to locate the right signal.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3279eb63-e499-4810-912c-982644780ea6
📒 Files selected for processing (6)
.github/actions/setup-kind-cluster/action.yaml.github/workflows/ci.yamle2e/k8s/scripts/install_nmp_e2e.she2e/test_data_designer.pye2e/test_jobs.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/jobs/create.py
Contributor
|
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…iteration All wait_for_job_logs calls now use a consistent 240s timeout to handle K8s OTLP log batching latency (previously ranged 30-120s, causing flakes). Temporarily pins kind-cpu-e2e to a known image tag to skip the ~10min image build while iterating on test changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…pods Two bugs caused ~10-20% of short-lived K8s job pods to lose their logs: 1. **Pipe read race**: cmd.Wait() was called before the stdout/stderr reader goroutines finished. Go's exec.Cmd.Wait() closes pipes on return, so the readers would get "file already closed" and miss the output entirely. Fixed by calling wg.Wait() before cmd.Wait(). 2. **Async batch export**: The BatchProcessor queued log records and exported them asynchronously. ForceFlush triggered the export but returned before the HTTP request completed, and os.Exit killed the in-flight request. Switched to SimpleProcessor which exports each record synchronously — appropriate for the launcher's short-lived single-job use case. Verified: 0/50 log misses on minikube (previously 4-7/30). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@services/core/jobs/jobs-launcher/cmd/run.go`:
- Around line 254-262: The current wait order in the command launcher can hang
when a child keeps stdout/stderr open, so adjust the flow in run.go around
cmd.Wait and wg.Wait. Wait for the process to exit first using cmd.Wait to
capture the main process exit code, then drain the output readers, and add a
bounded fallback so the launcher cannot block forever if EOF never arrives. Keep
the change localized to the launcher logic that coordinates cmd.Wait, wg.Wait,
and the log reader goroutines.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8e26e3ff-d1a1-4f92-b63c-58a8d9333ec7
📒 Files selected for processing (3)
services/core/jobs/jobs-launcher/cmd/otel.goservices/core/jobs/jobs-launcher/cmd/run.goservices/core/jobs/jobs-launcher/cmd/run_test.go
mikeknep
reviewed
Jun 26, 2026
- Lower log wait timeouts from 240s to 60s (launcher fix makes logs available immediately) - Remove install_nmp_e2e.sh (K8s Developer Guide already covers this) - Remove pinned image tag from kind-cpu-e2e, restore build-cpu-smoke-images dependency - Remove unused cluster_url output from setup-kind-cluster action (consumers read NMP_E2E_CLUSTER_URL from env directly) - Update K8s Developer Guide known issues to reflect fixes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rocessor Revert to BatchProcessor (better throughput for jobs with many log lines) now that the defer actually runs. The root cause of lost logs was two-fold: 1. os.Exit in the cobra Run callback skipped deferred OTEL shutdown 2. cmd.Wait() closed stdout/stderr pipes before readers finished Fix (1) by stashing the exit code and calling os.Exit from Execute() after cobra returns and all defers complete. Fix (2) was already in place (wg.Wait before cmd.Wait from prior commit). Verified: 0/50 log misses on minikube with BatchProcessor. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Revert all wait_for_job_logs timeouts to their original values since the launcher log fix makes them unnecessary. Fix stale comment referencing the synchronous processor (we're using BatchProcessor). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
- Add pytest markers: subprocess_only (skipped when NMP_BASE_URL set) and container_only (skipped unless NMP_BASE_URL set) - CI kind-cpu-e2e now runs all e2e/ tests instead of hardcoded file list - Fix smoke health tests to use /status (works on both subprocess and K8s) instead of /health/ready which is internal-only on K8s - Mark auth tests subprocess_only (require auth-enabled platform config) - Replace _skip_subprocess pattern in test_jobs.py with container_only marker Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
changed the title
…a-designer-e2e-tests-for-minikubek8s Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The K8s backend now unconditionally provisions persistent job storage (PVC mount + env var) when the cluster has a PVC configured, matching the subprocess backend which always creates a persistent directory. This eliminates the need for plugins to declare NEMO_JOB_PERSISTENT_JOB_STORAGE_PATH in their compile() environment list — the backend handles it as infrastructure, not application concern. DD's compile() reverts to environment=[] since it never used persistent storage at runtime. Plugins that previously declared the env var (evaluator, agents, anonymizer, etc.) continue to work — their explicit declaration overrides the auto-provisioned default. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Instead of auto-provisioning PVC mounts for every job (wasteful) or crashing with KeyError when the env var is missing (footgun), StoragePaths.persistent is now a property that raises a clear RuntimeError if accessed when not provisioned. Changes: - StoragePaths: dataclass → class with property that guards access - dispatcher.py: tolerate missing PERSISTENT env var, pass None - bridge.py (DD): use os.environ.get(), pass None when absent - K8s backend: revert auto-provisioning, add TODO for first-class requires_persistent_storage field on job spec - test_dispatcher.py: update test to verify property raises on access Jobs that declare the env var in compile() (evaluator, agents, etc.) work unchanged. Jobs that don't (DD) no longer crash — they just can't access ctx.storage.persistent without getting a descriptive error telling them what to do. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…name CI job - ngc_api_key fixture: skip test when key is missing or a CI placeholder (previously asserted and passed with "not-used-for-*" values that caused 400 errors at runtime) - test_job_pause_resume: mark flaky with reruns=2 (pod can error before pause request arrives on Kind due to scheduling timing) - Rename CI job from "Kind CPU e2e (jobs + data-designer)" to "Kind CPU e2e" since it now runs the full e2e/ suite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
commented
Jun 26, 2026
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
crookedstorm
approved these changes
Jun 26, 2026
crookedstorm
left a comment
crookedstorm
left a comment
Contributor
There was a problem hiding this comment.
LGTM from the standpoint of the github segments with concerns mentioned inline for discussion and thought, not blocking.
mikeknep
approved these changes
Jun 26, 2026
mikeknep
left a comment
mikeknep
left a comment
Contributor
There was a problem hiding this comment.
StoragePaths changes LGTM!
ironcommit
reviewed
Jun 26, 2026
…a-designer-e2e-tests-for-minikubek8s Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
deleted the
mgrossman/aircore-844-validate-and-adapt-data-designer-e2e-tests-for-minikubek8s
branch
June 26, 2026 21:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validate and adapt the data designer and jobs e2e tests for Kubernetes, fix a launcher log-loss bug, and add a CI job that runs the full e2e suite against a Kind cluster on every PR.
Data Designer K8s fix
NEMO_JOB_PERSISTENT_JOB_STORAGE_PATHfrom the environment, but on K8s the env var is only injected when the step declares it incompile(). DD never declared it, so the bridge crashed withKeyErroron every K8s run.compile()(which would mount a PVC that DD never uses), we madeStoragePaths.persistenta guarding property that raises a clearRuntimeErrorwhen accessed without being provisioned. The bridge and dispatcher now tolerate the missing env var by passingNone.compile()no longer crashes the pod — it raises a descriptive error only if the job actually tries to use persistent storage.Jobs launcher log reliability fix
Two bugs in the Go jobs-launcher caused ~10-20% of fast-exiting K8s job pods to silently lose their OTLP logs:
cmd.Wait()was called before stdout/stderr reader goroutines finished. Go'sexec.Cmd.Wait()closes pipes on return, so readers got "file already closed" and missed the output entirely. Fixed by callingwg.Wait()beforecmd.Wait().os.Exitskipping defers: The cobraRuncallback calledos.Exit(exitCode), which skipped the deferredotelShutdownthat flushes the OTLP batch processor. Fixed by stashing the exit code and callingos.ExitfromExecute()after cobra returns.E2e test K8s compatibility
test_job_passing_data_between_steps: Added persistent storage env var to step environment (K8s backend requires explicit declaration).pause/resume,invalid_image): Changed from unconditional@pytest.mark.skipto@pytest.mark.container_only— now run on K8s, skip on subprocess.test_job_pause_resume: Cancel after verifying resume works instead of waiting forsleep 300(K8s restarts the timer on pod re-create)./status(works on both subprocess and K8s) instead of/health/ready(internal-only on K8s).subprocess_onlyandcontainer_onlypytest markers with auto-skip logic in conftest.subprocess_only(require auth-enabled platform config).CI:
kind-cpu-e2ejob +setup-kind-clustercomposite actionkind-cpu-e2eworkflow job runs the fulle2e/directory against a Kind cluster (not just hardcoded files).setup-kind-clustercomposite action fromkind-cpu-smoketo deduplicate ~165 lines of cluster setup (kind/kubectl/helm install, cluster creation, gateway verification, image pre-pull, helm deploy, API readiness).ci-statusso it gates PRs.Test plan
kind-cpu-e2eCI job passeskind-cpu-smokeCI job passespython-e2e-testCI job passesCloses AIRCORE-844
Closes AIRCORE-845
🤖 Generated with Claude Code