Skip to content

test: assert service-selected workers follow the service's runtime hint - #1100

Merged
leongdl merged 2 commits into
aws-deadline:mainlinefrom
leongdl:test/service-selected-default-pythonexpr
Sep 30, 2026
Merged

leongdl merged 2 commits into
aws-deadline:mainlinefrom
leongdl:test/service-selected-default-pythonexpr

Conversation

@leongdl

@leongdl leongdl commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What was the problem/requirement? (What/Why)

TestServiceSelectedDefaultsToPython fails on the Windows canaries: Expected agent log to contain 'Selected session runtime: python (hint=None)'. The job succeeds; only the log assertion fails.

The service now stamps a runtimeHint on every session: pythonexpr by default, rust for allowlisted accounts. botocore 1.43.105 added AssignedSession.metadata to the Deadline model. Older botocore dropped the field, so the agent always saw hint=None. Expecting None only matched how the service behaved during development. Choosing the hint is the service's job, and the agent's job is to route on it.

What was the solution? (How)

  • Replaced TestServiceSelectedDefaultsToPython and the two skipped hint classes (TestServiceSelectedWithRustHint, TestServiceSelectedWithPythonexprHint) with one TestServiceSelectedFollowsServiceHint. It requires python (hint='pythonexpr') or rust (hint='rust'), so the test fails if the hint is dropped anywhere between the service and the scheduler. The no-hint default stays covered by test/unit/sessions/runtime/test_select.py.
  • _assert_log_contains now accepts a sequence of patterns and passes if any one matches: grep -F -e … -e … on Linux, a Select-String -Pattern 'a','b' -SimpleMatch array on Windows. Single-string callers are unchanged.
  • The class is xfail(strict=False) on Linux. The Linux e2e worker runs Python 3.9, which caps botocore at 1.42.x, and 1.42.x has no metadata, so the hint never arrives there. Remove the mark once the Linux worker's botocore keeps the field; until then it reports XPASS.

What is the impact of this change?

Test-only. Windows now asserts that the hint reaches the worker. Linux xfails until its botocore catches up. The test also stays correct through a later move to rust without edits.

How was this change tested?

  • ruff check, ruff format --check (pinned ruff ~= 0.15.22) and mypy --config-file test/e2e/mypy.ini pass.
  • Ran the built grep and PowerShell commands locally against real canary agent log lines:
log line Linux (grep) Windows (pwsh 7.4)
python (hint='pythonexpr') match match
rust (hint='rust') match —
python (hint=None) no match no match
python (hint='rust') (ignored hint) no match —
  • pytest --collect-only with OPERATING_SYSTEM=linux and windows collects the new test. On Linux the xfail condition evaluates to true. The e2e suite itself runs in CI.

Was this change documented?

Module and class docstrings updated.

Is this a breaking change?

No.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl
leongdl requested a review from a team as a code owner September 30, 2026 03:13
@github-actions github-actions Bot added the waiting-on-maintainers Waiting on the maintainers to review. label Sep 30, 2026
Comment thread test/e2e/test_session_runtime.py Outdated
retired, at the same time as the hint-following classes below are enabled.
"""Worker with session_runtime=service-selected defaults to python.

The service stamps runtimeHint=pythonexpr by default, but the worker only

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new premise here contradicts the skip reason on TestServiceSelectedWithPythonexprHint below. This docstring says the service already stamps runtimeHint=pythonexpr by default, but line 318-323 skips that class because it "Requires the test account to be in the intermediate allowlist state ... so the service stamps runtimeHint=pythonexpr" — i.e. it is gated on a precondition this docstring asserts is now satisfied. The module docstring (lines 25-26) carries the same stale claim.

If the pythonexpr hint is now stamped by default, TestServiceSelectedWithPythonexprHint should be un-skipped (it is the only test that actually pins hint='pythonexpr', and it would cover the branch this class stopped covering). If it is not reliably stamped, this docstring overstates it. Either way one of the two needs updating so a future reader can tell which state the account is in.

Comment thread test/e2e/test_session_runtime.py Outdated

The service stamps runtimeHint=pythonexpr by default, but the worker only
sees it when its botocore model has AssignedSession.metadata, so the hint
is None on some platforms. Both route to python, so the assertion does not

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the hint is None on some platforms" attributed to the botocore model looks like the wrong cause. The hint is read at scheduler.py:1170 from session_spec["metadata"]["runtimeHint"], and the botocore service model the agent uses is shipped with the package — it is identical on Linux and Windows, so it cannot make the hint present on one OS and absent on the other. If the hint really is intermittently absent, the cause is more likely a stale/vendored model override or the endpoint/region the test account resolves to, not the platform.

Since this whole diff exists to make the docstring match reality, it is worth naming the actual reason (or saying "absent in some environments — cause not yet pinned down"); a plausible-but-wrong explanation here will send the next person debugging a hint mismatch down the wrong path.

Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl leongdl changed the title test: accept the pythonexpr default in the service-selected e2e test test: assert service-selected workers follow the service's runtime hint Sep 30, 2026
Comment thread test/e2e/test_session_runtime.py
Comment thread test/e2e/test_session_runtime.py
@leongdl

leongdl commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Review guide

TLDR: The service picks the runtime hint (pythonexpr by default, rust when allowlisted), and the agent routes on it. The old test expected hint=None, which was only true while the service didn't send a hint. The new test checks that the hint reaches the worker and that the agent picks the matching runtime.

Where to look (one file, ~5 min):

  1. TestServiceSelectedFollowsServiceHint: replaces the three old service-selected classes. It passes only on python (hint='pythonexpr') or rust (hint='rust').
  2. The Linux xfail: the Linux e2e worker runs Python 3.9, which caps botocore below 1.43.105, the first release with AssignedSession.metadata. There the hint is dropped before the agent sees it. It's strict=False, so it shows XPASS once Linux catches up. Remove it then.
  3. _assert_log_contains: now takes a list of patterns and passes if any one matches (grep -F -e … -e … / Select-String -Pattern 'a','b'). Single-string callers are unchanged.

What it catches:

agent log result
python (hint='pythonexpr') pass
rust (hint='rust') pass
python (hint=None): hint dropped fail
python (hint='rust'): hint ignored fail

The no-hint default is still covered by the unit tests in test/unit/sessions/runtime/test_select.py.

Comment thread test/e2e/test_session_runtime.py
@leongdl
leongdl merged commit c94d974 into aws-deadline:mainline Sep 30, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-maintainers Waiting on the maintainers to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants