Summary
Seven small fixes, each a few lines.
run_experiment() has no summary output. Add a verbose: bool = True parameter that prints a result table at the end using rich. (executor/client.py)
- 207 partial-ingest failures are logged but the run proceeds as if everything worked. The caller should know which scores failed. (
scores_client.py:45-54)
task_output.pop("_interaction_trace_id", ...) mutates the dict returned by the task. Take a shallow copy first. (client.py:152)
evaluation_runs[-1] in client.py:208 assumes the last item in the list belongs to the current example. No await exists between the append and this read today, but a single await added later would silently corrupt results. Capture the object directly at append time instead.
validate_log_level runs after Pydantic's Literal coercion so the check always passes. It's dead code. Remove it. (config.py)
- CI build provenance is hardcoded to
None. Read the standard Buildkite env vars (BUILDKITE_BUILD_ID, BUILDKITE_JOB_ID, etc.) when present. (client.py:111)
except httpx.HTTPError: raise in scores_client.py:42-43 catches an exception and immediately re-raises it unchanged. Remove it.
Done when
Summary
Seven small fixes, each a few lines.
run_experiment()has no summary output. Add averbose: bool = Trueparameter that prints a result table at the end usingrich. (executor/client.py)scores_client.py:45-54)task_output.pop("_interaction_trace_id", ...)mutates the dict returned by the task. Take a shallow copy first. (client.py:152)evaluation_runs[-1]inclient.py:208assumes the last item in the list belongs to the current example. Noawaitexists between the append and this read today, but a singleawaitadded later would silently corrupt results. Capture the object directly at append time instead.validate_log_levelruns after Pydantic'sLiteralcoercion so the check always passes. It's dead code. Remove it. (config.py)None. Read the standard Buildkite env vars (BUILDKITE_BUILD_ID,BUILDKITE_JOB_ID, etc.) when present. (client.py:111)except httpx.HTTPError: raiseinscores_client.py:42-43catches an exception and immediately re-raises it unchanged. Remove it.Done when