Skip to content

Add prompt subset support for benchmarks#58

Open
tlrmchlsmth wants to merge 4 commits into
mainfrom
prompt-subset-cache-debug
Open

Add prompt subset support for benchmarks#58
tlrmchlsmth wants to merge 4 commits into
mainfrom
prompt-subset-cache-debug

Conversation

@tlrmchlsmth

@tlrmchlsmth tlrmchlsmth commented Jun 25, 2026

Copy link
Copy Markdown
Member

Summary

This PR lets us limit the number of prompts we're working with if we want to jack up the prefix cache hit rate in evals or benchmarks

  • add workload-level prompt_subset / prompt_subset_seed config and Starlark support
  • add eval CLI flags for cycling a deterministic GSM8K/GPQA subset for duration-based runs
  • partition prompt subsets globally before worker assignment to avoid worker collisions
  • replay subsets by stable dataset indices/seeds/offsets instead of storing full prompt conversations in memory

Tests

  • GOCACHE=/Users/tms/code/nyann-bench/.gocache go test ./... -count=1
  • GOCACHE=/Users/tms/code/nyann-bench/.gocache go test ./pkg/dataset -count=1

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth tlrmchlsmth force-pushed the prompt-subset-cache-debug branch from 66f0a79 to de7db17 Compare June 25, 2026 02:29
…ebug

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>

# Conflicts:
#	cmd/nyann-bench/common.go
#	pkg/dataset/corpus.go
#	pkg/dataset/dataset_test.go
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant