Adding David's Eval scripts

[kyleclo]

No description provided.

[davidheineman]

Command to run (where s3-prefix is path(s) to the eval results in s3):

pip install -e ".[all]"

# Download and process evaluation results
olmo-cookbook-analysis download \
  --bucket-name "ai2-llm" \
  --s3-prefix "evaluation/peteish32" \
  --local-results-path "peteish32"

# Run paired permutation test
olmo-cookbook-analysis run \
  --type instances \
  --local-results-path "peteish32"

# Compare evaluation to external models
olmo-cookbook-analysis run \
  --type benchmarks \
  --local-results-path "peteish32"

The first command should produce graphs like this:

The second command should produce graphs like this:

Currently this just dumps figures in the img/ folder. I'm still not sure if this is the right form-factor?