Skip to content

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented Aug 11, 2025

adds single-scalar aggregating from garak results, biased by tier

consumes:

  • report digest object

relies on:

  • absolute scores in digest
  • relative scores in digest
  • defcons in digest
  • tier defs in digest

outputs:

  • 2s.f. / 1d.p. score [1.0,5.0]

usage:

  • python -m garak.analyze.tbsa <report.jsonl filepath>

open questions/todo:

  • choose between current garak calibration and calibration in report file
  • what if there's no calibration data available overall
  • do we fill in gaps if major (0.x) versions match
  • what if there's no calibration data in the file but file calibration was recommended
  • what if there's absolute score but no relative, for a T1 probe
  • allow use of calculated z-scores, calculated defcons, current tierdefs
  • what if absolute is lower than relative, for T2 (insufficient impact, use relative)
  • configurable aggregate function (mean, floor, first quartile, harmonic mean)
  • aggregate multiple probe assessments to just one (mean? floor? harmonic? as specified by probe?)
  • load current / custom calibration
  • how strictly should we fail? (all missing relative? any missing relative? cutoff?)
  • how to handle groups? mean of scores per group? one DC per group?
  • tests
  • should this be included in report digest (i guess), let's do that non-circularly

@leondz leondz self-assigned this Aug 11, 2025
@leondz leondz added the reporting Reporting, analysis, and other per-run result functions label Aug 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
reporting Reporting, analysis, and other per-run result functions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant