Skip to content

Conversation

jmartin-tech
Copy link
Collaborator

Fix #1344

Corrects role based last_message() selection and correct ETL format for judge detectors based on #1254

Verification

List the steps needed to make sure this thing works

  • Supporting configuration file judge_det.yaml
plugins:
  detectors:
    judge:
      Refusal:
        detector_model_type: nim
        detector_model_name: meta/llama-3.1-70b-instruct
        confidence_cutoff: 7
  • NIM_API_KEY=<my-api-key> python3 -m garak -m test.Blank -p test.Blank -d judge.Refusal --config judge_det.yaml
  • Verify the run succeeds
  • Verify all automation tests pass

@leondz leondz mentioned this pull request Aug 28, 2025
1 task
Copy link
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • note re: land order
  • test message request

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catches. both changed areas also altered in #1337, mind land order & that role update propagates. test /should/ catch this in its current form even though it doesn't target these two fixes directly

Signed-off-by: Jeffrey Martin <[email protected]>
@jmartin-tech jmartin-tech changed the title Fix/judge detectors conversation refactor judge detectors conversation refactor Aug 28, 2025
jmartin-tech added a commit to jmartin-tech/garak that referenced this pull request Aug 28, 2025
@jmartin-tech jmartin-tech merged commit 0a7991c into NVIDIA:main Aug 29, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Aug 29, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Possible bug with detector judge.Refusal when set up with nim generator.
2 participants