Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support configuring Validators #8

Open
cloudronin opened this issue Sep 19, 2024 · 1 comment
Open

Support configuring Validators #8

cloudronin opened this issue Sep 19, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@cloudronin
Copy link
Member

We need a way to configure Pythia to only run one or more configured/enabled validators as part of its processing. Right now its not clear if it runs all of the validators all the time or not.

@cloudronin cloudronin added the enhancement New feature or request label Sep 19, 2024
@cloudronin cloudronin added this to the v0.2 milestone Sep 19, 2024
@pratacosmin
Copy link
Collaborator

pratacosmin commented Sep 26, 2024

Validators

Using validators will increase the processing time for the trace. The first call can take up to 2 minutes because some validators required to download models or additional data. This time will drop to something like 15-20 sec depending also on the number of enabled validators.

Pythia Offers a variety of LLM validators in order to check the input and output for the model.
This validators can be enabled or disabled using the config file configurations/validators/config.yaml.

    - name: detect_pii
      description: "The Anonymize Scanner acts as your digital guardian, ensuring your user prompts remain confidential and free from sensitive data exposure."
      source: llm Guard
      enabled: true
      input: input_reference
      output: input_response

The response for the validators Will be set on the pythia Traces.
image

@cloudronin cloudronin modified the milestones: v0.2, v0.1 Sep 27, 2024
@blidiselalin blidiselalin reopened this Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants