Rhesis: Open-Source Gen AI Testing

Your team defines expectations, Rhesis generates and executes thousands of test scenarios. So that you know what you ship.

Rhesis is an open-source testing platform that transforms how Gen AI teams validate their applications. Collaborative test management turns domain expertise into comprehensive automated testing: legal defines requirements, marketing sets expectations, engineers build quality, and everyone knows exactly how the Gen AI application performs before users do.

🎯 Why Rhesis?

The Gen AI Testing Challenge

Gen AI applications present unique testing challenges that traditional approaches can't handle:

• Non-deterministic outputs: Same input, different responses
• Unexpected edge cases: Unpredictable user inputs lead to problematic outputs
• Ethical risks: Biased, harmful, or inappropriate content generation
• Compliance requirements: Industry-specific regulatory standards

Traditional testing with hand-coded scenarios can't scale to unlimited user creativity. Rhesis addresses these challenges through collaborative test management that generates comprehensive automated coverage.

Make testing a peer to development

You've transformed your product with Gen AI, now transform how you test it. Testing deserves the same sophistication as your development tooling.

Your whole team should define what matters

Your legal, marketing, and domain experts know what can actually go wrong. Rhesis makes testing everyone's responsibility.

Know what you're shipping

The best AI teams understand their system's capabilities before release. Get complete visibility into how your Gen AI performs across thousands of real-world scenarios.

✨ Key Features

• Collaborative Test Management: Your entire team contributes requirements, legal, compliance, marketing, domain experts, all without writing code
• Automated Test Generation: Automatically generate thousands of test scenarios from team expertise, requirements and existing knowledge sources
• Comprehensive Coverage: Scale from dozens of manual tests to thousands of automated scenarios that match your AI's complexity
• Edge Case Discovery: Find potential failures before your users do with sophisticated scenario generation
• Compliance Validation: Ensure Gen AI systems meet regulatory and ethical standards with team-defined requirements
• Performance Analytics: Track quality metrics over time

🌐 Open Source & Community-Driven

Rhesis is built by Gen AI developers who experienced inadequate testing tools firsthand. The core platform and SDK remain MIT-licensed forever, with a clear commitment: core functionality never moves to paid tiers. All commercial code lives in dedicated ee/ folders.

Join our community calls to discuss roadmap, features, and contributions. Connect via Discord for announcements.

📑 Repository Structure

This monorepo contains the complete Rhesis ecosystem:

rhesis/
├── apps/
│   ├── backend/       # FastAPI backend service
│   ├── frontend/      # React frontend application
│   ├── worker/        # Celery worker for background tasks
│   ├── chatbot/       # Conversational testing interface
│   └── polyphemus/    # Uncensored LLM for comprehensive test generation
├── sdk/               # Python SDK for Rhesis
├── infrastructure/    # Infrastructure as code
├── scripts/           # Utility scripts
└── docs/              # Documentation

🚀 Quick Start

Option 1: Use the cloud platform (fastest)

Get started in minutes at app.rhesis.ai:

1. Create a free account
2. Start generating test scenarios collaboratively
3. Invite your team to define requirements together

Option 2: Use the SDK

Install and configure the Python SDK:

pip install rhesis-sdk

import rhesis 

rhesis.api_key = "rh-XXXXXXXXXXXXXXXXXXXX"  # Get from app.rhesis.ai settings
rhesis.base_url = "https://api.rhesis.ai"  # optional

Quick example:

from rhesis.sdk.entities import TestSet
from rhesis.sdk.synthesizers import PromptSynthesizer

# Browse available test sets
for test_set in TestSet().all():
    print(test_set)

# Generate custom test scenarios
synthesizer = PromptSynthesizer(
    prompt="Generate tests for a medical chatbot that must never provide diagnosis"
)
test_set = synthesizer.generate(num_tests=100)

Option 3: Run locally with Docker

Clone and run the full platform locally:

git clone https://github.com/rhesis-ai/rhesis.git
cd rhesis
docker-compose up

Visit http://localhost:3000 to access your local instance.

Detailed setup instructions: See component READMEs in Component Documentation.

🔄 Versioning

Each component follows Semantic Versioning with component-specific tags:

• backend-v1.0.0 - Backend releases
• frontend-v2.3.1 - Frontend releases
• sdk-v0.5.2 - SDK releases

See CONTRIBUTING.md for details.

👥 Contributing

Rhesis thrives thanks to our community. Here's how you can contribute:

Ways to Contribute

• Code: Fix bugs, implement features, improve performance
• Test Sets: Contribute test cases for common AI failure modes
• Documentation: Enhance guides, tutorials, and API references
• Community Support: Help others in Discord or GitHub discussions
• Feedback: Report bugs, request features, share your experience

Contributing Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes with tests
4. Commit with clear messages
5. Push and open a pull request

We review PRs regularly and maintain a welcoming environment through our code of conduct.

Detailed guidelines: CONTRIBUTING.md
Release process: RELEASING.md

📝 License

Community Edition: MIT License - see LICENSE file for details.

Enterprise Edition: Enterprise features located in ee/ folders are subject to separate commercial licenses. Contact [email protected] for enterprise licensing information.

🆘 Support

• Documentation: docs.rhesis.ai
• Discord Community: discord.rhesis.ai
• GitHub Discussions: Community discussions
• Email: [email protected]
• Issues: Report bugs or request features

---

Made with in Potsdam, Germany

Learn more at rhesis.ai