An ethical agent auditor for reviewing AI agent decisions in real time — scoring proposed actions against harm minimization, fairness, transparency, privacy, and safety with rule flags, RAG context, and transparent, exportable feedback.
Pairs with Ethical Decision Simulator — the simulator creates ethical decisions; the auditor evaluates decisions made by agents.
Educational tool: This app reviews proposed agent actions for ethical risk — it does not provide legal advice, compliance certification, therapy, or crisis support. Model outputs may be incorrect; users are responsible for their own decisions and organizational policies.
Privacy (cloud demo): Agent actions and context you enter are sent to Groq for inference when using the public Streamlit Cloud demo (or any cloud deployment with
LLM_PROVIDER=groq). Do not submit classified data, credentials, PII, or sensitive organizational details you cannot share with Groq. Prefer curated Examples or synthetic actions for testing. Session state is not written to a project database; see Privacy & data below.
Table of Contents
| Section | Description |
|---|---|
| Get started | |
| 🚀 Live Demo | Try the live agent-audit demo (Cloud or local) |
| ✨ Features | Capabilities at a glance |
| ⚡ Quick Start | Clone, install, run, test, Docker |
| Overview | |
| 🎯 Problem & Motivation | Why runtime ethical oversight matters |
| 🛠️ Tech Stack | Languages, libraries, CI |
| 📊 Data Sources & Attribution | Principles, examples, EDS scenarios |
| 📌 Version history | Release highlights |
| Technical | |
| 🏗️ Architecture & Design Choices | System design and pipeline |
| ↳ Full architecture doc | Detailed system design (Mermaid) |
| ↳ Development Journey | Capability pipeline diagram |
| 🛡️ Safety Considerations | Ethics, guardrails, and data privacy |
| 🔄 CI/CD | GitHub Actions, Docker, and deployment |
| 📈 Project Status | v1.0.0 capabilities |
| 📁 Repository Layout | File tree |
| Legal & contact | |
| 📄 License | MIT License |
| 🤝 Contact / Next Steps | Feedback and V2 roadmap |
▶ Open the live app on Streamlit Cloud
Before you open the app:
- Cold start: This app runs on Streamlit Community Cloud and may go to sleep after inactivity. If you see “Zzzz — This app has gone to sleep due to inactivity”, click “Yes, get this app back up!” to wake it — anyone can do this; no special permissions required. Startup may take a minute after you click.
- Privacy: The public demo sends proposed agent actions and context to Groq for inference. Do not enter classified data, credentials, PII, or sensitive details you cannot share with a third-party LLM API. Prefer Examples or synthetic actions. Session results stay in browser memory only — not stored in a project database. For local-only inference, run with
LLM_PROVIDER=ollama(see Quick Start). Full details: Privacy & data.
Local demo: streamlit run app.py · Docker: docker compose up --build → http://localhost:8501
Audit — overall score, risk band, rule flags, and per-principle evidence with suggestions. LLM-generated narrative text may differ by provider, model, and run.
Examples — load agent-action scenarios or Ethical Decision Simulator dilemmas into Audit with one click.
LangSmith observability (optional) — traced audit_agent_action run for the delete-without-confirmation example, including inputs, latency, and structured audit output. Trace contents and scores may differ across models, prompts, and runs; the public Streamlit demo does not require LangSmith.
- Audit — paste a proposed agent action (+ optional context); get overall score, risk level, rule flags, and five-principle breakdown
- Hybrid evaluation — deterministic rule pre-checks injected into Groq LLM scoring
- RAG context — Chroma retrieval over principles, agent examples, and EDS dilemmas
- Examples — curated risky/safe agent actions with one-click load into Audit
- EDS pairing — load dilemmas from Ethical Decision Simulator for complementary audits
- Export JSON — download structured
AuditReportafter a run - LangSmith tracing — optional observability when configured (private dashboard; not required for end users)
- Simulated mode — automatic fallback when the API is unavailable (clearly labeled)
- Python 3.11+ or Docker
- Groq API key for live cloud models
- (Optional) Ollama with
gemma3:4bfor local development - (Optional) LangSmith API key for tracing
git clone https://github.com/rvong65/ethical-agent-auditor
cd ethical-agent-auditor
python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements.txt
copy .streamlit\secrets.toml.example .streamlit\secrets.toml
# Edit secrets.toml — set GROQ_API_KEY (optional LangSmith keys)
python scripts\build_chroma_index.py
streamlit run app.pyLocal Ollama profile:
$env:LLM_PROVIDER = "ollama"
streamlit run app.pydocker compose up --build
# Open http://localhost:8501Optional: create a .env file with GROQ_API_KEY=your_key (never commit). Without a key, the app runs in Simulated mode.
pytest tests/ -v --ignore=tests/test_api_integration.py --ignore=tests/test_langsmith.py # 36 offline (matches CI)
pytest tests/test_api_integration.py -v -m api # live Groq (local / optional)
pytest tests/test_langsmith.py -v -m langsmith # LangSmith smoke (optional)See docs/architecture.md#deployment-topologies for local, Docker, and Streamlit Cloud details.
As AI agents act with less human supervision, a single proposed action — delete user data, deny a loan, hide that a decision was automated — can cause irreversible harm. Teams need runtime ethical oversight: transparent scores, deterministic red flags, and actionable suggestions — not a black-box “approved/blocked” bit.
This project makes that visible: paste a proposed action (or load an EDS dilemma), run the hybrid pipeline, and inspect principle scores, rule flags, recommendations, and JSON exports. It is an educational ethics auditing tool, not legal authority or a compliance product.
| Layer | Technology |
|---|---|
| UI | Streamlit (st.navigation, Home / Audit / Examples) |
| Cloud LLM | Groq API (GPT-OSS 20B, GPT-OSS 120B) |
| Local LLM | Ollama (gemma3:4b) — optional dev profile |
| Schemas | Pydantic v2 AuditReport / principle scores |
| RAG | ChromaDB (embedded, pre-built data/chroma/) |
| Observability | LangSmith @traceable (optional) |
| Tests | pytest — 36 offline + live API + LangSmith smoke |
| CI / Docker | GitHub Actions + Dockerfile / docker-compose.yml |
Principles and agent-action examples are original project content. EDS dilemmas are bundled from Ethical Decision Simulator for complementary audits — not a proprietary third-party dataset.
Cloud demo uses production Groq models only (preview/evaluation models are excluded).
| Model | Role | License / acknowledgment |
|---|---|---|
gemma3:4b (Ollama) |
Local dev LLM | Gemma Terms of Use |
openai/gpt-oss-20b (Groq) |
Cloud LLM — default / fast | OpenAI open-weight via Groq — Groq Terms |
openai/gpt-oss-120b (Groq) |
Cloud LLM — high-reasoning tier | OpenAI open-weight via Groq — Groq Terms |
| Asset | Path |
|---|---|
| Principles & rubrics | src/data/principles.json |
| Agent-action examples | src/data/examples.json |
| EDS dilemmas | src/data/eds_scenarios.json |
| Pre-built RAG index | data/chroma/ (committed for reproducible deploys) |
| Version | Date | Highlights |
|---|---|---|
| 1.0.0 | 2026-07-21 | MVP: hybrid auditor, Examples + EDS, CI, Docker, docs |
Full details: CHANGELOG.md
Proposed agent actions flow through rules → RAG → resilient LLM inference → JSON parse → aggregation → Streamlit presentation. Deterministic rule flags are injected into the LLM prompt so pre-checks are not ignored. Mock fallback and educational disclaimers keep the tool usable and responsible without silent simulation.
Full design — goals, end-to-end diagram, module map, deployment topologies (local / Docker / Streamlit Cloud), and architecture-level safety: docs/architecture.md
Key design decisions
| Decision | Rationale |
|---|---|
| Hybrid rules + LLM | Fast deterministic signals plus nuanced multi-criteria judgment |
| Flags in LLM context | Pre-checks influence scores and evidence |
| Five agent-audit principles | Distinct from EDS philosophical frameworks; applied runtime oversight |
| Embedded Chroma (pre-built) | Streamlit Cloud friendly without a hosted vector DB |
Shared execute_audit path |
UI and pytest exercise the same Run Audit pipeline |
| GPT-OSS 20B / 120B only | Avoid Llama 8B/70B deprecation (2026-08-16) |
| Optional LangSmith | Observability for operators; not an end-user product surface |
| Mock fallback with labels | App stays usable when Groq rate-limits or API key is missing — never silent |
Capability pipeline (Audit, Examples, EDS pairing, tests, and deploy). Full design: docs/architecture.md#development-journey.
flowchart LR
A[Principles + rules] --> B[Groq evaluator<br/>Pydantic reports]
B --> C[Chroma RAG<br/>LangSmith traceable]
C --> D[Streamlit UI<br/>Home · Audit · Examples]
D --> E[EDS pairing<br/>pipeline tests]
E --> F[GPT-OSS models<br/>Docker · CI · docs]
| Principle | Implementation |
|---|---|
| Educational use only | Framed as an alignment auditing tool — not legal advice, compliance certification, or crisis intervention (Home footer + README) |
| Transparency | Structured AuditReport, labeled live vs mock runs, per-principle evidence |
| No warranty | Disclaimers state LLM outputs may be wrong; users apply organizational policies |
| Honest simulation | Mock fallback on rate limits or API failure — always labeled Simulated mode, never silent |
| Data minimization | Session-only state; no project database; JSON export is user-initiated |
| Third-party inference | Groq/Ollama terms apply; optional LangSmith receives traces when enabled |
| Privacy awareness | Cloud-demo privacy notice, in-app Privacy & data expander, README data-flow table |
Ethical Agent Auditor is designed for educational alignment auditing, not as a certified data-processing platform. Understand where your input goes:
| Data you submit | Where it may be sent | Notes |
|---|---|---|
| Proposed agent actions & context | LLM provider (Groq on cloud demo, or local Ollama) | Cloud: processed per Groq Terms and their privacy policy |
| Model & tracing settings | Same LLM provider / LangSmith (optional) | Included in prompts; LangSmith traces when configured |
| Rule-check results | Same LLM provider | Deterministic flags are passed to the model as audit context |
| Session results & sidebar state | Streamlit session memory | Cleared when the session ends; not written to a project database by this app |
| JSON exports (Audit) | Your browser only | User-initiated download after an audit run |
Public Streamlit Cloud demo: Treat it like any shared SaaS UI — no classified data, credentials, PII, or sensitive organizational details you cannot share with Groq. Prefer built-in Examples or synthetic agent actions for testing.
Sensitive environments: Run locally or in Docker with LLM_PROVIDER=ollama so inference stays on your network (requires local Ollama with gemma3:4b).
Note: This app does not intentionally log agent actions to disk. Streamlit Community Cloud, Groq, and LangSmith may retain operational or API logs under their own policies — review their documentation if compliance matters.
GitHub Actions runs on every push and pull request to main / master:
| Job | Step | Action |
|---|---|---|
| offline-tests | Trigger | Push or PR to main / master |
| Environment | ubuntu-latest, Python 3.11 |
|
| Install | pip install -r requirements.txt |
|
| Test | pytest ignoring API/LangSmith — 36 offline tests (rules, schemas, aggregator, all agent examples + EDS dilemmas on mock) |
|
| docker-tests | Build | docker build -t ethical-agent-auditor:ci . |
| Test | Run 36 offline tests inside the container | |
| Run | docker compose up on port 8501 (no API key required — Simulated mode) |
|
| Health | Streamlit /_stcore/health |
|
| api-smoke-tests | Trigger | Push to main only (after offline-tests pass) |
| Test | pytest tests/test_api_integration.py -m api — 3 live Groq smokes (needs GROQ_API_KEY secret) |
|
| langsmith-smoke-tests | Trigger | Push to main only |
| Test | pytest tests/test_langsmith.py -m langsmith (needs GROQ_API_KEY + LANGCHAIN_API_KEY) |
Workflow file: .github/workflows/tests.yml
Releases: Version history in CHANGELOG.md.
No API keys required for CI gate — offline tests and the Docker job use fixtures and mocks only; Docker verifies the image builds, tests pass, and Streamlit responds.
API / LangSmith jobs require repository secrets on main. Run them locally anytime with the commands in Quick Start.
Streamlit Cloud deploys independently from the main branch when connected (app.py + requirements.txt + secret GROQ_API_KEY). LangSmith secrets are optional and not required for the public demo.
Docker locally:
docker compose up --build
docker run --rm ethical-agent-auditor:ci pytest tests/ -v --ignore=tests/test_api_integration.py --ignore=tests/test_langsmith.pyCurrent release: v1.0.0 — Streamlit demo · GitHub repo
| Capability | Included |
|---|---|
| Hybrid rules + RAG + LLM audit pipeline | ✅ |
Five ethical principles + structured AuditReport |
✅ |
| Home / Audit / Examples UI | ✅ |
| EDS dilemma pairing | ✅ |
| GPT-OSS 20B / 120B on Groq | ✅ |
| 36 offline tests + live API + LangSmith smoke | ✅ |
| GitHub Actions CI + Docker | ✅ |
| Architecture docs + CHANGELOG | ✅ |
| README screenshots (Audit, Examples, LangSmith) | ✅ |
├── app.py # Streamlit navigation entry
├── Dockerfile # Production / CI container image
├── docker-compose.yml # One-command local run
├── CHANGELOG.md # Version-by-version change history
├── pages/
│ ├── home.py # Landing + how it works + privacy
│ ├── audit.py # Run Audit (shared execute_audit path)
│ └── examples.py # Agent actions + EDS dilemmas
├── src/
│ ├── config.py # Secrets bootstrap (no key values exposed)
│ ├── auditor/ # Rules, evaluator, aggregator
│ ├── ethics/ # Principles, prompts, parser
│ ├── llm/ # Groq / Ollama / mock / resilient provider
│ ├── rag/ # Chroma store + retriever
│ ├── scenarios/ # EDS loader + audit converter
│ ├── models/ # Pydantic AuditReport types
│ ├── data/ # principles, examples, eds_scenarios JSON
│ └── ui/ # Theme, components, sidebar, privacy, audit_flow
├── data/chroma/ # Pre-built vector index (committed)
├── docs/
│ ├── architecture.md # Full system design reference
│ ├── screenshots/ # README images (audit, examples, langsmith-trace)
│ └── assets/ # icon, favicon, logo (light/dark)
├── scripts/
│ └── build_chroma_index.py # Rebuild Chroma from source JSON
├── tests/ # 36 offline + API + LangSmith
├── .github/workflows/
│ └── tests.yml # CI — offline, Docker, API, LangSmith
├── .streamlit/
│ ├── config.toml # Theme (amber accent, teal sidebar)
│ └── secrets.toml.example # GROQ + optional LangSmith template
├── requirements.txt
├── pytest.ini
└── LICENSE # MIT
MIT License — see LICENSE.
Open to feedback, suggestions, and mission-aligned collaboration on responsible AI tooling.
| Area | Direction |
|---|---|
| Integration | REST/webhook hook for external agent gateways |
| Customization | User-uploaded / organization-specific principle sets |
| Observability | Optional shareable audit-trace links (where LangSmith supports) |
| Content | Additional agent-action scenario packs (enterprise, healthcare, platform) |
| UX | Side-by-side audit of two proposed actions |