Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trulens-auto #1388

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

trulens-auto #1388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0936c2a
first
sfc-gh-pmardziel Aug 26, 2024
050346c
Merge remote-tracking branch 'origin/main' into piotrm/trulens-auto
sfc-gh-pmardziel Aug 26, 2024
6f61e15
adjust contents of auto modules and refine example notebook
sfc-gh-pmardziel Aug 27, 2024
d5a430d
note
sfc-gh-pmardziel Aug 27, 2024
27b5289
more info in example notebook
sfc-gh-pmardziel Aug 27, 2024
e957907
remove unneeded
sfc-gh-pmardziel Aug 27, 2024
f8f8295
format
sfc-gh-pmardziel Aug 27, 2024
16cbc03
get doc from module docstring
sfc-gh-pmardziel Aug 27, 2024
41ede8d
notes from feedback chat
sfc-gh-pmardziel Aug 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
365 changes: 365 additions & 0 deletions examples/experimental/auto_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,365 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Auto\n",
"\n",
"Demonstrations of the `trulens-auto` package features."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys\n",
"\n",
"# Add base dir to path to be able to access test folder.\n",
"base_dir = Path().cwd().parent.parent.resolve()\n",
"if str(base_dir) not in sys.path:\n",
" print(f\"Adding {base_dir} to sys.path\")\n",
" sys.path.append(str(base_dir))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Main import\n",
"\n",
"The `auto` module contains everything to get started with __TruLens__. and requires installation of only one package: `trulens-auto`. It can be displayed in a notebook to list its contents."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ! pip install trulens-auto\n",
"\n",
"from trulens import auto\n",
"\n",
"auto"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Auto installation\n",
"\n",
"Requests for features that require another optional `trulens-*` to be installed will install them automatically."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ! pip uninstall -y trulens-providers-litellm\n",
"from trulens.auto import LiteLLM\n",
"\n",
"LiteLLM"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Error handling\n",
"\n",
"Accessing `auto` attributes that do not correspond to any member will also produce the above index."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Submodules\n",
"\n",
"Submodules of `auto` feature a subsets of `auto` content with a few additions each to delve deeper.\n",
"\n",
"- `instrument` contains recorders, enums for configuring recorders, and the interface classes that recorders inherit from.\n",
"\n",
"- `providers` contains providers.\n",
"\n",
"- `feedback` contains everything needed to define feedback functions or feedback function implementations.\n",
"\n",
"- `dashboard` contains dashboard functions.\n",
"\n",
"- `connectors` contains connectors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### Instruments\n",
"from trulens.auto import instrument\n",
"\n",
"instrument"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### Providers\n",
"from trulens.auto import providers\n",
"\n",
"providers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#\n",
"from trulens.auto import providers\n",
"\n",
"providers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### feedback-related\n",
"\n",
"from trulens.auto import feedback\n",
"\n",
"feedback"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### dashboard functions\n",
"\n",
"from trulens.auto import dashboard\n",
"\n",
"dashboard"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Minimal Example\n",
"\n",
"Minimal usage requires only the import of the recorder class."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from trulens.auto import TruCustomApp\n",
"\n",
"from examples.dev.dummy_app.app import DummyApp\n",
"\n",
"# Create custom app:\n",
"ca = DummyApp()\n",
"\n",
"# Create trulens wrapper:\n",
"ta = TruCustomApp(ca)\n",
"\n",
"with ta as recorder:\n",
" res = ca.respond_to_query(\"hello there\")\n",
"res"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Minimal Example Option 1 (Implementation TODO)\n",
"\n",
"- Add dashboard methods to Apps so we don't have to import session."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from trulens.auto import TruCustomApp\n",
"\n",
"from examples.dev.dummy_app.app import DummyApp\n",
"\n",
"# Create custom app:\n",
"ca = DummyApp()\n",
"\n",
"# Create trulens wrapper:\n",
"ta = TruCustomApp(ca)\n",
"\n",
"with ta as recorder:\n",
" res = ca.respond_to_query(\"hello there\")\n",
"\n",
"ta.run_dashboard() # Currently not included"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Minimal Example Option 2 (Implementation TODO)\n",
"\n",
"- Define a default session `trulens.auto.session` __instance__ so it doesn't need to be initialized."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from trulens.auto import session # \"session\" is a default session instance\n",
"\n",
"from examples.dev.dummy_app.app import DummyApp\n",
"\n",
"# Create custom app:\n",
"ca = DummyApp()\n",
"\n",
"# Create trulens wrapper:\n",
"ta = session.App(ca)\n",
"\n",
"with ta as recorder:\n",
" res = ca.respond_to_query(\"hello there\")\n",
"\n",
"session.run_dashboard() # Currently not included"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Minimal Example Option 3 (Implementation TODO)\n",
"\n",
"- Initialize session but use that to create App, either by type or automatic type guess."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from trulens.auto import TruSession\n",
"\n",
"from examples.dev.dummy_app.app import DummyApp\n",
"\n",
"session = TruSession()\n",
"\n",
"# Create custom app:\n",
"ca = DummyApp()\n",
"\n",
"# Create trulens wrapper:\n",
"ta = session.Custom(ca) # or session.App(ca) # last option preferable\n",
"\n",
"with ta as recorder:\n",
" res = ca.respond_to_query(\"hello there\")\n",
"\n",
"session.run_dashboard()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Current dashboard\n",
"\n",
"- Requires importing another thing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# To add dashboard:\n",
"from trulens.auto import run_dashboard\n",
"\n",
"run_dashboard()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# To add feedback:\n",
"from trulens.auto import Feedback\n",
"from trulens.auto import HuggingfaceDummy\n",
"\n",
"# Create provider\n",
"provider = HuggingfaceDummy()\n",
"\n",
"# Create feedback functions.\n",
"lang_match = Feedback(provider.language_match).on_input_output()\n",
"sentiment = Feedback(provider.positive_sentiment).on_output()\n",
"\n",
"# Create custom app:\n",
"ca = DummyApp()\n",
"\n",
"# Create trulens wrapper:\n",
"ta = TruCustomApp(\n",
" ca,\n",
" app_name=\"customapp\",\n",
" feedbacks=[lang_match, sentiment],\n",
")\n",
"\n",
"# Invoke app\n",
"with ta as recorder:\n",
" res = ca.respond_to_query(\"hello there\")\n",
"res\n",
"\n",
"# Get feedback results; note that dummy providers return nonsensical feedback.\n",
"for fdef, fres in recorder.get().feedback_and_future_results:\n",
" print(fdef.name, fres.result().result)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "trulens-Hw0Dq8Lu-py3.11",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading
Loading