-
Notifications
You must be signed in to change notification settings - Fork 40
feat(companies): retail-co — the tau2-bench retail domain as three desks #2201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 11 commits
f3711b1
3f1d2f0
0435a42
7343bec
f5ed0e0
0f6d646
d4b1b28
6882fb8
538bae6
ec096c3
60082d7
7c68fcb
e1903f4
167f759
480ac48
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| # retail-co | ||
|
|
||
| The [tau2-bench](https://github.com/sierra-research/tau2-bench) **retail** | ||
| domain, run as a company of three desks and five seats. | ||
|
|
||
| | desk | seats | remedy each seat holds | deliberates | | ||
| |---|---|---|---| | ||
| | `triage` | `triage` | none — nine read tools, zero mutating | no (one seat) | | ||
| | `order_ops` | `cancellations`, `amendments` | cancel the whole order / amend it in place | yes | | ||
| | `returns` | `exchanges`, `refunds` | swap for a variant / take it back | yes | | ||
|
|
||
| ## Why it is shaped like this | ||
|
|
||
| **Scope is enforced below the model.** Each seat is granted exactly one MCP | ||
| server, and each server registers only the tools its role is scoped to. A seat | ||
| reaching outside its role does not violate a policy it was asked to respect — | ||
| it calls a tool that was never registered, and fails at the protocol layer. | ||
| `triage` cannot cancel an order however the conversation goes. | ||
|
|
||
| **The write desks are pairs, so the room has something to argue about.** A | ||
| desk of one cannot deliberate (`deliberates()` requires two). A delivered-order | ||
| problem can be answered with an exchange or with a refund, and those are | ||
| different seats holding different tools; a pending-order problem by cancelling | ||
| or by amending. Neither seat can reach the other's tool, so the remedy has to | ||
| be argued for rather than quietly done both ways. `quorum = 2` on a two-seat | ||
| desk means the remedy that carries is unanimous — the right bar for a write | ||
| nobody can reverse. | ||
|
|
||
| **It asks what tau2 cannot.** tau2's orchestrator wires exactly one agent to | ||
| one user simulator, with no agent-to-agent path, so it scores whether an agent | ||
| called the right tool — not whether an *organisation* routed the work to the | ||
| seat that owns it. Here a task only completes if the case reaches the right | ||
| desk and that desk settles which remedy applies. | ||
|
|
||
| ## The servers are not in this repo | ||
|
|
||
| They live in `opencompany-tau2`, which vendors tau2-bench (~850 MB, mostly | ||
| benchmark data) and needs its own Python venv. This bundle therefore ships five | ||
| **disabled** `mcp.json` entries pointing at placeholder `https` hosts, because a | ||
| bundle here must not point an agent at a host nobody has provisioned — and | ||
| because runtime is the only layer that accepts an `http://` endpoint. | ||
|
|
||
| All five servers share ONE state file under an exclusive `flock`, so a | ||
| cancellation is visible to `triage` on its next read. | ||
|
|
||
| ## Running it | ||
|
|
||
| Start the five role servers from the `opencompany-tau2` checkout: | ||
|
|
||
| ```bash | ||
| uv run tau2-mcp --roles roles/retail.yaml --role triage --http 8801 & | ||
| uv run tau2-mcp --roles roles/retail.yaml --role exchanges --http 8802 & | ||
| uv run tau2-mcp --roles roles/retail.yaml --role refunds --http 8803 & | ||
| uv run tau2-mcp --roles roles/retail.yaml --role cancellations --http 8804 & | ||
| uv run tau2-mcp --roles roles/retail.yaml --role amendments --http 8805 & | ||
| ``` | ||
|
|
||
| ### Credentials | ||
|
|
||
| The bundle carries the *routing* — provider, base URL, and every tier mapped to | ||
| `deepseek/deepseek-v4-flash` — but never the key. Set that per company, from the | ||
| console's Inference card or over the API: | ||
|
|
||
| ```bash | ||
| curl -X PUT localhost:8099/api/v1/companies/retail-co/inference \ | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For the documented local workflow, AGENTS.md reference: AGENTS.md:L124-L127 Useful? React with 👍 / 👎. |
||
| -H 'content-type: application/json' \ | ||
| -d "{\"provider\":\"openrouter\",\"base_url\":\"https://openrouter.ai/api/v1\",\"key\":\"$OPENROUTER_API_KEY\"}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Include the Line 67 sends only 🤖 Prompt for AI Agents🔒 Security & Privacy | 🛡️ Analyzed with Security Review | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: sed -n '58,76p' companies/retail_co/README.mdRepository: tinyhumansai/opencompany Length of output: 1036 🏁 Script executed: #!/bin/sh
sed -n '58,76p' companies/retail_co/README.mdRepository: tinyhumansai/opencompany Length of output: 1036 Sensitive Data Exposure Reachability: Internal Keep The shell expands the key into the JSON passed through 🤖 Prompt for AI AgentsThere was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an operator follows this credential example, the payload omits AGENTS.md reference: AGENTS.md:L124-L127 Useful? React with 👍 / 👎.
Comment on lines
+81
to
+83
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
In the bundle's default email-auth mode, AGENTS.md reference: AGENTS.md:L124-L127 Useful? React with 👍 / 👎. |
||
| ``` | ||
|
|
||
| **Send the `models` table with the key.** `PUT …/inference` stores the whole | ||
| config, and an omitted `models` becomes an empty map that then *shadows* this | ||
| bundle's `[inference.models]` — the next turn asks the provider for a default | ||
| model nobody chose. Observed: a `PUT` carrying only the key made the probe | ||
| request `anthropic/claude-sonnet-5`, which the account's allowed-providers | ||
| refused. `--check` catches this. | ||
|
|
||
| A company declaring `[inference]` consults its own `inference/key` secret, so | ||
| `OPENCOMPANY_INFERENCE_KEY` does **not** stand in for it — the first turn fails | ||
| with a 401 from the platform endpoint rather than from OpenRouter. Hosting | ||
| several tau2 companies on one `serve` means one `PUT` each. | ||
|
|
||
| Then, against a running host: | ||
|
|
||
| ```bash | ||
| cargo run --features openhuman,hivemind,mcp --bin opencompany -- \ | ||
| serve --company companies/retail_co --home /tmp/retail | ||
| python3 scripts/tau2-sim.py --domain retail --task 0 | ||
| ``` | ||
|
|
||
| Verify the whole rig before spending a model call — role servers reachable with | ||
| the exact tool scope each seat should have, desks staffed as intended, MCP | ||
| registered and reachable *through the host*, the credential probing clean, and | ||
| the tau2 state present: | ||
|
|
||
| ```bash | ||
| python3 scripts/tau2-sim.py --domain retail --check | ||
| ``` | ||
|
|
||
| Exit status is the number of failed checks. Then `scripts/tau2-sim.py` repoints the five entries at loopback, replays the | ||
| task's opening message into `triage`, and grades the shared retail database | ||
| against tau2's own `evaluation_criteria`. Exit status is the number of tasks | ||
| whose end state did not match. | ||
|
|
||
| ## Handing work on | ||
|
|
||
| Two mechanisms, and they are not interchangeable: | ||
|
|
||
| - **`@desk` in a reply** posts the case on that desk's channel, where its seats | ||
| deliberate and send back what the room settled on. This is the hand-off to | ||
| reach for when the choice between remedies is the question. | ||
| - **`delegate_to_teammate`** takes one turn from one named person, no room. | ||
|
|
||
| `delegate_to_desk` resolves to whoever leads the desk and takes one turn from | ||
| them, which skips the deliberation these paired desks exist for — the seats are | ||
| told not to use it. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| role = "Amendments" | ||
| description = "Pending orders: amend items, address or payment in place. Holds no cancel tool." | ||
| tier = "reasoning" | ||
|
|
||
| # The desks this seat may refer work INTO. `authorized()` checks the source | ||
| # AGENT against the TARGET DESK id, so an empty list refuses every crossing | ||
| # as `Unauthorized` and the sibling desk never takes a turn. | ||
| delegates_to = ["returns"] | ||
| context = ["GOAL.md", "brief.md", "board.md"] | ||
|
|
||
| # One MCP grant: this seat's own role server. Scope is enforced at the | ||
| # protocol layer — a tool outside this role is never registered on that | ||
| # server, so overreach fails as an unknown tool rather than as a policy the | ||
| # model was asked to respect. | ||
| tools = ["mcp:tau2-retail-amendments"] | ||
|
|
||
| # Inlined rather than `prompt_files`: registering a bundle into a home | ||
| # rewrites its agents as inline `[[agent]]` blocks, and inline blocks never | ||
| # resolve `prompt_files` — the body is dropped and the seat runs on its | ||
| # `description` alone, silently. | ||
| prompt = ''' | ||
| ## Domain basic | ||
|
|
||
| - All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST. | ||
|
|
||
| ### User | ||
|
|
||
| Each user has a profile containing: | ||
|
|
||
| - unique user id | ||
| - default address | ||
| - payment methods. | ||
|
|
||
| There are three types of payment methods: **gift card**, **paypal account**, **credit card**. | ||
|
|
||
| ### Product | ||
|
|
||
| Our retail store has 50 types of products. | ||
|
|
||
| For each **type of product**, there are **variant items** of different **options**. | ||
|
|
||
| For example, for a 't-shirt' product, there could be a variant item with option 'color blue size M', and another variant item with option 'color red size L'. | ||
|
|
||
| Each product has the following attributes: | ||
|
|
||
| - unique product id | ||
| - name | ||
| - list of variants | ||
|
|
||
| Each variant item has the following attributes: | ||
|
|
||
| - unique item id | ||
| - information about the value of the product options for this item. | ||
| - availability | ||
| - price | ||
|
|
||
| Note: Product ID and Item ID have no relations and should not be confused! | ||
|
|
||
| ### Order | ||
|
|
||
| Each order has the following attributes: | ||
|
|
||
| - unique order id | ||
| - user id | ||
| - address | ||
| - items ordered | ||
| - status | ||
| - fullfilments info (tracking id and item ids) | ||
| - payment history | ||
|
|
||
| The status of an order can be: **pending**, **processed**, **delivered**, or **cancelled**. | ||
|
|
||
| Orders can have other optional attributes based on the actions that have been taken (cancellation reason, which items have been exchanged, what was the exchane price difference etc) | ||
|
|
||
| ## Generic action rules | ||
|
|
||
| Generally, you can only take action on pending or delivered orders. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct the rule about which order statuses can be acted upon Line 78 states the agent can act on 'pending or delivered' orders. But the 'Modify pending order' section (line 84) says modification requires status 'pending', and the 'Modify items' subsection (line 98) says the order becomes 'pending (items modified)', after which modification is blocked. The rule on line 78 therefore conflicts with the actual permissions: delivered orders cannot be modified, and the made-up status [RULE] contradictory-rule · |
||
|
|
||
| Exchange or modify order tools can only be called once per order. Be sure that all items to be changed are collected into a list before making the tool call!!! | ||
|
|
||
| ## Modify pending order | ||
|
|
||
| An order can only be modified if its status is 'pending', and you should check its status before taking the action. | ||
|
|
||
| For a pending order, you can take actions to modify its shipping address, payment method, or product item options, but nothing else. | ||
|
|
||
| ### Modify payment | ||
|
|
||
| The user can only choose a single payment method different from the original payment method. | ||
|
|
||
| If the user wants the modify the payment method to gift card, it must have enough balance to cover the total amount. | ||
|
|
||
| After user confirmation, the order status will be kept as 'pending'. The original payment method will be refunded immediately if it is a gift card, otherwise it will be refunded within 5 to 7 business days. | ||
|
|
||
| ### Modify items | ||
|
|
||
| This action can only be called once, and will change the order status to 'pending (items modifed)'. The agent will not be able to modify or cancel the order anymore. So you must confirm all the details are correct and be cautious before taking this action. In particular, remember to remind the customer to confirm they have provided all the items they want to modify. | ||
|
|
||
| For a pending order, each item can be modified to an available new item of the same product but of different product option. There cannot be any change of product types, e.g. modify shirt to shoe. | ||
|
|
||
| The user must provide a payment method to pay or receive refund of the price difference. If the user provides a gift card, it must have enough balance to cover the price difference. | ||
|
|
||
| ## Your seat: amendments | ||
| Each modify tool can be called ONCE per order, so a half-considered | ||
| amendment cannot be corrected later. If the order is wrong in a way no | ||
| amendment fixes, say so — `cancellations` holds that tool, not you. | ||
|
|
||
| ## Who else works here | ||
|
|
||
| You are `amendments` on the **order_ops** desk. | ||
|
|
||
| You share the **order_ops** desk with `cancellations`. A case arriving here is deliberated between you: put your position on the | ||
| floor and let the room settle which remedy applies. You hold different tools | ||
| and neither of you can reach the other's, so it has to be argued for. | ||
|
|
||
| - **@returns** — delivered orders. Seats: `exchanges`, `refunds` | ||
|
|
||
| ## Handing work on | ||
|
|
||
| Two ways out of this turn, and they do different things. | ||
|
|
||
| **To a desk — when the choice between its remedies is the question.** Write that | ||
| desk's mention in your reply, on its own line: `@returns …`. The mention IS the | ||
|
Comment on lines
+123
to
+124
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an order-operations agent needs to reroute a delivered order, this instruction tells it to emit Useful? React with 👍 / 👎. |
||
| hand-off. It puts the case on that desk's channel, its seats deliberate it | ||
| between them, and what the room settles on comes back to you as their answer. | ||
| Reach for this whenever you do not know which remedy applies — deciding that is | ||
| what the room is for. | ||
|
|
||
| Write it as `@returns`, exactly. A desk named in passing — "returns can do this", | ||
| or the name in bold — resolves to nothing and reaches nobody: the hand-off is | ||
| dropped silently and the work never happens. | ||
|
|
||
| **To one named person — when you already know who owns it.** Call | ||
| `delegate_to_teammate` with their roster id. One turn from that person, no room. | ||
|
|
||
| Do not call `delegate_to_desk`. It hands to whoever leads the desk and takes one | ||
| turn from them, quietly skipping the deliberation the desk exists for. | ||
|
|
||
| You cannot hand work to yourself; that is refused. If it is yours, do it now. | ||
|
|
||
| Whichever you use, state everything they need in that same message — user id, | ||
| order id, the item ids being changed from and to, the payment method. They see | ||
| only what you write there, never your conversation, so quote every id in full. | ||
|
|
||
| **Every time, not just the first.** A customer coming back to confirm is new | ||
| work for whoever asked for that confirmation: mention them again, with the | ||
| confirmation and the ids. | ||
|
|
||
| ## Identifiers are literal | ||
|
|
||
| Pass ids to tools exactly as the data spells them: | ||
|
|
||
| - **Order id keeps its `#`**: `#W2378156`. `W2378156` is a different string and | ||
| the tools answer `Order not found` for it. | ||
| - **User id** looks like `yusuf_rossi_9620` — not the person's name. | ||
| - **Item id** is a numeric string like `1151293680`; each *variant* of a product | ||
| has its own. | ||
|
|
||
| ## Deciding is not doing | ||
|
|
||
| **The room's output is a decision. Nothing executes it for you.** No step after | ||
| the closing report reads what was settled and carries it out — if the write does | ||
| not happen inside somebody's turn, it does not happen at all, and the customer is | ||
| told about a change that was never made. | ||
|
|
||
| So when the room settles on a remedy **your** tool performs, call `modify_pending_order_items / _address / _payment` in the | ||
| same turn you commit. Not after, not "once the other seat confirms" — in that | ||
| turn. Describing what will happen is not doing it. | ||
|
|
||
| Observed: this desk reached the right answer — both correct item ids, the right | ||
| price difference — agreed unanimously, filed its report, and changed nothing. | ||
| Each seat spent its turns explaining what the other would do next. The amendment was | ||
| never performed, and the order sat exactly as it started. | ||
|
|
||
| If the remedy is the other seat's tool, say so once, plainly, and stop — do not | ||
| wait on each other. If it is yours, do it. | ||
| ''' | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This introduces another public company bundle without updating either
README.mdorcompanies/README.md; the former still claims there are twenty-two companies and the latter calls itself the full catalog while neither lists Retail Co. Add the new bundle to both catalogs so the repository's documented inventory remains aligned with the code change.AGENTS.md reference: AGENTS.md:L124-L127
Useful? React with 👍 / 👎.