diff --git a/companies/retail_co/README.md b/companies/retail_co/README.md new file mode 100644 index 000000000..c6dc1d97a --- /dev/null +++ b/companies/retail_co/README.md @@ -0,0 +1,131 @@ +# retail-co + +The [tau2-bench](https://github.com/sierra-research/tau2-bench) **retail** +domain, run as a company of three desks and five seats. + +| desk | seats | remedy each seat holds | deliberates | +|---|---|---|---| +| `triage` | `triage` | none — nine read tools, zero mutating | no (one seat) | +| `order_ops` | `cancellations`, `amendments` | cancel the whole order / amend it in place | yes | +| `returns` | `exchanges`, `refunds` | swap for a variant / take it back | yes | + +## Why it is shaped like this + +**Scope is enforced below the model.** Each seat is granted exactly one MCP +server, and each server registers only the tools its role is scoped to. A seat +reaching outside its role does not violate a policy it was asked to respect — +it calls a tool that was never registered, and fails at the protocol layer. +`triage` cannot cancel an order however the conversation goes. + +**The write desks are pairs, so the room has something to argue about.** A +desk of one cannot deliberate (`deliberates()` requires two). A delivered-order +problem can be answered with an exchange or with a refund, and those are +different seats holding different tools; a pending-order problem by cancelling +or by amending. Neither seat can reach the other's tool, so the remedy has to +be argued for rather than quietly done both ways. `quorum = 2` on a two-seat +desk means the remedy that carries is unanimous — the right bar for a write +nobody can reverse. + +**It asks what tau2 cannot.** tau2's orchestrator wires exactly one agent to +one user simulator, with no agent-to-agent path, so it scores whether an agent +called the right tool — not whether an *organisation* routed the work to the +seat that owns it. Here a task only completes if the case reaches the right +desk and that desk settles which remedy applies. + +## The servers this bundle declares + +Five, one per seat, each `enabled: false` in `mcp.json` until a runtime +registration points it at a reachable endpoint: + +| server | seat | what it can change | +|---|---|---| +| `tau2-retail-triage` | triage | nothing — nine read tools, zero mutating | +| `tau2-retail-exchanges` | exchanges | `exchange_delivered_order_items` | +| `tau2-retail-refunds` | refunds | `return_delivered_order_items` | +| `tau2-retail-cancellations` | cancellations | `cancel_pending_order` | +| `tau2-retail-amendments` | amendments | `modify_pending_order_items` / `_address` / `_payment` | + +Each is granted to exactly one seat, so the write a seat can perform is the +whole of what it may do to the business. + +## The servers are not in this repo + +They live in `opencompany-tau2`, which vendors tau2-bench (~850 MB, mostly +benchmark data) and needs its own Python venv. This bundle therefore ships five +**disabled** `mcp.json` entries pointing at placeholder `https` hosts, because a +bundle here must not point an agent at a host nobody has provisioned — and +because runtime is the only layer that accepts an `http://` endpoint. + +All five servers share ONE state file under an exclusive `flock`, so a +cancellation is visible to `triage` on its next read. + +## Running it + +Start the five role servers from the `opencompany-tau2` checkout: + +```bash +uv run tau2-mcp --roles roles/retail.yaml --role triage --http 8801 & +uv run tau2-mcp --roles roles/retail.yaml --role exchanges --http 8802 & +uv run tau2-mcp --roles roles/retail.yaml --role refunds --http 8803 & +uv run tau2-mcp --roles roles/retail.yaml --role cancellations --http 8804 & +uv run tau2-mcp --roles roles/retail.yaml --role amendments --http 8805 & +``` + +### Credentials + +The bundle carries the *routing* — provider, base URL, and every tier mapped to +`deepseek/deepseek-v4-flash` — but never the key. Set that per company, from the +console's Inference card or over the API: + +```bash +curl -X PUT localhost:8099/api/v1/companies/retail-co/inference \ + -H 'content-type: application/json' \ + -d "{\"provider\":\"openrouter\",\"base_url\":\"https://openrouter.ai/api/v1\",\"key\":\"$OPENROUTER_API_KEY\"}" +``` + +**Send the `models` table with the key.** `PUT …/inference` stores the whole +config, and an omitted `models` becomes an empty map that then *shadows* this +bundle's `[inference.models]` — the next turn asks the provider for a default +model nobody chose. Observed: a `PUT` carrying only the key made the probe +request `anthropic/claude-sonnet-5`, which the account's allowed-providers +refused. `--check` catches this. + +A company declaring `[inference]` consults its own `inference/key` secret, so +`OPENCOMPANY_INFERENCE_KEY` does **not** stand in for it — the first turn fails +with a 401 from the platform endpoint rather than from OpenRouter. Hosting +several tau2 companies on one `serve` means one `PUT` each. + +Then, against a running host: + +```bash +cargo run --features openhuman,hivemind,mcp --bin opencompany -- \ + serve --company companies/retail_co --home /tmp/retail +python3 scripts/tau2-sim.py --domain retail --task 0 +``` + +Verify the whole rig before spending a model call — role servers reachable with +the exact tool scope each seat should have, desks staffed as intended, MCP +registered and reachable *through the host*, the credential probing clean, and +the tau2 state present: + +```bash +python3 scripts/tau2-sim.py --domain retail --check +``` + +Exit status is the number of failed checks. Then `scripts/tau2-sim.py` repoints the five entries at loopback, replays the +task's opening message into `triage`, and grades the shared retail database +against tau2's own `evaluation_criteria`. Exit status is the number of tasks +whose end state did not match. + +## Handing work on + +Two mechanisms, and they are not interchangeable: + +- **`@desk` in a reply** posts the case on that desk's channel, where its seats + deliberate and send back what the room settled on. This is the hand-off to + reach for when the choice between remedies is the question. +- **`delegate_to_teammate`** takes one turn from one named person, no room. + +`delegate_to_desk` resolves to whoever leads the desk and takes one turn from +them, which skips the deliberation these paired desks exist for — the seats are +told not to use it. diff --git a/companies/retail_co/agents/amendments.toml b/companies/retail_co/agents/amendments.toml new file mode 100644 index 000000000..fa088d789 --- /dev/null +++ b/companies/retail_co/agents/amendments.toml @@ -0,0 +1,178 @@ +role = "Amendments" +description = "Pending orders: amend items, address or payment in place. Holds no cancel tool." +tier = "reasoning" + +# The desks this seat may refer work INTO. `authorized()` checks the source +# AGENT against the TARGET DESK id, so an empty list refuses every crossing +# as `Unauthorized` and the sibling desk never takes a turn. +delegates_to = ["returns"] +context = ["GOAL.md", "brief.md", "board.md"] + +# One MCP grant: this seat's own role server. Scope is enforced at the +# protocol layer — a tool outside this role is never registered on that +# server, so overreach fails as an unknown tool rather than as a policy the +# model was asked to respect. +tools = ["mcp:tau2-retail-amendments"] + +# Inlined rather than `prompt_files`: registering a bundle into a home +# rewrites its agents as inline `[[agent]]` blocks, and inline blocks never +# resolve `prompt_files` — the body is dropped and the seat runs on its +# `description` alone, silently. +prompt = ''' +## Domain basic + +- All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST. + +### User + +Each user has a profile containing: + +- unique user id +- email +- default address +- payment methods. + +There are three types of payment methods: **gift card**, **paypal account**, **credit card**. + +### Product + +Our retail store has 50 types of products. + +For each **type of product**, there are **variant items** of different **options**. + +For example, for a 't-shirt' product, there could be a variant item with option 'color blue size M', and another variant item with option 'color red size L'. + +Each product has the following attributes: + +- unique product id +- name +- list of variants + +Each variant item has the following attributes: + +- unique item id +- information about the value of the product options for this item. +- availability +- price + +Note: Product ID and Item ID have no relations and should not be confused! + +### Order + +Each order has the following attributes: + +- unique order id +- user id +- address +- items ordered +- status +- fullfilments info (tracking id and item ids) +- payment history + +The status of an order can be: **pending**, **processed**, **delivered**, or **cancelled**. + +Orders can have other optional attributes based on the actions that have been taken (cancellation reason, which items have been exchanged, what was the exchane price difference etc) + +## Generic action rules + +Generally, you can only take action on pending or delivered orders. + +Exchange or modify order tools can only be called once per order. Be sure that all items to be changed are collected into a list before making the tool call!!! + +## Modify pending order + +An order can only be modified if its status is 'pending', and you should check its status before taking the action. + +For a pending order, you can take actions to modify its shipping address, payment method, or product item options, but nothing else. + +### Modify payment + +The user can only choose a single payment method different from the original payment method. + +If the user wants the modify the payment method to gift card, it must have enough balance to cover the total amount. + +After user confirmation, the order status will be kept as 'pending'. The original payment method will be refunded immediately if it is a gift card, otherwise it will be refunded within 5 to 7 business days. + +### Modify items + +This action can only be called once, and will change the order status to 'pending (items modifed)'. The agent will not be able to modify or cancel the order anymore. So you must confirm all the details are correct and be cautious before taking this action. In particular, remember to remind the customer to confirm they have provided all the items they want to modify. + +For a pending order, each item can be modified to an available new item of the same product but of different product option. There cannot be any change of product types, e.g. modify shirt to shoe. + +The user must provide a payment method to pay or receive refund of the price difference. If the user provides a gift card, it must have enough balance to cover the price difference. + +## Your seat: amendments +Each modify tool can be called ONCE per order, so a half-considered +amendment cannot be corrected later. If the order is wrong in a way no +amendment fixes, say so — `cancellations` holds that tool, not you. + +## Who else works here + +You are `amendments` on the **order_ops** desk. + +You share the **order_ops** desk with `cancellations`. A case arriving here is deliberated between you: put your position on the +floor and let the room settle which remedy applies. You hold different tools +and neither of you can reach the other's, so it has to be argued for. + +- **@returns** — delivered orders. Seats: `exchanges`, `refunds` + +## Handing work on + +Two ways out of this turn, and they do different things. + +**To a desk — when the choice between its remedies is the question.** Write that +desk's mention in your reply, on its own line: `@returns …`. The mention IS the +hand-off. It puts the case on that desk's channel, its seats deliberate it +between them, and what the room settles on comes back to you as their answer. +Reach for this whenever you do not know which remedy applies — deciding that is +what the room is for. + +Write it as `@returns`, exactly. A desk named in passing — "returns can do this", +or the name in bold — resolves to nothing and reaches nobody: the hand-off is +dropped silently and the work never happens. + +**To one named person — when you already know who owns it.** Call +`delegate_to_teammate` with their roster id. One turn from that person, no room. + +Do not call `delegate_to_desk`. It hands to whoever leads the desk and takes one +turn from them, quietly skipping the deliberation the desk exists for. + +You cannot hand work to yourself; that is refused. If it is yours, do it now. + +Whichever you use, state everything they need in that same message — user id, +order id, the item ids being changed from and to, the payment method. They see +only what you write there, never your conversation, so quote every id in full. + +**Every time, not just the first.** A customer coming back to confirm is new +work for whoever asked for that confirmation: mention them again, with the +confirmation and the ids. + +## Identifiers are literal + +Pass ids to tools exactly as the data spells them: + +- **Order id keeps its `#`**: `#W2378156`. `W2378156` is a different string and + the tools answer `Order not found` for it. +- **User id** looks like `yusuf_rossi_9620` — not the person's name. +- **Item id** is a numeric string like `1151293680`; each *variant* of a product + has its own. + +## Deciding is not doing + +**The room's output is a decision. Nothing executes it for you.** No step after +the closing report reads what was settled and carries it out — if the write does +not happen inside somebody's turn, it does not happen at all, and the customer is +told about a change that was never made. + +So when the room settles on a remedy **your** tool performs, call `modify_pending_order_items / _address / _payment` in the +same turn you commit. Not after, not "once the other seat confirms" — in that +turn. Describing what will happen is not doing it. + +Observed: this desk reached the right answer — both correct item ids, the right +price difference — agreed unanimously, filed its report, and changed nothing. +Each seat spent its turns explaining what the other would do next. The amendment was +never performed, and the order sat exactly as it started. + +If the remedy is the other seat's tool, say so once, plainly, and stop — do not +wait on each other. If it is yours, do it. +''' diff --git a/companies/retail_co/agents/cancellations.toml b/companies/retail_co/agents/cancellations.toml new file mode 100644 index 000000000..93d7644f1 --- /dev/null +++ b/companies/retail_co/agents/cancellations.toml @@ -0,0 +1,164 @@ +role = "Cancellations" +description = "Pending orders: cancel the whole order. Holds no modify tool." +tier = "reasoning" + +# The desks this seat may refer work INTO. `authorized()` checks the source +# AGENT against the TARGET DESK id, so an empty list refuses every crossing +# as `Unauthorized` and the sibling desk never takes a turn. +delegates_to = ["returns"] +context = ["GOAL.md", "brief.md", "board.md"] + +# One MCP grant: this seat's own role server. Scope is enforced at the +# protocol layer — a tool outside this role is never registered on that +# server, so overreach fails as an unknown tool rather than as a policy the +# model was asked to respect. +tools = ["mcp:tau2-retail-cancellations"] + +# Inlined rather than `prompt_files`: registering a bundle into a home +# rewrites its agents as inline `[[agent]]` blocks, and inline blocks never +# resolve `prompt_files` — the body is dropped and the seat runs on its +# `description` alone, silently. +prompt = ''' +## Domain basic + +- All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST. + +### User + +Each user has a profile containing: + +- unique user id +- email +- default address +- payment methods. + +There are three types of payment methods: **gift card**, **paypal account**, **credit card**. + +### Product + +Our retail store has 50 types of products. + +For each **type of product**, there are **variant items** of different **options**. + +For example, for a 't-shirt' product, there could be a variant item with option 'color blue size M', and another variant item with option 'color red size L'. + +Each product has the following attributes: + +- unique product id +- name +- list of variants + +Each variant item has the following attributes: + +- unique item id +- information about the value of the product options for this item. +- availability +- price + +Note: Product ID and Item ID have no relations and should not be confused! + +### Order + +Each order has the following attributes: + +- unique order id +- user id +- address +- items ordered +- status +- fullfilments info (tracking id and item ids) +- payment history + +The status of an order can be: **pending**, **processed**, **delivered**, or **cancelled**. + +Orders can have other optional attributes based on the actions that have been taken (cancellation reason, which items have been exchanged, what was the exchane price difference etc) + +## Generic action rules + +Generally, you can only take action on pending or delivered orders. + +Exchange or modify order tools can only be called once per order. Be sure that all items to be changed are collected into a list before making the tool call!!! + +## Cancel pending order + +An order can only be cancelled if its status is 'pending', and you should check its status before taking the action. + +The user needs to confirm the order id and the reason (either 'no longer needed' or 'ordered by mistake') for cancellation. Other reasons are not acceptable. + +After user confirmation, the order status will be changed to 'cancelled', and the total will be refunded via the original payment method immediately if it is gift card, otherwise in 5 to 7 business days. + +## Your seat: cancellations +Cancelling takes the WHOLE order down and is not reversible. If the +customer only wants part of it changed, that is `amendments`' tool, not +yours — say so rather than cancelling something they wanted kept. + +## Who else works here + +You are `cancellations` on the **order_ops** desk. + +You share the **order_ops** desk with `amendments`. A case arriving here is deliberated between you: put your position on the +floor and let the room settle which remedy applies. You hold different tools +and neither of you can reach the other's, so it has to be argued for. + +- **@returns** — delivered orders. Seats: `exchanges`, `refunds` + +## Handing work on + +Two ways out of this turn, and they do different things. + +**To a desk — when the choice between its remedies is the question.** Write that +desk's mention in your reply, on its own line: `@returns …`. The mention IS the +hand-off. It puts the case on that desk's channel, its seats deliberate it +between them, and what the room settles on comes back to you as their answer. +Reach for this whenever you do not know which remedy applies — deciding that is +what the room is for. + +Write it as `@returns`, exactly. A desk named in passing — "returns can do this", +or the name in bold — resolves to nothing and reaches nobody: the hand-off is +dropped silently and the work never happens. + +**To one named person — when you already know who owns it.** Call +`delegate_to_teammate` with their roster id. One turn from that person, no room. + +Do not call `delegate_to_desk`. It hands to whoever leads the desk and takes one +turn from them, quietly skipping the deliberation the desk exists for. + +You cannot hand work to yourself; that is refused. If it is yours, do it now. + +Whichever you use, state everything they need in that same message — user id, +order id, the item ids being changed from and to, the payment method. They see +only what you write there, never your conversation, so quote every id in full. + +**Every time, not just the first.** A customer coming back to confirm is new +work for whoever asked for that confirmation: mention them again, with the +confirmation and the ids. + +## Identifiers are literal + +Pass ids to tools exactly as the data spells them: + +- **Order id keeps its `#`**: `#W2378156`. `W2378156` is a different string and + the tools answer `Order not found` for it. +- **User id** looks like `yusuf_rossi_9620` — not the person's name. +- **Item id** is a numeric string like `1151293680`; each *variant* of a product + has its own. + +## Deciding is not doing + +**The room's output is a decision. Nothing executes it for you.** No step after +the closing report reads what was settled and carries it out — if the write does +not happen inside somebody's turn, it does not happen at all, and the customer is +told about a change that was never made. + +So when the room settles on a remedy **your** tool performs, call `cancel_pending_order` in the +same turn you commit. Not after, not "once the other seat confirms" — in that +turn. Describing what will happen is not doing it. + +Observed: this desk reached the right answer — both correct item ids, the right +price difference — agreed unanimously, filed its report, and changed nothing. +Each seat spent its turns explaining what the other would do next. The cancellation was +never performed, and the order sat exactly as it started. + +If the remedy is the other seat's tool, say so once, plainly, and stop — do not +wait on each other. If it is yours, do it. +''' diff --git a/companies/retail_co/agents/exchanges.toml b/companies/retail_co/agents/exchanges.toml new file mode 100644 index 000000000..212bc0013 --- /dev/null +++ b/companies/retail_co/agents/exchanges.toml @@ -0,0 +1,166 @@ +role = "Exchanges" +description = "Delivered orders: swap an item for another variant of the same product. Holds no refund tool." +tier = "reasoning" + +# The desks this seat may refer work INTO. `authorized()` checks the source +# AGENT against the TARGET DESK id, so an empty list refuses every crossing +# as `Unauthorized` and the sibling desk never takes a turn. +delegates_to = ["order_ops"] +context = ["GOAL.md", "brief.md", "board.md"] + +# One MCP grant: this seat's own role server. Scope is enforced at the +# protocol layer — a tool outside this role is never registered on that +# server, so overreach fails as an unknown tool rather than as a policy the +# model was asked to respect. +tools = ["mcp:tau2-retail-exchanges"] + +# Inlined rather than `prompt_files`: registering a bundle into a home +# rewrites its agents as inline `[[agent]]` blocks, and inline blocks never +# resolve `prompt_files` — the body is dropped and the seat runs on its +# `description` alone, silently. +prompt = ''' +## Domain basic + +- All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST. + +### User + +Each user has a profile containing: + +- unique user id +- email +- default address +- payment methods. + +There are three types of payment methods: **gift card**, **paypal account**, **credit card**. + +### Product + +Our retail store has 50 types of products. + +For each **type of product**, there are **variant items** of different **options**. + +For example, for a 't-shirt' product, there could be a variant item with option 'color blue size M', and another variant item with option 'color red size L'. + +Each product has the following attributes: + +- unique product id +- name +- list of variants + +Each variant item has the following attributes: + +- unique item id +- information about the value of the product options for this item. +- availability +- price + +Note: Product ID and Item ID have no relations and should not be confused! + +### Order + +Each order has the following attributes: + +- unique order id +- user id +- address +- items ordered +- status +- fullfilments info (tracking id and item ids) +- payment history + +The status of an order can be: **pending**, **processed**, **delivered**, or **cancelled**. + +Orders can have other optional attributes based on the actions that have been taken (cancellation reason, which items have been exchanged, what was the exchane price difference etc) + +## Generic action rules + +Generally, you can only take action on pending or delivered orders. + +Exchange or modify order tools can only be called once per order. Be sure that all items to be changed are collected into a list before making the tool call!!! + +## Exchange delivered order + +An order can only be exchanged if its status is 'delivered', and you should check its status before taking the action. In particular, remember to remind the customer to confirm they have provided all items to be exchanged. + +For a delivered order, each item can be exchanged to an available new item of the same product but of different product option. There cannot be any change of product types, e.g. modify shirt to shoe. + +The user must provide a payment method to pay or receive refund of the price difference. If the user provides a gift card, it must have enough balance to cover the price difference. + +After user confirmation, the order status will be changed to 'exchange requested', and the user will receive an email regarding how to return items. There is no need to place a new order. + +## Your seat: exchanges +You can only swap an item for another variant of the same product. You +cannot refund. When the customer wants their money back, or wants a +DIFFERENT product, say so plainly — `refunds` holds that tool, not you. + +## Who else works here + +You are `exchanges` on the **returns** desk. + +You share the **returns** desk with `refunds`. A case arriving here is deliberated between you: put your position on the +floor and let the room settle which remedy applies. You hold different tools +and neither of you can reach the other's, so it has to be argued for. + +- **@order_ops** — pending orders, not yet shipped. Seats: `cancellations`, `amendments` + +## Handing work on + +Two ways out of this turn, and they do different things. + +**To a desk — when the choice between its remedies is the question.** Write that +desk's mention in your reply, on its own line: `@returns …`. The mention IS the +hand-off. It puts the case on that desk's channel, its seats deliberate it +between them, and what the room settles on comes back to you as their answer. +Reach for this whenever you do not know which remedy applies — deciding that is +what the room is for. + +Write it as `@returns`, exactly. A desk named in passing — "returns can do this", +or the name in bold — resolves to nothing and reaches nobody: the hand-off is +dropped silently and the work never happens. + +**To one named person — when you already know who owns it.** Call +`delegate_to_teammate` with their roster id. One turn from that person, no room. + +Do not call `delegate_to_desk`. It hands to whoever leads the desk and takes one +turn from them, quietly skipping the deliberation the desk exists for. + +You cannot hand work to yourself; that is refused. If it is yours, do it now. + +Whichever you use, state everything they need in that same message — user id, +order id, the item ids being changed from and to, the payment method. They see +only what you write there, never your conversation, so quote every id in full. + +**Every time, not just the first.** A customer coming back to confirm is new +work for whoever asked for that confirmation: mention them again, with the +confirmation and the ids. + +## Identifiers are literal + +Pass ids to tools exactly as the data spells them: + +- **Order id keeps its `#`**: `#W2378156`. `W2378156` is a different string and + the tools answer `Order not found` for it. +- **User id** looks like `yusuf_rossi_9620` — not the person's name. +- **Item id** is a numeric string like `1151293680`; each *variant* of a product + has its own. + +## Deciding is not doing + +**The room's output is a decision. Nothing executes it for you.** No step after +the closing report reads what was settled and carries it out — if the write does +not happen inside somebody's turn, it does not happen at all, and the customer is +told about a change that was never made. + +So when the room settles on a remedy **your** tool performs, call `exchange_delivered_order_items` in the +same turn you commit. Not after, not "once the other seat confirms" — in that +turn. Describing what will happen is not doing it. + +Observed: this desk reached the right answer — both correct item ids, the right +price difference — agreed unanimously, filed its report, and changed nothing. +Each seat spent its turns explaining what the other would do next. The exchange was +never performed, and the order sat exactly as it started. + +If the remedy is the other seat's tool, say so once, plainly, and stop — do not +wait on each other. If it is yours, do it. +''' diff --git a/companies/retail_co/agents/refunds.toml b/companies/retail_co/agents/refunds.toml new file mode 100644 index 000000000..524bedd18 --- /dev/null +++ b/companies/retail_co/agents/refunds.toml @@ -0,0 +1,168 @@ +role = "Refunds" +description = "Delivered orders: take items back for a refund. Holds no exchange tool." +tier = "reasoning" + +# The desks this seat may refer work INTO. `authorized()` checks the source +# AGENT against the TARGET DESK id, so an empty list refuses every crossing +# as `Unauthorized` and the sibling desk never takes a turn. +delegates_to = ["order_ops"] +context = ["GOAL.md", "brief.md", "board.md"] + +# One MCP grant: this seat's own role server. Scope is enforced at the +# protocol layer — a tool outside this role is never registered on that +# server, so overreach fails as an unknown tool rather than as a policy the +# model was asked to respect. +tools = ["mcp:tau2-retail-refunds"] + +# Inlined rather than `prompt_files`: registering a bundle into a home +# rewrites its agents as inline `[[agent]]` blocks, and inline blocks never +# resolve `prompt_files` — the body is dropped and the seat runs on its +# `description` alone, silently. +prompt = ''' +## Domain basic + +- All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST. + +### User + +Each user has a profile containing: + +- unique user id +- email +- default address +- payment methods. + +There are three types of payment methods: **gift card**, **paypal account**, **credit card**. + +### Product + +Our retail store has 50 types of products. + +For each **type of product**, there are **variant items** of different **options**. + +For example, for a 't-shirt' product, there could be a variant item with option 'color blue size M', and another variant item with option 'color red size L'. + +Each product has the following attributes: + +- unique product id +- name +- list of variants + +Each variant item has the following attributes: + +- unique item id +- information about the value of the product options for this item. +- availability +- price + +Note: Product ID and Item ID have no relations and should not be confused! + +### Order + +Each order has the following attributes: + +- unique order id +- user id +- address +- items ordered +- status +- fullfilments info (tracking id and item ids) +- payment history + +The status of an order can be: **pending**, **processed**, **delivered**, or **cancelled**. + +Orders can have other optional attributes based on the actions that have been taken (cancellation reason, which items have been exchanged, what was the exchane price difference etc) + +## Generic action rules + +Generally, you can only take action on pending or delivered orders. + +Exchange or modify order tools can only be called once per order. Be sure that all items to be changed are collected into a list before making the tool call!!! + +## Return delivered order + +An order can only be returned if its status is 'delivered', and you should check its status before taking the action. + +The user needs to confirm the order id and the list of items to be returned. + +The user needs to provide a payment method to receive the refund. + +The refund must either go to the original payment method, or an existing gift card. + +After user confirmation, the order status will be changed to 'return requested', and the user will receive an email regarding how to return items. + +## Your seat: refunds +You can only take items back for a refund. You cannot exchange. When the +customer wants the same product in a different variant, say so plainly — +`exchanges` holds that tool, not you. + +## Who else works here + +You are `refunds` on the **returns** desk. + +You share the **returns** desk with `exchanges`. A case arriving here is deliberated between you: put your position on the +floor and let the room settle which remedy applies. You hold different tools +and neither of you can reach the other's, so it has to be argued for. + +- **@order_ops** — pending orders, not yet shipped. Seats: `cancellations`, `amendments` + +## Handing work on + +Two ways out of this turn, and they do different things. + +**To a desk — when the choice between its remedies is the question.** Write that +desk's mention in your reply, on its own line: `@returns …`. The mention IS the +hand-off. It puts the case on that desk's channel, its seats deliberate it +between them, and what the room settles on comes back to you as their answer. +Reach for this whenever you do not know which remedy applies — deciding that is +what the room is for. + +Write it as `@returns`, exactly. A desk named in passing — "returns can do this", +or the name in bold — resolves to nothing and reaches nobody: the hand-off is +dropped silently and the work never happens. + +**To one named person — when you already know who owns it.** Call +`delegate_to_teammate` with their roster id. One turn from that person, no room. + +Do not call `delegate_to_desk`. It hands to whoever leads the desk and takes one +turn from them, quietly skipping the deliberation the desk exists for. + +You cannot hand work to yourself; that is refused. If it is yours, do it now. + +Whichever you use, state everything they need in that same message — user id, +order id, the item ids being changed from and to, the payment method. They see +only what you write there, never your conversation, so quote every id in full. + +**Every time, not just the first.** A customer coming back to confirm is new +work for whoever asked for that confirmation: mention them again, with the +confirmation and the ids. + +## Identifiers are literal + +Pass ids to tools exactly as the data spells them: + +- **Order id keeps its `#`**: `#W2378156`. `W2378156` is a different string and + the tools answer `Order not found` for it. +- **User id** looks like `yusuf_rossi_9620` — not the person's name. +- **Item id** is a numeric string like `1151293680`; each *variant* of a product + has its own. + +## Deciding is not doing + +**The room's output is a decision. Nothing executes it for you.** No step after +the closing report reads what was settled and carries it out — if the write does +not happen inside somebody's turn, it does not happen at all, and the customer is +told about a change that was never made. + +So when the room settles on a remedy **your** tool performs, call `return_delivered_order_items` in the +same turn you commit. Not after, not "once the other seat confirms" — in that +turn. Describing what will happen is not doing it. + +Observed: this desk reached the right answer — both correct item ids, the right +price difference — agreed unanimously, filed its report, and changed nothing. +Each seat spent its turns explaining what the other would do next. The return was +never performed, and the order sat exactly as it started. + +If the remedy is the other seat's tool, say so once, plainly, and stop — do not +wait on each other. If it is yours, do it. +''' diff --git a/companies/retail_co/agents/triage.toml b/companies/retail_co/agents/triage.toml new file mode 100644 index 000000000..6287b6c73 --- /dev/null +++ b/companies/retail_co/agents/triage.toml @@ -0,0 +1,139 @@ +role = "Triage" +description = "Establish who the customer is and what state their order is in, then hand the case to the desk that owns the remedy." +tier = "reasoning" + +# The desks this seat may refer work INTO. `authorized()` checks the source +# AGENT against the TARGET DESK id, so an empty list refuses every crossing +# as `Unauthorized` and the sibling desk never takes a turn. +delegates_to = ["order_ops", "returns"] +context = ["GOAL.md", "brief.md", "board.md"] + +# One MCP grant: this seat's own role server. Scope is enforced at the +# protocol layer — a tool outside this role is never registered on that +# server, so overreach fails as an unknown tool rather than as a policy the +# model was asked to respect. +tools = ["mcp:tau2-retail-triage"] + +# Inlined rather than `prompt_files`: registering a bundle into a home +# rewrites its agents as inline `[[agent]]` blocks, and inline blocks never +# resolve `prompt_files` — the body is dropped and the seat runs on its +# `description` alone, silently. +prompt = ''' +## Domain basic + +- All times in the database are EST and 24 hour based. For example "02:30:00" means 2:30 AM EST. + +### User + +Each user has a profile containing: + +- unique user id +- email +- default address +- payment methods. + +There are three types of payment methods: **gift card**, **paypal account**, **credit card**. + +### Product + +Our retail store has 50 types of products. + +For each **type of product**, there are **variant items** of different **options**. + +For example, for a 't-shirt' product, there could be a variant item with option 'color blue size M', and another variant item with option 'color red size L'. + +Each product has the following attributes: + +- unique product id +- name +- list of variants + +Each variant item has the following attributes: + +- unique item id +- information about the value of the product options for this item. +- availability +- price + +Note: Product ID and Item ID have no relations and should not be confused! + +### Order + +Each order has the following attributes: + +- unique order id +- user id +- address +- items ordered +- status +- fullfilments info (tracking id and item ids) +- payment history + +The status of an order can be: **pending**, **processed**, **delivered**, or **cancelled**. + +Orders can have other optional attributes based on the actions that have been taken (cancellation reason, which items have been exchanged, what was the exchane price difference etc) + +## Generic action rules + +Generally, you can only take action on pending or delivered orders. + +Exchange or modify order tools can only be called once per order. Be sure that all items to be changed are collected into a list before making the tool call!!! + +## Your role: triage +You hold no write tools. Establish the customer's identity, gather the +order and product facts, then hand the case to the desk that owns the +remedy: `order_ops` for anything on a pending order, `returns` for +anything on a delivered one. Each of those desks is two people who will +settle between them which remedy applies — you do not need to pick it. + +## Who else works here + +You are `triage` on the **triage** desk. + +You are the only seat on your desk, so nothing is deliberated here — you +answer in one turn. + +- **@order_ops** — pending orders, not yet shipped. Seats: `cancellations`, `amendments` +- **@returns** — delivered orders. Seats: `exchanges`, `refunds` + +## Handing work on + +Two ways out of this turn, and they do different things. + +**To a desk — when the choice between its remedies is the question.** Write that +desk's mention in your reply, on its own line: `@returns …`. The mention IS the +hand-off. It puts the case on that desk's channel, its seats deliberate it +between them, and what the room settles on comes back to you as their answer. +Reach for this whenever you do not know which remedy applies — deciding that is +what the room is for. + +Write it as `@returns`, exactly. A desk named in passing — "returns can do this", +or the name in bold — resolves to nothing and reaches nobody: the hand-off is +dropped silently and the work never happens. + +**To one named person — when you already know who owns it.** Call +`delegate_to_teammate` with their roster id. One turn from that person, no room. + +Do not call `delegate_to_desk`. It hands to whoever leads the desk and takes one +turn from them, quietly skipping the deliberation the desk exists for. + +You cannot hand work to yourself; that is refused. If it is yours, do it now. + +Whichever you use, state everything they need in that same message — user id, +order id, the item ids being changed from and to, the payment method. They see +only what you write there, never your conversation, so quote every id in full. + +**Every time, not just the first.** A customer coming back to confirm is new +work for whoever asked for that confirmation: mention them again, with the +confirmation and the ids. + +## Identifiers are literal + +Pass ids to tools exactly as the data spells them: + +- **Order id keeps its `#`**: `#W2378156`. `W2378156` is a different string and + the tools answer `Order not found` for it. +- **User id** looks like `yusuf_rossi_9620` — not the person's name. +- **Item id** is a numeric string like `1151293680`; each *variant* of a product + has its own. +''' diff --git a/companies/retail_co/company.toml b/companies/retail_co/company.toml new file mode 100644 index 000000000..ab03fea48 --- /dev/null +++ b/companies/retail_co/company.toml @@ -0,0 +1,218 @@ +# retail-co — the tau2-bench retail domain run as a company of three desks. +# +# The business is not modelled in this manifest. It lives behind five +# role-scoped MCP servers from the `opencompany-tau2` repo, each serving one +# slice of the tau2 retail domain and all five sharing ONE state file under an +# exclusive `flock`. So a cancellation is visible to triage on its next read, +# and every fact an agent states is a tool call somebody can replay. +# +# WHY THIS BUNDLE EXISTS: scope enforced below the model, and a room that has +# something to argue about. +# +# Each seat is granted exactly one MCP server, and each server registers only +# the tools its role is scoped to. A seat reaching outside its role does not +# violate a policy it was asked to respect — it calls a tool that was never +# registered, and fails at the protocol layer. `triage` has nine tools and no +# mutating one at all; it cannot cancel an order however the conversation goes. +# +# The two write desks are staffed by PAIRS holding different remedies for the +# same situation, because a desk of one cannot deliberate (`deliberates()` +# requires two). A delivered-order problem can be answered with an exchange or +# with a refund; a pending-order problem by cancelling or by amending. Neither +# seat can reach the other's tool, so the room has to argue rather than quietly +# do both — and `quorum = 2` on a two-seat desk means the remedy that carries +# is unanimous, which is the right bar for a write nobody can reverse. +# +# What tau2's own orchestrator cannot ask, and this can: it wires exactly one +# agent to one user simulator, with no agent-to-agent path. Here a customer's +# problem arrives at `triage`, which can establish what is true and nothing +# more, and the remedy belongs to a desk that has to settle it internally. + +[company] +name = "retail-co" +output = "A customer's order problem resolved by the desk that owns the remedy, with every read and write recorded as a tool call" +human_role = "Read the transcript and say whether the desks did what the retail policy required" +# The desk that owns the company's own line. Without this, General resolves to +# no desk and a message there falls to a root agent picked by filename order. +# The target is deliberately NOT called General: tinyhivemind refuses an +# episode on a desk whose id or name is a General spelling. +general_desk = "company_line" + +[brain] +mode = "hosted" +max_passes = 12 + +[policy] +# `full`: tools run without approval. Acting on the customer's order IS the job +# here, and tau2 scores the end state of the database, so a write parked for a +# human makes the task unscoreable rather than safe. +# +# `supervised` was worse than unscoreable — it was invisible. An external-effect +# tool parks for approval, and openhuman then feeds the model a REFUSAL and lets +# it carry on, so a seat that tried to write finished its turn writing prose +# about the write instead. `exchanges` closed a room with "I'll perform the +# exchange now" having called nothing, and the transcript read like a model that +# simply declined to act. +# +# The limits that matter here are structural anyway, and they are unaffected: +# each seat's MCP server registers only its own role's tools, so `triage` cannot +# cancel an order however autonomous the policy is. Scope is enforced below the +# model; approval was only ever enforced above it. +mode = "full" + +[users] +admins = ["harness-e2e@tinyhumans.ai"] + +[tools] +# Each `mcp:` grant is named literally. `grants_cover_server` refuses to let `*` +# reach an MCP server (`src/runtime/tools.rs`), so a wildcard here would silently +# acquire any server an operator later installed — and the whole point of this +# bundle is that a seat's reach is exactly its role's. +# +# No `web` and no `search`: every fact about this business is in the shared +# retail state. A desk that could look things up would answer questions about +# online retail in general instead of about THIS customer's order. +allow = [ + "mcp:tau2-retail-triage", + "mcp:tau2-retail-exchanges", + "mcp:tau2-retail-refunds", + "mcp:tau2-retail-cancellations", + "mcp:tau2-retail-amendments", +] + +[channels.operator] +enabled = true + +# BYOK routing. The KEY is never in the bundle — it is a company secret +# (`inference/key`), set from the console's Inference card or +# `PUT {scope}/inference`. A company that declares this block consults that +# secret rather than `OPENCOMPANY_INFERENCE_KEY`, so a host-level env var does +# not stand in for it: the first turn fails with a 401 from the platform +# endpoint, not from the provider named here. +[inference] +provider = "openrouter" +base_url = "https://openrouter.ai/api/v1" + +# Every tier on one model, deliberately. The seats differ by the tools they +# hold and the policy they carry, not by the model behind them — so a run that +# routes them apart would be measuring the models rather than the org chart. +# `vision-v1` is mapped too: nothing here sends an image, and leaving a tier +# unmapped is what makes a stray request fall back to a provider default +# nobody chose. +[inference.models] +chat-v1 = "deepseek/deepseek-v4-flash" +reasoning-v1 = "deepseek/deepseek-v4-flash" +agentic-v1 = "deepseek/deepseek-v4-flash" +vision-v1 = "deepseek/deepseek-v4-flash" + +# --------------------------------------------------------------------------- +# There is no `triage` DESK. `triage` is an agent, and a desk whose id equals +# one of its own members' id is ceremony that the runtime does not even +# represent consistently: the manifest says `triage`, `POST /chat` accepts +# `triage`, and `GET /desks` reports the same block back as `front_desk`. +# +# A desk exists to hold a ROOM — two or more seats with something to settle +# between them (`deliberates()` needs two). One seat is not a room, so the +# front door is the company's own `general` channel: the customer arrives, +# the responder ladder picks `triage` because reading the order is what it is +# for, and it refers the case on to whichever desk owns the remedy. +# --------------------------------------------------------------------------- + +# --------------------------------------------------------------------------- +# The company's own line, as a real desk. +# +# The id is NOT a General spelling: tinyhivemind refuses a desk whose identity +# collides with one (`reserved desk identity`), and a company that declares +# `id = "general"` boots clean and then fails every message on its main thread. +# The NAME is `General`, and `resolve_desk_id` matches a manifest desk by name +# as well as by id — with no General guard on that pass — so the company's own +# line resolves here instead of resolving to nothing. +# +# That is the actual defect this fixes. With general resolving to no desk, the +# desk selector bails out and the message falls to a *root* agent chosen by +# filename order: a delivered-order case was answered by `amendments`, the +# pending-order seat, holding three write tools and no exchange tool. +# --------------------------------------------------------------------------- + +[[group_chat]] +id = "company_line" +name = "Company Line" +description = "The company's own line. Everybody who works here." +# Documentation only — the runtime seats the live roster here, so a +# teammate created tomorrow is in this room without anybody editing +members = ["triage", "exchanges", "refunds", "cancellations", "amendments"] + +[group_chat.hive] +enabled = true +turn_budget = 12 +quorum = 2 +blind_round = true + +[group_chat.hive.referral] +enabled = true +reach = "desks" +max_hops = 2 +returns = true +peer_cap = 2 + +# --------------------------------------------------------------------------- +# Pending orders. Two seats, competing remedies. +# --------------------------------------------------------------------------- + +[[group_chat]] +id = "order_ops" +name = "Order Operations" +description = "Pending orders: cancel outright, or amend items, address or payment in place." +members = ["cancellations", "amendments"] + +[group_chat.hive] +enabled = true +# Two seats, and enough budget to survive ONE bad proposal. +# +# Eight was too tight, and the way it failed is instructive: `exchanges` +# proposed with item ids it had invented, `refunds` objected and named the real +# ones — cross-inhibition working exactly as intended — and the correction then +# had nowhere to go. `exchanges` spent its remaining turns supporting its own +# corrected option, which adds no quorum (two DIFFERENT members must back it), +# and the budget ran out on "refunds, do you agree?". Three of the eight turns +# were a duplicate evidence line, an unmarked line that folds to nothing, and +# that self-support. +# +# Twelve leaves room for the correction cycle a two-seat desk needs. It is still +# short on purpose: conformity in a group of language models rises with +# interaction time, so a long episode buys correlated error rather than scrutiny. +turn_budget = 12 +quorum = 2 +blind_round = true + +[group_chat.hive.referral] +enabled = true +reach = "desks" +max_hops = 2 +returns = true +peer_cap = 2 + +# --------------------------------------------------------------------------- +# Delivered orders. Same shape. +# --------------------------------------------------------------------------- + +[[group_chat]] +id = "returns" +name = "Returns and Exchanges" +description = "Delivered orders: exchange an item for a variant of the same product, or take a return for refund." +members = ["exchanges", "refunds"] + +[group_chat.hive] +enabled = true +# Twelve, not eight: one bad proposal plus its correction does not fit in +# eight when self-support adds no quorum. See the order_ops note above. +turn_budget = 12 +quorum = 2 +blind_round = true + +[group_chat.hive.referral] +enabled = true +reach = "desks" +max_hops = 2 +returns = true +peer_cap = 2 diff --git a/companies/retail_co/mcp.json b/companies/retail_co/mcp.json new file mode 100644 index 000000000..c70533e51 --- /dev/null +++ b/companies/retail_co/mcp.json @@ -0,0 +1,30 @@ +{ + "$comment": "This company's whole work environment is five role-scoped MCP servers from the `opencompany-tau2` repo, each serving one slice of the tau2-bench retail domain over streamable HTTP and all five sharing ONE state file under an exclusive flock. They ship DISABLED with placeholder https endpoints, because a bundle in this repo must not point an agent at a host nobody has provisioned. For a LOCAL run, leave them disabled here and register the loopback endpoints at runtime \u2014 `PUT /mcp/servers/{name}` is the only layer that accepts an http:// endpoint. `scripts/tau2-sim.py --domain retail` does exactly that.", + "mcpServers": { + "tau2-retail-triage": { + "url": "https://tau2-retail-triage.invalid/mcp", + "description": "Read-only front desk: find a user from name and zip, read their orders, read product and stock detail. Nine tools, none of which mutate.", + "enabled": false + }, + "tau2-retail-exchanges": { + "url": "https://tau2-retail-exchanges.invalid/mcp", + "description": "Delivered orders, remedy = exchange: swap an item for another variant of the same product. One mutating tool; holds no refund tool.", + "enabled": false + }, + "tau2-retail-refunds": { + "url": "https://tau2-retail-refunds.invalid/mcp", + "description": "Delivered orders, remedy = refund: take items back. One mutating tool; holds no exchange tool.", + "enabled": false + }, + "tau2-retail-cancellations": { + "url": "https://tau2-retail-cancellations.invalid/mcp", + "description": "Pending orders, remedy = cancel the whole order. One mutating tool; holds no modify tool.", + "enabled": false + }, + "tau2-retail-amendments": { + "url": "https://tau2-retail-amendments.invalid/mcp", + "description": "Pending orders, remedy = amend in place (items, address, payment). Three mutating tools; holds no cancel tool.", + "enabled": false + } + } +} diff --git a/scripts/tau2-sim.py b/scripts/tau2-sim.py new file mode 100755 index 000000000..91d42f993 --- /dev/null +++ b/scripts/tau2-sim.py @@ -0,0 +1,671 @@ +#!/usr/bin/env python3 +"""Replay tau2-bench tasks through an OpenCompany company, and grade the result. + +This is the tau2 counterpart to ``scripts/vending-sim.py``. Where that one runs a +business forward on a clock, this one replays **tau2-bench tasks** — a customer's +opening message, and the database end-state tau2 says a correct handling +produces — through a company whose desks each hold one remedy. + +What it is for is the thing tau2 itself cannot ask. Its orchestrator wires +exactly one agent to one user simulator, with no agent-to-agent path, so it can +score whether an agent called the right tool but not whether an ORGANISATION +routed the work to the seat that owns it. Here a task only completes if the case +reaches the right desk and that desk settles which remedy applies. + +Domains +------- + +``retail`` and ``airline`` are supported. ``telecom`` is NOT, and the reason is +in the task set rather than in this script: 2,048 of its expected actions are +``grant_app_permission``, 1,127 ``toggle_airplane_mode``, 1,040 ``reboot_device`` +— actions performed by tau2's **user simulator** on its own simulated handset, +not by the agent. Nothing in an agent-side database records them, so a run here +could not be graded even if every desk behaved perfectly. Telecom needs the user +simulator wired in as a participant first. + +The servers are NOT in this repo. They live in `opencompany-tau2`, which vendors +tau2-bench (~850 MB, mostly benchmark data) and needs its own venv — which is why +these bundles ship DISABLED placeholder entries in ``mcp.json`` and this script +repoints them at loopback. Start them first, one per seat: + + cd ../opencompany-tau2 + uv run tau2-mcp --roles roles/retail.yaml --role triage --http 8801 & + ... # etc + +Then, against a running ``opencompany serve --company companies/``: + + python3 scripts/tau2-sim.py --domain retail --task 0 + python3 scripts/tau2-sim.py --domain airline --tasks 7,12 --out run.json + +Stdlib only, so it runs wherever ``python3`` does. Exit status is the number of +tasks whose end state did not match tau2's ``evaluation_criteria``, so a +CI-style caller can treat zero as "the company handled every case correctly". +""" + +from __future__ import annotations + +import argparse +import http.cookiejar +import json +import re +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path +from typing import Any + +# Per-domain wiring. `seats` maps a seat to the loopback port its role server +# listens on, in the order the servers are started; `entry` is the desk a task's +# opening message is posted to, the way a customer would arrive. +# +# `writes` is the grading table: for each mutating action tau2 asserts, how to +# read the same fact back out of the shared database. Reads are deliberately not +# graded — they are how an agent gets there, and two correct handlings can read +# different things. +DOMAINS = { + "retail": { + "company": "retail-co", + # Enter at the desk that owns delivered orders: two seats holding + # competing remedies. Small enough to read, and it still has to + # reach outside itself for what only another seat can answer — + # which is the crossing worth watching. + "entry": "returns", + "state": ".state/retail.json", + # seat -> (port, tools in scope, of which mutating). The counts are the + # contract this whole design rests on: `triage` holding a write tool, or + # `exchanges` reaching the refund tool, means the scoping silently broke + # and every later result is meaningless. `--check` asserts them. + "seats": { + "triage": (8801, 9, 0), + "exchanges": (8802, 9, 1), + "refunds": (8803, 8, 1), + "cancellations": (8804, 8, 1), + "amendments": (8805, 11, 3), + }, + "desks": { + "order_ops": ["cancellations", "amendments"], + "returns": ["exchanges", "refunds"], + }, + "collection": "orders", + "key": "order_id", + "writes": { + "exchange_delivered_order_items": { + "status": "exchange requested", + "fields": { + "exchange_items": "item_ids", + "exchange_new_items": "new_item_ids", + "exchange_payment_method_id": "payment_method_id", + }, + }, + "return_delivered_order_items": { + "status": "return requested", + "fields": { + "return_items": "item_ids", + "return_payment_method_id": "payment_method_id", + }, + }, + "cancel_pending_order": {"status": "cancelled", "fields": {}}, + # The modify_* tools are once-per-order, so the end state is the + # comparison — not which call produced it. + "modify_pending_order_items": {"status": "pending", "fields": {}}, + "modify_pending_order_address": {"status": "pending", "fields": {}}, + "modify_pending_order_payment": {"status": "pending", "fields": {}}, + }, + }, + "airline": { + "company": "airline-co", + "entry": "triage", + "state": ".state/airline.json", + "seats": { + "triage": (8811, 8, 0), + "booking": (8812, 9, 1), + "changes": (8813, 11, 3), + "refunds": (8814, 10, 2), + }, + "desks": { + "triage": ["triage"], + "booking": ["booking"], + "post_booking": ["changes", "refunds"], + }, + "collection": "reservations", + "key": "reservation_id", + "writes": { + # A cancelled reservation is REMOVED from the collection rather than + # flagged, so absence is the assertion. + "cancel_reservation": {"absent": True, "fields": {}}, + "update_reservation_flights": {"fields": {"cabin": "cabin"}, "flights": "flights"}, + "update_reservation_baggages": { + "fields": {"total_baggages": "total_baggages", + "nonfree_baggages": "nonfree_baggages"}, + }, + "update_reservation_passengers": {"fields": {"passengers": "passengers"}}, + # A booking creates the row; presence under the asserted id is the + # assertion, since tau2 does not fix the generated id in advance. + "book_reservation": {"present": True, "fields": {}}, + }, + }, +} + +UNSUPPORTED = { + "telecom": ( + "telecom tasks are graded on the USER's device, not the agent's database: " + "2,048 expected actions are `grant_app_permission`, 1,127 " + "`toggle_airplane_mode`, 1,040 `reboot_device`. Those are performed by " + "tau2's user simulator on its own simulated handset, so no agent-side " + "state records them and a run here cannot be scored. Wire the user " + "simulator in as a participant before enabling this domain." + ), +} + +ADMIN_EMAIL = "harness-e2e@tinyhumans.ai" + + +class Host: + """The running company, over its HTTP API.""" + + def __init__(self, base: str, company: str) -> None: + self.base = base.rstrip("/") + self.scope = f"/api/v1/companies/{company}" + jar = http.cookiejar.CookieJar() + self.opener = urllib.request.build_opener( + urllib.request.HTTPCookieProcessor(jar) + ) + + def call(self, method: str, path: str, body: Any = None, timeout: float = 120): + data = None if body is None else json.dumps(body).encode() + req = urllib.request.Request(self.base + path, data=data, method=method) + req.add_header("accept", "application/json") + if data is not None: + req.add_header("content-type", "application/json") + try: + with self.opener.open(req, timeout=timeout) as resp: + raw = resp.read() + return resp.status, (json.loads(raw) if raw else None) + except urllib.error.HTTPError as err: + raw = err.read() + try: + return err.code, json.loads(raw) + except ValueError: + return err.code, raw.decode(errors="replace") + except (urllib.error.URLError, OSError) as err: + # Nothing listening, or the host died mid-run. Reported as 0 rather + # than raising, so `--check` can say "unreachable" instead of dying + # with a traceback — and so it is never mistaken for a 404 from a + # host that IS answering. + return 0, f"unreachable: {err}" + + def sign_in(self) -> None: + """No-op when auth is `none`; otherwise the loopback dev-code flow.""" + status, _ = self.call("GET", f"{self.scope}/chat/history?limit=1") + if status == 200: + return + status, body = self.call("POST", f"{self.scope}/auth/request", {"email": ADMIN_EMAIL}) + code = (body or {}).get("dev_code") if isinstance(body, dict) else None + if not code: + raise SystemExit(f"sign-in: no dev_code from auth/request ({status}: {body})") + status, body = self.call("POST", f"{self.scope}/auth/verify", {"code": code}) + if status >= 300: + raise SystemExit(f"sign-in: verify refused ({status}: {body})") + + def register_mcp(self, name: str, endpoint: str) -> tuple[int, Any]: + """Point one declared server at this run's loopback role server. + + Runtime is the only layer that accepts an ``http://`` endpoint — a + server declared in a bundle's ``mcp.json`` must be ``https``. That is + why this bundle ships all five disabled and pointing at placeholders. + + **PUT, not POST.** The name is already declared, so adding it is a 409 + telling you to override instead — and a POST that 409s leaves the desks + holding the shipped, disabled placeholder and deliberating confidently + with no tools at all. + """ + status, body = self.call( + "PUT", + f"{self.scope}/mcp/servers/{urllib.parse.quote(name)}", + {"endpoint": endpoint, "enabled": True}, + ) + if status < 300: + return status, body + return self.call( + "POST", f"{self.scope}/mcp/servers", {"name": name, "endpoint": endpoint} + ) + + def activity(self, desk: str) -> tuple[int, bool]: + """How many rows this channel holds, and whether a room has closed on it. + + The settle loop's event source. A closing report is journaled under the + reserved `hive-report` author when an episode ends — whatever it ended + as — so its arrival means the room is done and further waiting buys + nothing. The count catches the other shape: a single-responder desk, or + a referral answering on a channel with no room at all. + """ + # `desk=`, not `chat=`. The handler's query struct is `#[serde(default)]`, + # so an unknown parameter is dropped in silence and `desk` falls back to + # the General line: `?chat=anything`, including a desk that does not + # exist, returns General's transcript with a 200. This loop was polling + # General on every tick while claiming to watch the desk it settles on. + status, body = self.call( + "GET", f"{self.scope}/chat/history?desk={urllib.parse.quote(desk)}" + ) + if status != 200 or not isinstance(body, list): + return (0, False) + closed = any((m.get("channel") or m.get("from") or "") == "hive-report" for m in body) + return (len(body), closed) + + def say(self, desk: str, text: str, timeout: float = 3600, parent: str | None = None): + """Put one message to `desk`, holding the POST open for the turn. + + `parent` threads this message onto an earlier one. Without it every turn + is a new line in the channel, and `reply_thread` roots each one on + itself — "N messages would mean N threads instead of N *topics*". A + follow-up then opens its OWN episode, deriving a fresh topic id from its + own words (observed: a confirmation produced the topic `#yes-i`, a room + deliberating about the word "yes"), and the prior exchange is demoted to + the cross-thread index, which the agent is told not to read: "do NOT + read or answer from them unless this message explicitly refers to one". + So "go ahead as you described" pointed at a conversation the room was + barred from consulting. + + Threaded, the follow-up lands inside the room that asked, that room + keeps its own transcript in view, and the seat that made the decision is + the one that acts on the answer. + """ + body = {"text": text, "chat": desk} + if parent is not None: + body["parent"] = str(parent) + return self.call("POST", f"{self.scope}/chat", body, timeout=timeout) + + +def load_tasks(data_dir: Path, domain: str) -> list[dict]: + path = data_dir / "tau2" / "domains" / domain / "tasks.json" + if not path.exists(): + raise SystemExit( + f"no tau2 task file at {path}\n" + "Pass --tau2 pointing at the opencompany-tau2 checkout's " + "vendor/tau2-bench/data directory." + ) + raw = json.loads(path.read_text()) + return raw if isinstance(raw, list) else raw.get("tasks", []) + + +# tau2 writes `user_scenario.instructions` in the second person, because they are +# directions to ITS user simulator — "You are Yusuf Rossi", "you wish to +# exchange". Handed to an agent verbatim they read as stage directions, and the +# agent answers them as such: the first run of this script had `triage` reply +# "You said: You are Yusuf Rossi in zip code 19122…" and nothing else. +# +# This flips them to first person so the desk receives something a customer +# could plausibly have written. It is a crude stand-in for the user simulator, +# and only for the OPENING message — a task needing genuine back-and-forth (a +# confirmation, a preference the desk has to ask for) still needs the simulator +# wired in as a participant. See `--help`. +_PERSON = [ + (r"\byou'd\b", "I'd"), (r"\bYou'd\b", "I'd"), + (r"\byou're\b", "I'm"), (r"\bYou're\b", "I'm"), + (r"\byou've\b", "I've"), (r"\bYou've\b", "I've"), + (r"\byou are\b", "I am"), (r"\bYou are\b", "I am"), + (r"\byou have\b", "I have"), (r"\bYou have\b", "I have"), + (r"\byou wish\b", "I wish"), (r"\bYou wish\b", "I wish"), + (r"\byou want\b", "I want"), (r"\bYou want\b", "I want"), + (r"\byourself\b", "myself"), (r"\byours\b", "mine"), + (r"\byour\b", "my"), (r"\bYour\b", "My"), + (r"\byou\b", "I"), (r"\bYou\b", "I"), +] + + +def as_customer(text: str) -> str: + """tau2's second-person directions, rewritten as the customer's own words.""" + for pattern, repl in _PERSON: + text = re.sub(pattern, repl, text) + # "to I" / "for I" — the object case the blunt swap above gets wrong. + text = re.sub(r"\b(to|for|with|at|from|of) I\b", r"\1 me", text) + return text + + +def opening_message(task: dict) -> str: + """The customer's first line, as tau2 states it. + + `known_info` carries the identity the agent has to establish (name, zip), + which a real customer would volunteer; `reason_for_call` is what they want. + """ + ui = (task.get("user_scenario") or {}).get("instructions") or {} + known = as_customer((ui.get("known_info") or "").strip()) + reason = as_customer((ui.get("reason_for_call") or "").strip()) + return f"{known}\n\n{reason}".strip() if known else reason + + +# tau2's retail and airline policies require the agent to state the action and +# get an explicit "yes" before any write, so a one-turn replay cannot complete +# those tasks however well the desks behave: the room correctly stops and asks. +# tau2 answers that with its user simulator; this is the bounded stand-in. +# +# It is deliberately dumb — it confirms what the desk proposed, and says nothing +# the task did not already state. A task needing a genuine CHOICE from the user +# (which of two variants, refund or exchange) is not completable this way and +# will fail here; that is the honest result, not something to paper over with a +# cleverer script. +FOLLOW_UP = ( + "Yes — I confirm, go ahead exactly as you described. " + "Use the original payment method on the order. I have nothing to add." +) + + +def communicated(task: dict, replies: list[str]) -> tuple[list[str], list[str]]: + """Which `communicate_info` strings actually reached the customer. + + tau2 grades most tasks on ``reward_basis = ["DB", "NL_ASSERTION"]``: the + database end state AND natural-language assertions about what the agent + said, the latter judged by an LLM in tau2's own harness. This checks neither + — it is a literal substring test over what the desks replied, which is a + deterministic PROXY for the communication half and nothing more. A task can + pass here and still fail tau2's judge. + """ + want = [str(x) for x in ((task.get("evaluation_criteria") or {}).get("communicate_info") or [])] + blob = "\n".join(r or "" for r in replies) + return want, [w for w in want if w not in blob] + + +def grade(task: dict, state: dict, spec: dict) -> tuple[bool, str]: + """Compare the shared database against tau2's expected end state. + + Only the write actions are graded. Reads are how an agent gets there, and + tau2's own scoring does not require a particular path through them — two + correct handlings can read different things. + """ + wants = [a for a in ((task.get("evaluation_criteria") or {}).get("actions") or []) + if a.get("name") in spec["writes"]] + if not wants: + return True, "no gradeable write expected" + + rows = state.get(spec["collection"]) or {} + problems = [] + for want in wants: + name = want["name"] + rule = spec["writes"][name] + args = want.get("arguments") or {} + rid = args.get(spec["key"]) + row = rows.get(rid) + + if rule.get("absent"): + if row is not None: + problems.append(f"{name}: {rid} is still present") + continue + if row is None: + problems.append(f"{name}: {rid} not in {spec['collection']}") + continue + if rule.get("present"): + continue + + want_status = rule.get("status") + if want_status is not None and row.get("status") != want_status: + problems.append(f"{name}: status={row.get('status')!r}, wanted {want_status!r}") + for field, arg in (rule.get("fields") or {}).items(): + if arg in args and row.get(field) != args[arg]: + problems.append(f"{name}: {field} does not match {arg}") + # `flights` is asserted as a list of {flight_number, date} pairs; the row + # stores richer objects, so compare only the keys tau2 named. + if "flights" in rule and "flights" in args: + got = [{k: f.get(k) for k in ("flight_number", "date")} + for f in (row.get("flights") or [])] + want_f = [{k: f.get(k) for k in ("flight_number", "date")} for f in args["flights"]] + if got != want_f: + problems.append(f"{name}: flights do not match") + + return (not problems), "; ".join(problems) or "matches" + + +def check(host: "Host", spec: dict, domain: str, state_path: Path) -> int: + """Preflight: assert every layer this run depends on, spending no model call. + + Each of these has failed silently at least once while this bundle was being + built, and each looked like a model problem from the transcript alone: + + * a role server reading a state file deleted under it — every tool call came + back `Error executing tool`, which reads as the agent using them wrong; + * `mcp.json` declaring server names the runner never registered, so a desk + deliberated confidently with no tools at all; + * a scope quietly widening, which makes a pass meaningless rather than loud; + * no inference key, which surfaces as a 401 on the first turn rather than at + boot; + * a desk with one member where two were intended — `deliberates()` needs + two, so the room never convenes and one seat decides alone. + """ + bad = 0 + + def ok(label: str, good: bool, detail: str = "") -> None: + nonlocal bad + bad += 0 if good else 1 + mark = "ok " if good else "FAIL" + print(f" [{mark}] {label}{(' — ' + detail) if detail else ''}") + + print(f"role servers ({domain})") + for seat, (port, want_tools, want_mut) in spec["seats"].items(): + url = f"http://127.0.0.1:{port}/mcp" + body = json.dumps({"jsonrpc": "2.0", "id": 1, + "method": "tools/list", "params": {}}).encode() + req = urllib.request.Request(url, data=body, method="POST") + req.add_header("content-type", "application/json") + req.add_header("accept", "application/json, text/event-stream") + try: + with urllib.request.urlopen(req, timeout=20) as resp: + raw = resp.read().decode(errors="replace") + except Exception as err: # noqa: BLE001 — any failure is the same verdict + ok(f"{seat} :{port}", False, f"unreachable ({err})") + continue + tools = len(re.findall(r'"name":"[a-z_]+"', raw)) + mut = raw.count("write/mutates") + ok(f"{seat} :{port}", tools == want_tools and mut == want_mut, + f"{tools} tools / {mut} mutating, wanted {want_tools}/{want_mut}") + + print("company") + status, desks = host.call("GET", f"{host.scope}/desks") + ok("desks readable", status == 200, f"HTTP {status}") + if status == 200 and isinstance(desks, list): + got = {d.get("id"): sorted(d.get("members") or []) for d in desks} + for desk, members in spec["desks"].items(): + ok(f"desk {desk}", got.get(desk) == sorted(members), + f"members={got.get(desk)}, wanted {sorted(members)}") + if len(members) >= 2: + ok(f"desk {desk} can deliberate", len(got.get(desk) or []) >= 2, + "needs two seats") + + print("mcp wiring") + status, servers = host.call("GET", f"{host.scope}/mcp/servers") + if status != 200 or not isinstance(servers, list): + ok("server list", False, f"HTTP {status}") + else: + by_name = {x.get("name"): x for x in servers} + for seat in spec["seats"]: + name = f"tau2-{domain}-{seat}" + row = by_name.get(name) + live = bool(row and row.get("enabled")) + ok(f"{name} enabled", live, + "" if live else ("not registered" if row is None else "declared but disabled")) + if row and row.get("enabled"): + st, tools = host.call("GET", f"{host.scope}/mcp/servers/{urllib.parse.quote(name)}/tools") + ok(f"{name} reachable from the host", st == 200 and isinstance(tools, list), + f"HTTP {st}") + + print("inference") + status, body = host.call("POST", f"{host.scope}/inference/test", {}, timeout=120) + fine = status == 200 and isinstance(body, dict) and body.get("ok") + ok("credential probes clean", bool(fine), + (body or {}).get("error", f"HTTP {status}") if not fine else "") + + print("tau2 state") + ok(f"{state_path} present", state_path.exists(), + "delete it AND restart the servers to reseed — they seed at boot") + + print(f"\n{'all checks passed' if not bad else str(bad) + ' check(s) failed'}") + return bad + + +def main() -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + ap.add_argument("--domain", default="retail", help="tau2 domain: " + ", ".join(DOMAINS)) + ap.add_argument("--base", default="http://127.0.0.1:8080", help="running `opencompany serve`") + ap.add_argument("--tau2", type=Path, default=Path("../opencompany-tau2"), + help="the opencompany-tau2 checkout (tasks + shared state)") + ap.add_argument("--check", action="store_true", + help="verify servers, desks, mcp wiring, credential and state, " + "then exit — spends no model call") + ap.add_argument("--task", help="one task id") + ap.add_argument("--tasks", help="comma-separated task ids") + ap.add_argument("--host", default="127.0.0.1", help="host the role servers bound to") + ap.add_argument("--out", type=Path, help="write the run as JSON") + ap.add_argument("--settle", type=float, default=240, + help="seconds to wait after each turn for a detached referral " + "to land before grading (the far desk runs after the POST " + "returns)") + ap.add_argument("--quiet", type=float, default=60, + help="stop settling once the channel has been silent this " + "long — the work is not coming") + ap.add_argument("--turns", type=int, default=8, + help="MAX turns per task, not a target — the loop stops as soon " + "as the end state matches. Turn 2+ sends the canned " + "confirmation (a stand-in for tau2's user simulator). A " + "low cap silently fails every task whose policy demands " + "confirmation before a write, which is most of them: the " + "desk asks, nobody answers, and it reads as a refusal to act") + ap.add_argument("--timeout", type=float, default=3600, help="seconds to hold one turn open") + args = ap.parse_args() + + if args.domain in UNSUPPORTED: + raise SystemExit(f"{args.domain}: {UNSUPPORTED[args.domain]}") + if args.domain not in DOMAINS: + ap.error(f"unknown domain {args.domain!r}; known: {', '.join(DOMAINS)}") + spec = DOMAINS[args.domain] + + state_path = args.tau2 / spec["state"] + host = Host(args.base, spec["company"]) + + if args.check: + host.sign_in() + for seat, (port, _t, _m) in spec["seats"].items(): + host.register_mcp(f"tau2-{args.domain}-{seat}", f"http://{args.host}:{port}/mcp") + return check(host, spec, args.domain, state_path) + + ids = [] + if args.task: + ids = [args.task] + if args.tasks: + ids += [t.strip() for t in args.tasks.split(",") if t.strip()] + if not ids: + ap.error("pass --task or --tasks") + + tasks = {str(t["id"]): t + for t in load_tasks(args.tau2 / "vendor" / "tau2-bench" / "data", args.domain)} + missing = [i for i in ids if i not in tasks] + if missing: + raise SystemExit(f"no such {args.domain} task(s): {', '.join(missing)}") + + host.sign_in() + for seat, (port, _tools, _mut) in spec["seats"].items(): + name = f"tau2-{args.domain}-{seat}" + status, body = host.register_mcp(name, f"http://{args.host}:{port}/mcp") + if status >= 300: + raise SystemExit(f"could not register {name}: {status} {body}") + print(f"registered {len(spec['seats'])} role servers for {args.domain}", file=sys.stderr) + + results = [] + failed = 0 + for tid in ids: + task = tasks[tid] + text = opening_message(task) + print(f"\n=== {args.domain} task {tid} ===\n{text}\n", file=sys.stderr) + + turns = [] + thread = None + ok, why = False, "no turn ran" + for turn in range(1, args.turns + 1): + status, body = host.say(spec["entry"], text, timeout=args.timeout, parent=thread) + # The first message roots the thread; every follow-up joins it. + if thread is None and isinstance(body, dict): + thread = body.get("messageId") + replies = ([r.get("text") for r in (body or {}).get("responses", [])] + if isinstance(body, dict) else []) + for r in replies: + print(f" [{spec['entry']}] {r}", file=sys.stderr) + turns.append({"turn": turn, "status": status, "sent": text, + "thread": thread, "replies": replies}) + + # A referral is DETACHED — `spawn_referred_turn` puts the question on + # the other desk's channel and returns; the POST answering here does + # not wait for that room to finish. Grading the instant the POST + # returns therefore races the work it is grading. + # + # Event-driven rather than a fixed wait. Sitting out the whole + # `--settle` budget is only correct when the work is still coming; + # when it is not, it is dead time, and it dominated the wall clock + # of every failing run — two turns of a 600s budget is twenty + # minutes spent waiting for something that already was not going to + # happen. So stop on the first of: the state matching, the room + # closing (a `hive-report` row), or the channel going quiet. + ok, why = False, "no state yet" + deadline = time.monotonic() + args.settle + last_seen, quiet_since = None, time.monotonic() + while True: + state = json.loads(state_path.read_text()) if state_path.exists() else {} + ok, why = grade(task, state, spec) + if ok: + break + rows, closed = host.activity(spec["entry"]) + if rows != last_seen: + last_seen, quiet_since = rows, time.monotonic() + if closed: + why += " (the room closed)" + break + if time.monotonic() - quiet_since >= args.quiet: + why += f" (nothing journaled for {args.quiet:.0f}s)" + break + if time.monotonic() >= deadline: + why += " (settle budget spent)" + break + time.sleep(5) + if ok: + break + if turn < args.turns: + # The desk is most likely holding for the confirmation its policy + # demands. Answer it once and let it act. + print(f" … not settled ({why}); confirming", file=sys.stderr) + text = FOLLOW_UP + + said = [r for t in turns for r in (t["replies"] or [])] + want_info, missing_info = communicated(task, said) + basis = (task.get("evaluation_criteria") or {}).get("reward_basis") or [] + + failed += 0 if ok else 1 + print(f" -> DB {'PASS' if ok else 'FAIL'} ({why}) in {len(turns)} turn(s)", file=sys.stderr) + if want_info: + print(f" communicate_info {len(want_info) - len(missing_info)}/{len(want_info)}" + + (f", missing {missing_info}" if missing_info else ""), file=sys.stderr) + if "NL_ASSERTION" in basis: + print(" note: tau2 also scores NL_ASSERTION here, which needs its judge", + file=sys.stderr) + + results.append({ + "id": tid, + # DB only. Named so a reader cannot mistake it for tau2's reward. + "db_passed": ok, + "db_detail": why, + "reward_basis": basis, + "communicate_info": {"expected": want_info, "missing": missing_info}, + "scored_here": "DB end state, plus a literal substring proxy for " + "communicate_info. NL_ASSERTION is NOT scored — it needs " + "tau2's LLM judge over the transcript.", + "turns": turns, + }) + + if args.out: + args.out.write_text(json.dumps({"domain": args.domain, "results": results}, indent=2) + "\n") + print(f"\nwrote {args.out}", file=sys.stderr) + print(f"\n{len(ids) - failed}/{len(ids)} passed", file=sys.stderr) + return failed + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/tau2-up.sh b/scripts/tau2-up.sh new file mode 100755 index 000000000..1e8543a09 --- /dev/null +++ b/scripts/tau2-up.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# Bring a tau2 company up end to end, and print where to watch it. +# +# One command from nothing to a running company with its role servers wired, +# its credential set, every layer verified, and a console URL. Everything it +# starts is logged under --logs and torn down by `tau2-up.sh down`. +# +# scripts/tau2-up.sh # bring it up, print the console link +# scripts/tau2-up.sh --task 0 # ...then run one tau2 task +# scripts/tau2-up.sh down # stop everything it started +# +# The role servers live in the `opencompany-tau2` checkout (--tau2), which +# vendors tau2-bench and needs its own venv; they are NOT in this repo. See +# companies/retail_co/README.md. +set -uo pipefail + +DOMAIN=retail +BASE_PORT=8099 +TAU2="${OC_TAU2:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../opencompany-tau2" 2>/dev/null && pwd)}" +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +HOME_DIR="${OC_RIG_HOME:-/tmp/tau2-rig}" +LOGS="${OC_LOGS:-/tmp/tau2-logs}" +CONSOLE=1 +TASK="" +SEATS=(triage exchanges refunds cancellations amendments) + +while [ $# -gt 0 ]; do + case "$1" in + down) DOWN=1 ;; + --task) TASK="$2"; shift ;; + --tasks) TASK="$2"; shift ;; + --domain) DOMAIN="$2"; shift ;; + --tau2) TAU2="$2"; shift ;; + --port) BASE_PORT="$2"; shift ;; + --logs) LOGS="$2"; shift ;; + --no-console) CONSOLE=0 ;; + -h|--help) sed -n '2,14p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $1" >&2; exit 2 ;; + esac + shift +done + +say() { printf '\033[1m%s\033[0m\n' "$*"; } +ok() { printf ' \033[32m✓\033[0m %s\n' "$*"; } +bad() { printf ' \033[31m✗\033[0m %s\n' "$*"; } + +stop_all() { + say "stopping" + pkill -f "tau2mcp.server" 2>/dev/null && ok "role servers" || true + pkill -f "opencompany.*serve.*${DOMAIN}_co" 2>/dev/null || pkill -f "opencompany serve" 2>/dev/null && ok "company" || true + pkill -f "vite.*--port" 2>/dev/null || true + [ -f "$LOGS/console.pid" ] && kill "$(cat "$LOGS/console.pid")" 2>/dev/null && ok "console" + rm -f "$LOGS/console.pid" +} + +if [ "${DOWN:-0}" = 1 ]; then stop_all; exit 0; fi + +mkdir -p "$LOGS" +[ -d "$TAU2" ] || { bad "no opencompany-tau2 checkout at $TAU2 — pass --tau2"; exit 1; } +PY="$TAU2/.venv/bin/python3" +[ -x "$PY" ] || { bad "no venv at $PY — run \`uv sync\` in $TAU2"; exit 1; } + +# `CARGO_TARGET_DIR` first — a shared target dir is the usual setup when +# several worktrees of this repo are built side by side, and the binary is +# then nowhere near $REPO. +BIN="${OC_BIN:-}" +[ -x "${BIN:-}" ] || BIN="${CARGO_TARGET_DIR:+$CARGO_TARGET_DIR/debug/opencompany}" +[ -x "${BIN:-}" ] || BIN="$REPO/target/debug/opencompany" +[ -x "${BIN:-}" ] || BIN="$(ls -t "$REPO"/target*/debug/opencompany 2>/dev/null | head -1)" +[ -x "${BIN:-}" ] || { bad "no opencompany binary — cargo build --features openhuman,hivemind,mcp --bin opencompany"; exit 1; } + +stop_all >/dev/null 2>&1 +sleep 2 + +# 1. Role servers. Reseed the shared state FIRST: they seed it at boot, so +# deleting it under a running server leaves every tool call erroring. +say "role servers ($DOMAIN)" +rm -f "$TAU2/.state/$DOMAIN.json" "$TAU2/.state/$DOMAIN.json.lock" +i=0 +for seat in "${SEATS[@]}"; do + ( cd "$TAU2" && nohup "$PY" -m tau2mcp.server --roles "roles/$DOMAIN.yaml" \ + --role "$seat" --http $((8801 + i)) > "$LOGS/$seat.log" 2>&1 & ) + i=$((i + 1)) +done +sleep 12 +i=0 +for seat in "${SEATS[@]}"; do + n=$(curl -s -X POST "http://127.0.0.1:$((8801 + i))/mcp" \ + -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' 2>/dev/null \ + | grep -oE '"name":"[a-z_]+"' | wc -l | tr -d ' ') + [ "${n:-0}" -gt 0 ] && ok "$seat :$((8801 + i)) — $n tools" || bad "$seat :$((8801 + i)) — no answer (see $LOGS/$seat.log)" + i=$((i + 1)) +done + +# 2. The company. +say "company" +rm -rf "$HOME_DIR"; mkdir -p "$HOME_DIR" +RUST_LOG="${RUST_LOG:-opencompany::hivemind=debug,opencompany::server::operator=info,opencompany=info}" \ +OPENCOMPANY_BIND="127.0.0.1:$BASE_PORT" \ + nohup "$BIN" serve --company "$REPO/companies/${DOMAIN}_co" --home "$HOME_DIR" \ + > "$LOGS/serve.log" 2>&1 & +for _ in $(seq 1 40); do + [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 3 "http://127.0.0.1:$BASE_PORT/healthz" 2>/dev/null)" = "200" ] && break + sleep 3 +done +if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 3 "http://127.0.0.1:$BASE_PORT/healthz")" = "200" ]; then + ok "serving on 127.0.0.1:$BASE_PORT (log: $LOGS/serve.log)" +else + bad "the company did not come up — see $LOGS/serve.log"; tail -5 "$LOGS/serve.log"; exit 1 +fi + +SCOPE="http://127.0.0.1:$BASE_PORT/api/v1/companies/${DOMAIN}-co" + +# 3. Credential. The models table goes WITH the key: an omitted `models` is +# stored as an empty map that shadows the manifest's, and the next turn asks +# the provider for a model nobody chose. +say "credential" +if [ -n "${OPENROUTER_API_KEY:-}" ]; then + M=deepseek/deepseek-v4-flash + curl -s -o /dev/null -X PUT "$SCOPE/inference" -H 'content-type: application/json' \ + -d "{\"provider\":\"openrouter\",\"base_url\":\"https://openrouter.ai/api/v1\",\"key\":\"$OPENROUTER_API_KEY\",\"models\":{\"chat-v1\":\"$M\",\"reasoning-v1\":\"$M\",\"agentic-v1\":\"$M\",\"vision-v1\":\"$M\"}}" + ok "set from \$OPENROUTER_API_KEY, every tier on $M" +else + bad "\$OPENROUTER_API_KEY is unset — set it, or configure Inference in the console" +fi + +# 4. Wire the servers and verify every layer before spending a model call. +say "preflight" +python3 "$REPO/scripts/tau2-sim.py" --domain "$DOMAIN" --check \ + --base "http://127.0.0.1:$BASE_PORT" --tau2 "$TAU2" || true + +# 5. The console: a Vite dev server proxying /api at this host. It picks its +# own port, so the URL is read back out of its output rather than assumed. +if [ "$CONSOLE" = 1 ] && [ -d "$REPO/frontend/node_modules" ]; then + say "console" + ( cd "$REPO/frontend" && OC_API_TARGET="http://127.0.0.1:$BASE_PORT" \ + nohup npm run dev > "$LOGS/console.log" 2>&1 & echo $! > "$LOGS/console.pid" ) + URL="" + for _ in $(seq 1 30); do + URL=$(grep -oE 'http://(localhost|127\.0\.0\.1):[0-9]+' "$LOGS/console.log" 2>/dev/null | head -1) + [ -n "$URL" ] && break + sleep 2 + done + [ -n "$URL" ] && ok "$URL" || bad "console did not report a URL — see $LOGS/console.log" +elif [ "$CONSOLE" = 1 ]; then + say "console" + bad "frontend/node_modules missing — run \`npm install\` in frontend/ to watch in a browser" +fi + +echo +say "watch it" +echo " console ${URL:-(not started)}" +echo " api http://127.0.0.1:$BASE_PORT/api/v1/companies/${DOMAIN}-co" +echo " host log tail -f $LOGS/serve.log" +echo " turns ls $HOME_DIR/harness/${DOMAIN}-co/*/workspace/sessions/*/*.md" +echo +echo " run a task: python3 scripts/tau2-sim.py --domain $DOMAIN --task 0 --base http://127.0.0.1:$BASE_PORT --tau2 $TAU2" +echo " stop: scripts/tau2-up.sh down" + +if [ -n "$TASK" ]; then + echo + say "task $TASK" + # No --turns here: the runner's own default is the max, and a cap set from + # this side silently fails every task whose policy wants a confirmation. + python3 "$REPO/scripts/tau2-sim.py" --domain "$DOMAIN" --tasks "$TASK" --settle 600 --quiet 90 \ + --base "http://127.0.0.1:$BASE_PORT" --tau2 "$TAU2" --out "$LOGS/run.json" + echo " record: $LOGS/run.json" +fi diff --git a/src/company/content_test.rs b/src/company/content_test.rs index 52ff88013..347bcf73f 100644 --- a/src/company/content_test.rs +++ b/src/company/content_test.rs @@ -142,13 +142,20 @@ const SEARCH_GRANTED_COMPANIES: [&str; 21] = [ /// a desk that could reach the web would answer about vending machines in general /// instead of about these eight. Withholding the network is what makes a decision /// there attributable to the fleet it was made about. -const SEARCH_DENIED_COMPANIES: [&str; 6] = [ +const SEARCH_DENIED_COMPANIES: [&str; 7] = [ "agentic_math_lab", "hive_math_lab", "e2e_harness", "e2e_setup", "openhuman_demo", "vending_machine_co", + // Denied on exactly `vending_machine_co`'s argument. Every fact this bundle + // reasons from — what is on the order, which variants are in stock, what the + // customer paid — is a tool call against the shared tau2 retail state, and a + // desk that could reach the web would answer about online retail in general + // instead of about THIS order. It is also scored against that state, so a + // fact from outside it is not merely off-topic, it is unattributable. + "retail_co", ]; /// Templates that simply do not grant `search` today. Unlike @@ -1359,7 +1366,16 @@ const SETUP_SEEDED_COMPANIES: [&str; 24] = [ /// and `openhuman_demo` also declare their own `[[mcp_server]]` inline — so /// seeded cards and a second declaration of `deepwiki` would both perturb what /// they exist to pin down. -const FIXTURE_COMPANIES: [&str; 3] = ["e2e_harness", "e2e_setup", "openhuman_demo"]; +const FIXTURE_COMPANIES: [&str; 4] = [ + "e2e_harness", + "e2e_setup", + "openhuman_demo", + // A benchmark fixture: it proves a mechanism and is asserted against + // exactly, by tau2's own `evaluation_criteria`. Seeded cards would be + // work nobody asked for sitting in a company whose only job is to answer + // one replayed task and be scored on the end state. + "retail_co", +]; /// Every company is either a vertical that ships setup content or a fixture that /// deliberately does not — and the classification is re-derived from the files diff --git a/src/company/types.rs b/src/company/types.rs index 244598a0e..721023f73 100644 --- a/src/company/types.rs +++ b/src/company/types.rs @@ -612,6 +612,21 @@ pub struct Company { /// Company logo as a self-contained data:image/... URL (issue: operator-set brand logo). #[serde(default)] pub logo_url: Option, + /// The desk that owns the company's own line — the General channel. + /// + /// Unset (the default) keeps the historical behaviour: General resolves to + /// no desk, so a message there is answered by a single responder off the + /// fallback ladder. + /// + /// **The named desk must not itself be called General.** tinyhivemind + /// refuses a hive episode on a desk whose id *or name* is a General + /// spelling (`reserved desk identity`), and that refusal lands in the turn + /// rather than at load — a company that names one boots clean and then + /// fails every message on its main thread. This key exists precisely so the + /// company line can reach a room without any desk having to be called + /// General: the channel is General, the desk it resolves to is not. + #[serde(default)] + pub general_desk: Option, } /// A `[[agent]]` roster entry. diff --git a/src/hivemind/prompt.rs b/src/hivemind/prompt.rs index 5bd406bf0..0d4a2020b 100644 --- a/src/hivemind/prompt.rs +++ b/src/hivemind/prompt.rs @@ -89,6 +89,38 @@ fn move_line(kind: &str) -> Option<&'static str> { const FIRST_PERSON_RULE: &str = "Lines marked `(you)` in the transcript are your own. Write about \ yourself in the first person — never by your own id — and name colleagues by their id as usual."; +/// **A turn is work, then one line — not one line instead of work.** +/// +/// The move grammar describes the LINE a turn ends with, and a seat reading only +/// that treats the whole turn as speech: it reasons from whatever facts happen to +/// be in the transcript, and when there are none it asks the room instead of +/// looking. Observed, with the tools sitting in its own belt: a two-seat desk +/// spent all eight turns asking each other who held which tool and exhausted its +/// budget without a single read; a seat wrote "I hold the tool and can produce the +/// answer this turn" and then produced a line about the answer rather than the +/// answer. The one time a seat did call a tool, the operator's message had +/// literally told it to. +/// +/// Nothing was blocking those calls. `speak()` runs the ordinary turn machinery, +/// the belt is built the same way, and the same agent on the same desk calls the +/// same tools freely when it answers outside a room. What was missing is that +/// nobody asked it to: the prompt requested a position and it gave one. +/// +/// So the one-line contract stays exactly as it was — the fold reads the final +/// line and nothing else — and this says what the turn is allowed to do BEFORE +/// that line, which is everything an ordinary turn may do. +const WORK_BEFORE_LINE: &str = "\ +Before you write that line, USE YOUR TOOLS. A turn is work and then one line, \ +not one line instead of work. Look up what you need — the order, the item, the \ +product's variants, the customer — rather than asking the room for a fact you \ +can fetch yourself, and rather than reasoning from what happens to be in the \ +transcript already. A seat that asks its colleagues what it could have read \ +costs the room a turn and adds nothing.\n\ +And when the room has already carried an option that YOUR tool performs, \ +perform it in this turn, then write the line saying you did. Deciding is not \ +doing: no step after the room closes will carry out what it settled on, so an \ +action nobody performs never happens, however clearly it was agreed."; + const DELIBERATE_RULES: &str = "\ The # on a topic and the ^ on a citation are part of the grammar: `!propose \ #canary ...` names an option, `!propose canary ...` names nothing and is \ @@ -428,6 +460,8 @@ impl<'a> EpisodePrompt<'a> { let head = "Reply with ONE line only, beginning with exactly one of these markers:"; let mut tail = DELIBERATE_RULES.to_owned(); tail.push('\n'); + tail.push_str(WORK_BEFORE_LINE); + tail.push('\n'); tail.push_str(FIRST_PERSON_RULE); if self.quorum.require_evidential { tail.push('\n'); diff --git a/src/hivemind/types.rs b/src/hivemind/types.rs index 57c779e8b..e32f9d836 100644 --- a/src/hivemind/types.rs +++ b/src/hivemind/types.rs @@ -654,7 +654,14 @@ pub fn effective_hive_config(record: &CompanyRecord, desk_id: &str) -> HiveConfi #[must_use] pub fn desk_episode(record: &CompanyRecord, chat: Option<&str>) -> Option { let chat = chat?; - if crate::server::chat_history::is_general_chat(Some(chat)) { + // A General spelling opens a room only when the company has named the desk + // that owns its line (`[company].general_desk`). Unset, General resolves to + // nothing and keeps the single-responder main thread, exactly as before. + // The desk it resolves to is never itself called General — `resolve_desk_id` + // refuses that — so tinyhivemind's reserved-identity invariant holds. + if crate::server::chat_history::is_general_chat(Some(chat)) + && record.resolve_desk_id(chat).is_none() + { return None; } let desk_id = record.resolve_desk_id(chat)?; diff --git a/src/ports/types.rs b/src/ports/types.rs index 05d514f84..1bee65829 100644 --- a/src/ports/types.rs +++ b/src/ports/types.rs @@ -4785,6 +4785,43 @@ impl CompanyRecord { /// them. With no order override the base order is returned unchanged, so the /// first declared member stays the lead by default. pub fn effective_desk_members(&self, desk_id: &str) -> Vec { + // **The general desk seats the whole roster, and keeps doing so.** + // + // It is a desk like any other except in one respect: who belongs to it + // is not a list somebody maintains, it is "everyone who works here". + // `POST {scope}/team` adds a teammate and touches no desk at all, so a + // fixed `members = [...]` would be right on the day it was written and + // wrong from the next hire onward — the newest teammate would be the one + // person unable to speak on the company's own line. + // + // Deriving it from the roster makes that unmaintainable-by-construction + // rather than merely maintained, and it is what `[company].general_desk` + // is for: the manifest names WHICH desk owns the line, and the runtime + // keeps its membership current. + if self + .manifest + .company + .general_desk + .as_deref() + .is_some_and(|named| named == desk_id) + { + // The same two sources `is_roster_agent` consults, manifest before + // overlay, so who the room seats and who the roster says works here + // cannot drift. + let mut all: Vec = Vec::new(); + for id in self + .manifest + .agents + .iter() + .map(|a| a.id.clone()) + .chain(self.overlay_agents.iter().map(|a| a.id.clone())) + { + if !self.is_retired(&id) && !all.contains(&id) { + all.push(id); + } + } + return all; + } let mut members: Vec = self .manifest .group_chats @@ -4916,6 +4953,34 @@ impl CompanyRecord { if let Some(exact) = self.manifest.group_chats.iter().find(|c| c.id == key) { return Some(exact.id.clone()); } + // **The company's own line, pointed at a desk that owns it.** + // + // Without this, General resolves to nothing: the desk selector bails + // out and the message falls to a *root* agent picked off the fallback + // ladder — in practice whichever agent file sorts first. Observed: a + // delivered-order case answered by the pending-order seat, holding + // three write tools and no tool for the job. + // + // The target is required to exist and is required NOT to be a General + // spelling itself. tinyhivemind refuses an episode on a desk whose id + // or name is one, so resolving General onto such a desk would trade a + // message answered by the wrong agent for a message that fails + // outright. Silently declining leaves the historical behaviour, which + // is the same thing every other rung of this function does. + if tinyhivemind_core::chat::is_general_chat(Some(key)) + && let Some(target) = self.manifest.company.general_desk.as_deref() + && let Some(desk) = self + .manifest + .group_chats + .iter() + .find(|c| c.id == target) + .filter(|c| { + !tinyhivemind_core::chat::is_general_chat(Some(&c.id)) + && !tinyhivemind_core::chat::is_general_chat(Some(&c.name)) + }) + { + return Some(desk.id.clone()); + } if !tinyhivemind_core::chat::is_general_chat(Some(key)) && let Some(exact) = self .overlay_desks diff --git a/src/server/operator.rs b/src/server/operator.rs index 2370516c7..617b8c21b 100644 --- a/src/server/operator.rs +++ b/src/server/operator.rs @@ -193,27 +193,40 @@ async fn list_desks(scope: ScopedCompany) -> Result>, crate::s .map(|record| { // Manifest (blueprint) desks first, then operator-created overlay // desks — the same order the harness `desk_lead` resolver searches. - let manifest_desks = record.manifest.group_chats.iter().map(|chat| { - let members = record.effective_desk_members(&chat.id); - // The overlay subset: effective members not declared in the - // manifest for this desk. - let overlay_members = members - .iter() - .filter(|m| !chat.members.contains(m)) - .cloned() - .collect(); - DeskDto { - id: chat.id.clone(), - name: chat.name.clone(), - description: chat.description.clone(), - members, - overlay_members, - // Manifest desks are always lead-routed — the blueprint - // syntax carries no responder field (issue #1835). - responder: ResponderMode::Lead, - overlay_created: false, - } - }); + // The general desk is not listed beside General — it IS General. + // `[company].general_desk` names the desk the company's own line + // resolves to, so projecting it as its own channel puts the same + // room in the sidebar twice: once as the main thread everybody + // already has, once under whatever id the manifest gave it. Same + // reasoning, and same `is_general_chat` shape, as the overlay-desk + // exclusion below. + let general_desk = record.manifest.company.general_desk.clone(); + let manifest_desks = record + .manifest + .group_chats + .iter() + .filter(move |chat| general_desk.as_deref() != Some(chat.id.as_str())) + .map(|chat| { + let members = record.effective_desk_members(&chat.id); + // The overlay subset: effective members not declared in the + // manifest for this desk. + let overlay_members = members + .iter() + .filter(|m| !chat.members.contains(m)) + .cloned() + .collect(); + DeskDto { + id: chat.id.clone(), + name: chat.name.clone(), + description: chat.description.clone(), + members, + overlay_members, + // Manifest desks are always lead-routed — the blueprint + // syntax carries no responder field (issue #1835). + responder: ResponderMode::Lead, + overlay_created: false, + } + }); // An overlay desk whose own **id** is a General spelling is not // projected (issue #1781 review, Codex P2) — the grandfathered // shape `POST .../desks` accepted `general` / `main` ids under