Automates creation of task JSON files and reward functions for GUI automation testing.
Optimization: LLM generates both task name and reward function in a single call per task.
-
Create
.envfile with API key:ANTHROPIC_API_KEY=your_api_key_here -
Run the script:
uv run task_automation.py
The script will ask for:
- SPA name (e.g.,
notion,figma) - Version (e.g.,
2.0) - Initial backend state name (default:
default_backend) - Initial state JSON - paste the JSON or press Enter for SPA default
- Environment JSON - paste or use default
- Output paths for tasks and rewards file
- CSV file path
prompt,verification
"Create a new page titled 'Meeting Notes' in the workspace","Check that a page with title 'Meeting Notes' exists in the pages collection"
"Add a text block with content 'Hello World' to page-1","Verify a text block exists with content 'Hello World' on page-1"Column name variations accepted:
prompt/instructions/user_promptverification/success_criteria/verification_requirements
dojo-eval/
├── tasks/
│ ├── create-new-page-v2.json
│ └── add-text-block-v2.json
├── rewards/
│ └── notion_v2.py # Reward functions appended here
└── tasks_created.csv # Summary of all created tasks
tasks_created.csv format:
id,reward_func
create-new-page-v2,_validate_create_new_page
add-text-block-v2,_validate_add_text_blockNotes:
- If the rewards file already exists, new functions are appended
- If interrupted with Ctrl+C, tasks created so far are saved to the CSV
{
"spa": "figma",
"version": "2.0",
"id": "button-component-hierarchy-analysis-v2",
"name": "Button Component Hierarchy Analysis",
"description": "Button Component Hierarchy Analysis",
"tier": "pro",
"initial_backend_state_name": "default_backend",
"instructions": "{\"user_prompt\":\"...\",\"success_criteria\":\"...\"}",
"reward_function": "_validate_button_component_hierarchy_analysis",
"max_steps": 20,
"timeout_seconds": 180,
"environment_type": "gui",
"initial_state": "{...}",
"environment": "{...}",
...
}-
Single LLM call per task generates:
- Task name (3-6 words)
- Complete reward function with validation logic
-
Mechanical processing:
- Task ID derived from name (slugified +
-v2) - Reward function name derived from ID
- All other JSON fields from configuration
- Task ID derived from name (slugified +
-
On completion (or Ctrl+C interrupt):
- Saves
tasks_created.csvwith task ID to reward function mapping - Appends reward functions to existing rewards file (if any)
- Saves