Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

Dojo Eval - Task & Reward Function Generator

Automates creation of task JSON files and reward functions for GUI automation testing.

Optimization: LLM generates both task name and reward function in a single call per task.

Setup

  1. Create .env file with API key:

    ANTHROPIC_API_KEY=your_api_key_here
    
  2. Run the script:

    uv run task_automation.py

Configuration Prompts

The script will ask for:

  1. SPA name (e.g., notion, figma)
  2. Version (e.g., 2.0)
  3. Initial backend state name (default: default_backend)
  4. Initial state JSON - paste the JSON or press Enter for SPA default
  5. Environment JSON - paste or use default
  6. Output paths for tasks and rewards file
  7. CSV file path

CSV Format

prompt,verification
"Create a new page titled 'Meeting Notes' in the workspace","Check that a page with title 'Meeting Notes' exists in the pages collection"
"Add a text block with content 'Hello World' to page-1","Verify a text block exists with content 'Hello World' on page-1"

Column name variations accepted:

  • prompt / instructions / user_prompt
  • verification / success_criteria / verification_requirements

Output

dojo-eval/
├── tasks/
│   ├── create-new-page-v2.json
│   └── add-text-block-v2.json
├── rewards/
│   └── notion_v2.py              # Reward functions appended here
└── tasks_created.csv             # Summary of all created tasks

tasks_created.csv format:

id,reward_func
create-new-page-v2,_validate_create_new_page
add-text-block-v2,_validate_add_text_block

Notes:

  • If the rewards file already exists, new functions are appended
  • If interrupted with Ctrl+C, tasks created so far are saved to the CSV

Example Task JSON Output

{
  "spa": "figma",
  "version": "2.0",
  "id": "button-component-hierarchy-analysis-v2",
  "name": "Button Component Hierarchy Analysis",
  "description": "Button Component Hierarchy Analysis",
  "tier": "pro",
  "initial_backend_state_name": "default_backend",
  "instructions": "{\"user_prompt\":\"...\",\"success_criteria\":\"...\"}",
  "reward_function": "_validate_button_component_hierarchy_analysis",
  "max_steps": 20,
  "timeout_seconds": 180,
  "environment_type": "gui",
  "initial_state": "{...}",
  "environment": "{...}",
  ...
}

How It Works

  1. Single LLM call per task generates:

    • Task name (3-6 words)
    • Complete reward function with validation logic
  2. Mechanical processing:

    • Task ID derived from name (slugified + -v2)
    • Reward function name derived from ID
    • All other JSON fields from configuration
  3. On completion (or Ctrl+C interrupt):

    • Saves tasks_created.csv with task ID to reward function mapping
    • Appends reward functions to existing rewards file (if any)

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages