README.md

AutoSpec AI — Android E2E Test Generator

A GitHub Action that automatically generates Android E2E tests from commit diffs using LLMs. Analyzes your code changes and produces Maestro (YAML) or Espresso (Kotlin) tests.

Why Maestro as Default?

• Declarative YAML — no compilation, no Gradle dependency conflicts
• Works against any APK — no test dependencies injected into your build
• CI-friendly — Maestro CLI + Android emulator on GitHub Actions
• LLM-friendly — YAML is far less fragile than LLM-generated Kotlin that must compile against specific SDK versions

Espresso is available as an opt-in (test_framework: espresso) for teams with existing Espresso infrastructure.

Quick Start

name: Generate Android E2E Tests
on:
  pull_request:
    branches: [main]

jobs:
  autospec:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Generate Tests
        uses: autospec-ai/android@v1
        with:
          llm_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

This will:

1. Analyze the PR diff for UI-impacting changes
2. Auto-detect your project context (SDK versions, Compose usage, activities)
3. Generate Maestro YAML tests for changed screens
4. Create a PR with the generated tests

Features

• Two-phase LLM generation — plans tests first, then generates code
• Auto-detection — reads build.gradle and AndroidManifest.xml to understand your project
• File classification — categorizes changed files (Activity, Fragment, Composable, ViewModel, etc.) and scores UI impact
• Severity tagging — prioritizes tests by user impact (sev1-sev4)
• API mock generation — detects API dependencies and generates mock configurations
• Screenshot comparison — optional screenshot capture assertions
• Accessibility checks — optional accessibility validation

Inputs

LLM Configuration

Input	Default	Description
llm_provider	anthropic	anthropic, openai, or custom
llm_api_key	required	API key (use GitHub secrets)
llm_model	provider default	Model override
llm_base_url		Custom endpoint for OpenAI-compatible providers

Android Configuration

Input	Default	Description
application_id	auto-detect	Android application ID
module_path	app	Path to Android module
min_sdk	auto-detect	Minimum SDK version
build_variant	debug	Build variant to target
launcher_activity	auto-detect	Launcher activity class
android_framework	hybrid	compose, views, or hybrid

Test Framework

Input	Default	Description
test_framework	maestro	maestro or espresso
test_directory	.maestro/generated	Where tests are written
test_pattern	.maestro/**/*.yaml	Glob for existing tests
emulator_api	34	API level for emulator
device_profile	pixel_6	Device profile name

Diff & Behavior

Input	Default	Description
diff_mode	auto	auto, pr, or push
include_paths		Comma-separated path prefixes to include
exclude_paths	test/,androidTest/,...	Paths to exclude
auto_commit	false	Commit tests to current branch
auto_pr	true	Create a PR with generated tests
max_test_files	5	Max test files per run
dry_run	false	Preview without writing
custom_instructions		Additional LLM instructions

Optional Features

Input	Default	Description
generate_api_mocks	false	Generate API mock config
mock_strategy	maestro-mock	maestro-mock, mockwebserver, or okhttp-interceptor
fixture_extraction_threshold	3	Mock count before extracting to shared file
screenshot_comparison	false	Add screenshot assertions
accessibility_checks	false	Add accessibility assertions

Outputs

Output	Description
tests_generated	Number of test files generated
test_files	JSON array of generated test file paths
pr_number	PR number (if auto_pr enabled)
summary	Human-readable summary
fixture_files	JSON array of fixture file paths

Examples

Maestro (default)

- name: Generate Maestro Tests
  uses: autospec-ai/android@v1
  with:
    llm_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    test_framework: maestro
    android_framework: compose
    generate_api_mocks: true

Generated test example:

appId: com.example.app
---
- launchApp:
    clearState: true
- waitForAnimationToEnd
- tapOn:
    id: "email_input"
- inputText: "user@example.com"
- tapOn:
    id: "password_input"
- inputText: "password123"
- tapOn:
    id: "login_button"
- waitForAnimationToEnd
- assertVisible:
    id: "home_screen"

Espresso

- name: Generate Espresso Tests
  uses: autospec-ai/android@v1
  with:
    llm_api_key: ${{ secrets.OPENAI_API_KEY }}
    llm_provider: openai
    test_framework: espresso
    test_directory: app/src/androidTest/java
    test_pattern: "app/src/androidTest/**/*.kt"
    accessibility_checks: true

With API Mocking

- name: Generate Tests with Mocks
  uses: autospec-ai/android@v1
  with:
    llm_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    generate_api_mocks: true
    mock_strategy: maestro-mock
    fixture_extraction_threshold: 3

Dry Run

- name: Preview Tests
  uses: autospec-ai/android@v1
  with:
    llm_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    dry_run: true

How It Works

1. Diff Analysis — extracts changed files from PR or push, filters out non-source files (APKs, build artifacts, Gradle wrappers)
2. File Classification — categorizes each file (Activity, Fragment, Composable, ViewModel, layout XML, etc.) and determines UI impact
3. Project Detection — parses build.gradle for SDK versions, Compose/Hilt/Navigation usage; parses AndroidManifest.xml for activities and permissions
4. Test Planning (LLM Phase 1) — analyzes changes and produces a prioritized test plan with severity levels
5. Test Generation (LLM Phase 2) — generates Maestro YAML or Espresso Kotlin for each planned test
6. Post-Processing — normalizes output, injects animation waits (Maestro) or missing imports (Espresso), applies optional features
7. Fixture Extraction — if API mocks exceed threshold, extracts into shared fixture files
8. Commit/PR — writes tests and creates a PR or commits directly

Running Tests Locally

# Maestro
maestro test .maestro/generated

# Espresso
./gradlew connectedAndroidTest

Development

npm install
npm test        # 159 tests
npm run lint    # ESLint
npm run build   # ncc bundle to dist/

License

MIT