Skip to content
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions .gitignore

@RasulOs RasulOs Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I already said, or add memory/ into local or remove local/. In general, I agree that scripts/ and tests/ could be removed from .gitignore but don't add same script/test files you added in previous commits

Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,21 @@ credentials/**
!credentials/README.md
!credentials/TEMPLATE.md

# User-owned local overlay. Mirrors a tracked path to override or extend it
# without ever touching a tracked file, so `git pull --ff-only` keeps working.
local/**
!local/
!local/README.md

tmp/

# Local QA helpers and experiments. The harness product is Markdown-only.
scripts/
tests/
# Curator reports are regenerable. Run scripts/curate.py to recreate them.
.curator/

# Secrets, just in case.
.env

# scripts/ and tests/ are tracked: the curator and its test suite ship with the
# harness. Only their build artifacts are ignored.
scripts/__pycache__/
tests/__pycache__/
19 changes: 11 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,18 @@ Do not import local drivers directly for ordinary agent work.
dependencies. Agents should still import only `mobilerun_core`.
Skip the pip step if the libraries are pinned. If offline, continue with the current version. On any other failure, read `UPDATE.md`.
2. Decide the target platform before acting.
3. For Android work, read `platforms/android/GUIDE.md`.
4. For iOS work, read `platforms/ios/GUIDE.md`.
5. Do not load all files.
6. When the foreground app id is known, read only that app card if it exists:
3. Read `core/mobile-ux-primitives/GUIDE.md` before observing an unfamiliar screen. It applies to both platforms and belongs above the platform split, not inside it — read it once you know a screen is coming, before the platform guide's own instructions.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading core/mobile-ux-primitives/GUIDE.md before observing an unfamiliar screen will make agent much slower. Remove

4. For Android work, read `platforms/android/GUIDE.md`.
5. For iOS work, read `platforms/ios/GUIDE.md`.
6. Do not load all files.
7. When the foreground app id is known, read only that app card if it exists:
- Android: `apps/android/<package>/CARD.md`
- iOS: `apps/ios/<bundle-id>/CARD.md`
7. Read platform recovery only after a control, setup, state, or connectivity failure.
8. Read the credentials guide under `core/credentials` when a screen asks for login, API keys, OTP, 2FA, payment, passcode, or other secrets.
9. Write to `credentials/<app-id>.md` only when the user explicitly asks for local credential files.
10. Read `core/memory/GUIDE.md` before reading or writing files under `memory/`.
Then read the same path under `local/` if it exists (`local/apps/android/<package>/CARD.md`, `local/apps/ios/<bundle-id>/CARD.md`). That file is the user's own, and it wins wherever it disagrees with the shipped card. It may be the only card that exists — the user's private or internal apps live there. Read `local/README.md` before writing anything under `local/`.
8. Read platform recovery only after a connectivity, setup, or state-extraction failure. For an in-app action that didn't produce the expected result, or a dialog/permission prompt covering the screen, read `core/debugging/GUIDE.md` or `core/blockers/GUIDE.md` first — those are not connectivity problems.
9. Read the credentials guide under `core/credentials` when a screen asks for login, API keys, OTP, 2FA, payment, passcode, or other secrets.
10. Write to `credentials/<app-id>.md` only when the user explicitly asks for local credential files.
11. Read `core/memory/GUIDE.md` before reading or writing files under `memory/`.

## Non-Negotiables

Expand All @@ -37,6 +39,7 @@ Do not import local drivers directly for ordinary agent work.
- Stop on credentials, payment, or destructive consent. Continue only if the user explicitly authorized the exact action; otherwise ask the user.
- Store durable operational facts or useful information for the subsequent runs in `memory/` only after reading `core/memory/GUIDE.md`.
- Store credentials in `credentials/` only if the user explicitly asks for local credential files.
- Treat `local/` as user-owned and authoritative: it outranks the tracked file it mirrors. Write there only when the user asks for a local customization, and never move its content into a tracked file without asking — it is deliberately not shared.

## Platform Routing

Expand Down
37 changes: 35 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,12 +127,16 @@ device.start_app("com.android.settings")
Skill-based runtimes can load `SKILL.md`; all runtimes should start with
`AGENTS.md`. It routes agents to the smallest needed file:

- `core/mobile-ux-primitives/GUIDE.md` before observing an unfamiliar screen — cross-platform, read before the platform split.
- `platforms/android/GUIDE.md` for Android work.
- `platforms/ios/GUIDE.md` for iOS work.
- `platforms/<platform>/recovery/GUIDE.md` only when control fails.
- `core/debugging/GUIDE.md` or `core/blockers/GUIDE.md` for an in-app action failure or a dialog covering the screen.
- `platforms/<platform>/recovery/GUIDE.md` only when a connectivity/setup/state-extraction failure occurs.
- the credentials guide under `core/credentials` only when a credential or human-gated screen appears.
- `core/memory/GUIDE.md` only when reading or writing local agent-owned memory.
- `core/learn-from-tutorial/GUIDE.md` when the current screen turns out to be the app's own tutorial or onboarding walkthrough.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line

- `apps/android/<package>/CARD.md` or `apps/ios/<bundle-id>/CARD.md` only for the foreground app.
- the same path under `local/` after any tracked file it loads — your own copy, which wins on conflict. See [Customising Cards Without Merge Conflicts](#customising-cards-without-merge-conflicts).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line

- `UPDATE.md` only when the session-start `git pull --ff-only` fails.


Expand Down Expand Up @@ -172,4 +176,33 @@ two apart.

## Local State

`memory/` and `credentials/` are local, ignored folders. The repository tracks only their rules/templates. Agents may write operational memory after reading `core/memory/GUIDE.md`.
`local/`, `memory/`, and `credentials/` are local, ignored folders. The repository tracks only their rules/templates.

| Folder | Written by | Weight |
| --- | --- | --- |
| `local/` | you | authoritative — the agent obeys it and never shares it |
| `memory/` | the agent, after reading `core/memory/GUIDE.md` | provisional — re-verified before use |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these problems are solved if you add memory/ into local/

| `credentials/` | you, and only if you ask for local credential files | secrets; see the guide under `core/credentials` |

## Customising Cards Without Merge Conflicts

Session start runs `git pull --ff-only`, so editing a tracked file breaks your
next update. Put your version under `local/` at the same path instead:

```text
apps/android/com.google.android.gm/CARD.md # shipped, tracked
local/apps/android/com.google.android.gm/CARD.md # yours, wins on conflict
local/apps/android/com.acme.internal/CARD.md # yours only — private/internal apps
```

The agent reads the shipped card first, then yours, and yours wins where the
two disagree. If only yours exists, it simply is the card — which is where
internal builds and private apps belong.

`local/` is gitignored except its README, so the pull keeps fast-forwarding
even when upstream changes a card you have overridden. Cards are found by path,
so there is no index to update. `scripts/curate.py` does not read `local/`,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove it

so nothing personal leaks into a shared promotion.

Full details in `local/README.md`. Note that `git clean -xdf` deletes ignored

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not related to mobile-harness, remove it

files, `local/` included.
14 changes: 10 additions & 4 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,15 @@ local backends.
## Load Order

1. Read `AGENTS.md`.
2. Read `platforms/android/GUIDE.md` for Android work.
3. Read `platforms/ios/GUIDE.md` for iOS work.
4. Read recovery, credentials, memory, and app-card files only when routed
there by `AGENTS.md` or the platform guide.
2. Read `core/mobile-ux-primitives/GUIDE.md` before observing an unfamiliar screen — cross-platform, load it before the platform split below.
3. Read `platforms/android/GUIDE.md` for Android work.
4. Read `platforms/ios/GUIDE.md` for iOS work.
5. Read recovery, credentials, memory, and app-card files only when routed
there by `AGENTS.md` or the platform guide. For an in-app action failure or
a dialog/permission prompt (not a connectivity failure), that means
`core/debugging/GUIDE.md` or `core/blockers/GUIDE.md`.
6. After any tracked file you load, read the same path under `local/` if it
exists. That is the user's own copy and it wins on conflict. See
`local/README.md`.

For setup and runtime registration, read `install.md`.
8 changes: 8 additions & 0 deletions apps/android/com.ebay.mobile/CARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,19 @@ targets eBay on Android.
- Prefer `find_nodes`, `tap_node`, and resource ids over fixed coordinates.
- Verify the result sort before collecting data; visually plausible results can
still be sorted by a default ranking.
- Results use infinite scroll. Keep scrolling until the collected count stops
growing; there is no next-page control. <!-- generalizable: infinite-scroll-no-pagination -->

## Traps

- Sort and filter bottom sheets can be delayed in the accessibility tree after
opening. Re-observe before deciding an option is absent.
- The active sort option is often omitted from the sort sheet, which lists only
the other choices. Absence can mean "already selected", not "not available".
- eBay can resume on a previous screen. Confirm the current page before
searching, sorting, or scraping.
- Search result cards can repeat similar titles. Verify the intended listing
opened after tapping.
- Currency and marketplace depend on the device/account storefront.
- If eBay asks to sign in, add a payment method, or verify identity, stop and
read `core/credentials/GUIDE.md`.
11 changes: 11 additions & 0 deletions apps/android/com.google.android.gm/CARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,19 @@ Use this card only when Gmail is the foreground package or the task explicitly t
- After launching, wait for inbox or account picker before acting.
- If Gmail asks to add an account, sign in, or verify identity, stop and read `core/credentials/GUIDE.md`.

## Compose

- `compose_button` opens the composer. Fields: `peoplekit_autocomplete_chip_group` (To), `subject`, `editor` (body). Sent recipients show as `peoplekit_chip` buttons.
- The body `editor` is **not clickable** — tap `composearea_tap_trap_bottom` to focus it. That places the caret at position 0, so `key('delete')` there is a no-op; tap directly on the text to edit the end.
- Add recipients one at a time: re-resolve the empty `EditText` inside the chip group each time (it moves as chips wrap), tap it, type the address, then type `,` to chip it.
- Navigate up (`Navigate up` in `compose_toolbar`) saves the draft; verify it under drawer → `Drafts`.

## Traps

- **`type()` goes to whatever field is actually focused.** If the body is not focused, an entire body silently appends to the subject with no error. Read the destination field back after every write.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove it. It is not related to gm but a general skill in mobilerun-core-local

- **`clear_input()` does not clear the body** (rich text) and has been seen to clear the *subject* while `editor` reported `is_focused=True`. Do not trust it to target the field you think you are in.
- A contacts-permission dialog (`Allow Gmail to access your contacts?`) and a `Help me write` smart-features bottom sheet can appear mid-typing and swallow keystrokes. Decline both (`DON'T ALLOW`, `No thanks`) — neither is needed to compose — then re-verify what was typed.

@RasulOs RasulOs Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line. User can need it

- Tapping a fixed coordinate in the To row hits an existing chip and opens a contact sheet or chip popup menu (`Remove the recipient`); `key('back')` backs out.
- Inbox rows can have repeated text; verify the opened message subject after tapping.
- Search results can lag. Observe again before acting on the first result.
- Do not store email contents in memory unless the user explicitly asks.
22 changes: 22 additions & 0 deletions apps/android/com.instagram.android/CARD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Instagram Card

Package: `com.instagram.android`

Use this card only when Instagram is the foreground package or the task explicitly targets Instagram.

## Useful Labels

- Home, Search, Reels, and Profile are often the bottom nav tabs.
- The heart icon under a post toggles like state; a filled/colored heart means already liked.
- The paper-plane icon opens the share sheet for a post.

## Flow Notes

- The home feed loads more posts automatically near the bottom of the scroll; there's no "load more" button — keep scrolling until new posts stop appearing. <!-- generalizable: infinite-scroll-no-pagination -->
- Double-tapping a post image likes it — equivalent to tapping the heart once from an unliked state.
- Stories, if present, are a horizontally scrollable row above the feed, distinct from the vertically scrolling feed below.

## Traps

- Double-tapping an already-liked post does not unlike it — only the heart icon reliably toggles both directions.
- If Instagram asks to log in, verify a code, or confirm a phone number, stop and read `core/credentials`.
21 changes: 21 additions & 0 deletions apps/android/com.reddit.frontpage/CARD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Reddit Card

Package: `com.reddit.frontpage`

Use this card only when Reddit is the foreground package or the task explicitly targets Reddit.

## Useful Labels

- Up/down arrows next to a post or comment are vote controls; the count between them is net score, not a rating.
- A top-left menu icon often opens community/navigation options; a magnifying glass opens search.

## Flow Notes

- Subreddit and post feeds auto-load additional content near the bottom of the scroll; there is no numbered pagination control. <!-- generalizable: infinite-scroll-no-pagination -->
- Tapping an active vote arrow again returns it to neutral rather than flipping straight to the opposite vote — expect two taps to reverse a vote.
- Comment threads nest by indentation; a "N more replies" control often replaces a fully expanded thread.

## Traps

- Vote counts can lag briefly after tapping; re-observe rather than assuming the tap failed if the score doesn't change instantly.
- If Reddit asks to log in or verify an account, stop and read `core/credentials`.
9 changes: 9 additions & 0 deletions apps/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,12 @@ apps/ios/<bundle-id>/CARD.md
```

Cards are plain Markdown, not `SKILL.md`, so generic agents do not auto-load every app. Each card should stay focused on stable app-specific facts: package or bundle id, useful selectors, common flows, navigation structure and traps.

Cards here are tracked and shared. A user's own card goes at the same path under `local/`, which is gitignored:

@RasulOs RasulOs Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we have local/, memory/ and credentials/ all gitignored? Put them all under local/


```text
local/apps/android/<package>/CARD.md
local/apps/ios/<bundle-id>/CARD.md
```

Read the tracked card first, then the `local/` one; the `local/` one wins where they disagree, and may be the only one that exists. There is nothing to add to this file for either — cards are discovered by path, which is what keeps a local card from ever conflicting with a `git pull`. See `local/README.md`.
58 changes: 58 additions & 0 deletions core/blockers/GUIDE.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have recovery and we have blockers, why?

Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
name: blockers
description: Something is covering the screen and blocking progress — an OS runtime-permission prompt, an app-not-responding or update/rating nag, or an unrecognized modal. Classify what's actually there before treating a stalled task as a dead end, grant only a permission the task explicitly needs, and always ask the user before anything privacy-sensitive. Never scroll through or blind-back() out of a dialog you haven't identified.
---

# Blockers — clear the safe ones, never guess the rest

A task stalling — a tap landing nowhere, an expected element missing, a
scroll doing nothing — is usually **not** a dead end. It's often a dialog on
top of the screen you're not accounting for. Before treating it as a
selector or navigation failure (see `core/debugging`), check whether
something is actually blocking the view.

## When to check

Check the moment a step stops making progress, and proactively right after
launching an app or taking an action that commonly triggers a system prompt
(camera, location, first post or send, notifications). Don't keep tapping
into something you haven't identified — that's how a permission dialog
turns into several wasted, silently-failing actions.

Read `device.ui()` (or `find_nodes`) and look at what's actually on screen
before deciding what kind of blocker this is.

## Classifying what's there

| Kind | What it looks like | What you do |
|---|---|---|
| `nag` | "App isn't responding" (ANR), an in-app review prompt, an update nag | Dismiss it (the safe default action — usually "Not now"/"Later"/close) and re-check the screen; this isn't a real obstacle. |
| `unknown_modal` | A modal you don't recognize and can't confidently classify | Tap an explicit **Close/X** if one is visibly present. If there isn't one, **stop and tell the user** what's on screen — never scroll it or blind-`back()` out of it. |
| `permission_grantable` | Camera / microphone / storage / media / notifications / calendar, etc. | Grant it **only if the user's actual task explicitly needs it** (a scan task justifies camera). Tap "While using the app"/"Allow". If the need isn't obvious from the task, treat it like a sensitive scope below — ask instead of guessing. |
| `permission_sensitive` | **Contacts / SMS / call log / location** | **Always ask the user first — never auto-grant, regardless of the task.** This is a hard floor, not a judgment call. |

## The one judgment call: `permission_grantable`

Decide only against what the user actually asked for, not what would be
convenient. "Scan the QR code" justifies camera; it does not justify
location. If the link between the permission and the stated task isn't
obvious, treat it as sensitive: ask one short question and wait — offer
concrete options (grant / deny / let the user handle it on the device) —
the same pattern `core/credentials` uses for anything gated. Granting a
permission is consent on the user's device; when in doubt, ask rather than
assume.

## Never

- Never scroll a modal (it typically won't respond the way a normal screen
does) or `back()` out of one you haven't identified — you can dismiss the
wrong thing or leave the flow entirely without realizing it.
- Never tap "Allow" on a sensitive scope, or on a permission the task didn't
actually call for.
- Never treat a permission prompt or ANR as a task failure in itself — clear
it (or surface it to the user) and continue; report failure only if the
underlying task still can't proceed afterward.

Related: `core/credentials` (anything beyond a runtime permission — login,
payment, OTP, consent), `core/debugging` (what to do when the *cause* of a
stall turns out not to be a blocker after all).
Loading