-
Notifications
You must be signed in to change notification settings - Fork 35
Add core/mobile-ux-primitives, learn-from-tutorial, debugging, blockers, the curator, and a local/ overlay #15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
dcadb8f
e8244a4
99a7d01
db6b2fd
1bce476
0b6354a
6ffa161
92dbecf
2e0c199
80d6e20
4360085
35e48db
740f508
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,16 +15,18 @@ Do not import local drivers directly for ordinary agent work. | |
| dependencies. Agents should still import only `mobilerun_core`. | ||
| Skip the pip step if the libraries are pinned. If offline, continue with the current version. On any other failure, read `UPDATE.md`. | ||
| 2. Decide the target platform before acting. | ||
| 3. For Android work, read `platforms/android/GUIDE.md`. | ||
| 4. For iOS work, read `platforms/ios/GUIDE.md`. | ||
| 5. Do not load all files. | ||
| 6. When the foreground app id is known, read only that app card if it exists: | ||
| 3. Read `core/mobile-ux-primitives/GUIDE.md` before observing an unfamiliar screen. It applies to both platforms and belongs above the platform split, not inside it — read it once you know a screen is coming, before the platform guide's own instructions. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Reading |
||
| 4. For Android work, read `platforms/android/GUIDE.md`. | ||
| 5. For iOS work, read `platforms/ios/GUIDE.md`. | ||
| 6. Do not load all files. | ||
| 7. When the foreground app id is known, read only that app card if it exists: | ||
| - Android: `apps/android/<package>/CARD.md` | ||
| - iOS: `apps/ios/<bundle-id>/CARD.md` | ||
| 7. Read platform recovery only after a control, setup, state, or connectivity failure. | ||
| 8. Read the credentials guide under `core/credentials` when a screen asks for login, API keys, OTP, 2FA, payment, passcode, or other secrets. | ||
| 9. Write to `credentials/<app-id>.md` only when the user explicitly asks for local credential files. | ||
| 10. Read `core/memory/GUIDE.md` before reading or writing files under `memory/`. | ||
| Then read the same path under `local/` if it exists (`local/apps/android/<package>/CARD.md`, `local/apps/ios/<bundle-id>/CARD.md`). That file is the user's own, and it wins wherever it disagrees with the shipped card. It may be the only card that exists — the user's private or internal apps live there. Read `local/README.md` before writing anything under `local/`. | ||
| 8. Read platform recovery only after a connectivity, setup, or state-extraction failure. For an in-app action that didn't produce the expected result, or a dialog/permission prompt covering the screen, read `core/debugging/GUIDE.md` or `core/blockers/GUIDE.md` first — those are not connectivity problems. | ||
| 9. Read the credentials guide under `core/credentials` when a screen asks for login, API keys, OTP, 2FA, payment, passcode, or other secrets. | ||
| 10. Write to `credentials/<app-id>.md` only when the user explicitly asks for local credential files. | ||
| 11. Read `core/memory/GUIDE.md` before reading or writing files under `memory/`. | ||
|
|
||
| ## Non-Negotiables | ||
|
|
||
|
|
@@ -37,6 +39,7 @@ Do not import local drivers directly for ordinary agent work. | |
| - Stop on credentials, payment, or destructive consent. Continue only if the user explicitly authorized the exact action; otherwise ask the user. | ||
| - Store durable operational facts or useful information for the subsequent runs in `memory/` only after reading `core/memory/GUIDE.md`. | ||
| - Store credentials in `credentials/` only if the user explicitly asks for local credential files. | ||
| - Treat `local/` as user-owned and authoritative: it outranks the tracked file it mirrors. Write there only when the user asks for a local customization, and never move its content into a tracked file without asking — it is deliberately not shared. | ||
|
|
||
| ## Platform Routing | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -127,12 +127,16 @@ device.start_app("com.android.settings") | |
| Skill-based runtimes can load `SKILL.md`; all runtimes should start with | ||
| `AGENTS.md`. It routes agents to the smallest needed file: | ||
|
|
||
| - `core/mobile-ux-primitives/GUIDE.md` before observing an unfamiliar screen — cross-platform, read before the platform split. | ||
| - `platforms/android/GUIDE.md` for Android work. | ||
| - `platforms/ios/GUIDE.md` for iOS work. | ||
| - `platforms/<platform>/recovery/GUIDE.md` only when control fails. | ||
| - `core/debugging/GUIDE.md` or `core/blockers/GUIDE.md` for an in-app action failure or a dialog covering the screen. | ||
| - `platforms/<platform>/recovery/GUIDE.md` only when a connectivity/setup/state-extraction failure occurs. | ||
| - the credentials guide under `core/credentials` only when a credential or human-gated screen appears. | ||
| - `core/memory/GUIDE.md` only when reading or writing local agent-owned memory. | ||
| - `core/learn-from-tutorial/GUIDE.md` when the current screen turns out to be the app's own tutorial or onboarding walkthrough. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove this line |
||
| - `apps/android/<package>/CARD.md` or `apps/ios/<bundle-id>/CARD.md` only for the foreground app. | ||
| - the same path under `local/` after any tracked file it loads — your own copy, which wins on conflict. See [Customising Cards Without Merge Conflicts](#customising-cards-without-merge-conflicts). | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove this line |
||
| - `UPDATE.md` only when the session-start `git pull --ff-only` fails. | ||
|
|
||
|
|
||
|
|
@@ -172,4 +176,33 @@ two apart. | |
|
|
||
| ## Local State | ||
|
|
||
| `memory/` and `credentials/` are local, ignored folders. The repository tracks only their rules/templates. Agents may write operational memory after reading `core/memory/GUIDE.md`. | ||
| `local/`, `memory/`, and `credentials/` are local, ignored folders. The repository tracks only their rules/templates. | ||
|
|
||
| | Folder | Written by | Weight | | ||
| | --- | --- | --- | | ||
| | `local/` | you | authoritative — the agent obeys it and never shares it | | ||
| | `memory/` | the agent, after reading `core/memory/GUIDE.md` | provisional — re-verified before use | | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All these problems are solved if you add memory/ into local/ |
||
| | `credentials/` | you, and only if you ask for local credential files | secrets; see the guide under `core/credentials` | | ||
|
|
||
| ## Customising Cards Without Merge Conflicts | ||
|
|
||
| Session start runs `git pull --ff-only`, so editing a tracked file breaks your | ||
| next update. Put your version under `local/` at the same path instead: | ||
|
|
||
| ```text | ||
| apps/android/com.google.android.gm/CARD.md # shipped, tracked | ||
| local/apps/android/com.google.android.gm/CARD.md # yours, wins on conflict | ||
| local/apps/android/com.acme.internal/CARD.md # yours only — private/internal apps | ||
| ``` | ||
|
|
||
| The agent reads the shipped card first, then yours, and yours wins where the | ||
| two disagree. If only yours exists, it simply is the card — which is where | ||
| internal builds and private apps belong. | ||
|
|
||
| `local/` is gitignored except its README, so the pull keeps fast-forwarding | ||
| even when upstream changes a card you have overridden. Cards are found by path, | ||
| so there is no index to update. `scripts/curate.py` does not read `local/`, | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove it |
||
| so nothing personal leaks into a shared promotion. | ||
|
|
||
| Full details in `local/README.md`. Note that `git clean -xdf` deletes ignored | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is not related to mobile-harness, remove it |
||
| files, `local/` included. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,8 +17,19 @@ Use this card only when Gmail is the foreground package or the task explicitly t | |
| - After launching, wait for inbox or account picker before acting. | ||
| - If Gmail asks to add an account, sign in, or verify identity, stop and read `core/credentials/GUIDE.md`. | ||
|
|
||
| ## Compose | ||
|
|
||
| - `compose_button` opens the composer. Fields: `peoplekit_autocomplete_chip_group` (To), `subject`, `editor` (body). Sent recipients show as `peoplekit_chip` buttons. | ||
| - The body `editor` is **not clickable** — tap `composearea_tap_trap_bottom` to focus it. That places the caret at position 0, so `key('delete')` there is a no-op; tap directly on the text to edit the end. | ||
| - Add recipients one at a time: re-resolve the empty `EditText` inside the chip group each time (it moves as chips wrap), tap it, type the address, then type `,` to chip it. | ||
| - Navigate up (`Navigate up` in `compose_toolbar`) saves the draft; verify it under drawer → `Drafts`. | ||
|
|
||
| ## Traps | ||
|
|
||
| - **`type()` goes to whatever field is actually focused.** If the body is not focused, an entire body silently appends to the subject with no error. Read the destination field back after every write. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove it. It is not related to gm but a general skill in mobilerun-core-local |
||
| - **`clear_input()` does not clear the body** (rich text) and has been seen to clear the *subject* while `editor` reported `is_focused=True`. Do not trust it to target the field you think you are in. | ||
| - A contacts-permission dialog (`Allow Gmail to access your contacts?`) and a `Help me write` smart-features bottom sheet can appear mid-typing and swallow keystrokes. Decline both (`DON'T ALLOW`, `No thanks`) — neither is needed to compose — then re-verify what was typed. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove this line. User can need it |
||
| - Tapping a fixed coordinate in the To row hits an existing chip and opens a contact sheet or chip popup menu (`Remove the recipient`); `key('back')` backs out. | ||
| - Inbox rows can have repeated text; verify the opened message subject after tapping. | ||
| - Search results can lag. Observe again before acting on the first result. | ||
| - Do not store email contents in memory unless the user explicitly asks. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Instagram Card | ||
|
|
||
| Package: `com.instagram.android` | ||
|
|
||
| Use this card only when Instagram is the foreground package or the task explicitly targets Instagram. | ||
|
|
||
| ## Useful Labels | ||
|
|
||
| - Home, Search, Reels, and Profile are often the bottom nav tabs. | ||
| - The heart icon under a post toggles like state; a filled/colored heart means already liked. | ||
| - The paper-plane icon opens the share sheet for a post. | ||
|
|
||
| ## Flow Notes | ||
|
|
||
| - The home feed loads more posts automatically near the bottom of the scroll; there's no "load more" button — keep scrolling until new posts stop appearing. <!-- generalizable: infinite-scroll-no-pagination --> | ||
| - Double-tapping a post image likes it — equivalent to tapping the heart once from an unliked state. | ||
| - Stories, if present, are a horizontally scrollable row above the feed, distinct from the vertically scrolling feed below. | ||
|
|
||
| ## Traps | ||
|
|
||
| - Double-tapping an already-liked post does not unlike it — only the heart icon reliably toggles both directions. | ||
| - If Instagram asks to log in, verify a code, or confirm a phone number, stop and read `core/credentials`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Reddit Card | ||
|
|
||
| Package: `com.reddit.frontpage` | ||
|
|
||
| Use this card only when Reddit is the foreground package or the task explicitly targets Reddit. | ||
|
|
||
| ## Useful Labels | ||
|
|
||
| - Up/down arrows next to a post or comment are vote controls; the count between them is net score, not a rating. | ||
| - A top-left menu icon often opens community/navigation options; a magnifying glass opens search. | ||
|
|
||
| ## Flow Notes | ||
|
|
||
| - Subreddit and post feeds auto-load additional content near the bottom of the scroll; there is no numbered pagination control. <!-- generalizable: infinite-scroll-no-pagination --> | ||
| - Tapping an active vote arrow again returns it to neutral rather than flipping straight to the opposite vote — expect two taps to reverse a vote. | ||
| - Comment threads nest by indentation; a "N more replies" control often replaces a fully expanded thread. | ||
|
|
||
| ## Traps | ||
|
|
||
| - Vote counts can lag briefly after tapping; re-observe rather than assuming the tap failed if the score doesn't change instantly. | ||
| - If Reddit asks to log in or verify an account, stop and read `core/credentials`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,3 +8,12 @@ apps/ios/<bundle-id>/CARD.md | |
| ``` | ||
|
|
||
| Cards are plain Markdown, not `SKILL.md`, so generic agents do not auto-load every app. Each card should stay focused on stable app-specific facts: package or bundle id, useful selectors, common flows, navigation structure and traps. | ||
|
|
||
| Cards here are tracked and shared. A user's own card goes at the same path under `local/`, which is gitignored: | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why we have local/, memory/ and credentials/ all gitignored? Put them all under local/ |
||
|
|
||
| ```text | ||
| local/apps/android/<package>/CARD.md | ||
| local/apps/ios/<bundle-id>/CARD.md | ||
| ``` | ||
|
|
||
| Read the tracked card first, then the `local/` one; the `local/` one wins where they disagree, and may be the only one that exists. There is nothing to add to this file for either — cards are discovered by path, which is what keeps a local card from ever conflicting with a `git pull`. See `local/README.md`. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We have recovery and we have blockers, why? |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| --- | ||
| name: blockers | ||
| description: Something is covering the screen and blocking progress — an OS runtime-permission prompt, an app-not-responding or update/rating nag, or an unrecognized modal. Classify what's actually there before treating a stalled task as a dead end, grant only a permission the task explicitly needs, and always ask the user before anything privacy-sensitive. Never scroll through or blind-back() out of a dialog you haven't identified. | ||
| --- | ||
|
|
||
| # Blockers — clear the safe ones, never guess the rest | ||
|
|
||
| A task stalling — a tap landing nowhere, an expected element missing, a | ||
| scroll doing nothing — is usually **not** a dead end. It's often a dialog on | ||
| top of the screen you're not accounting for. Before treating it as a | ||
| selector or navigation failure (see `core/debugging`), check whether | ||
| something is actually blocking the view. | ||
|
|
||
| ## When to check | ||
|
|
||
| Check the moment a step stops making progress, and proactively right after | ||
| launching an app or taking an action that commonly triggers a system prompt | ||
| (camera, location, first post or send, notifications). Don't keep tapping | ||
| into something you haven't identified — that's how a permission dialog | ||
| turns into several wasted, silently-failing actions. | ||
|
|
||
| Read `device.ui()` (or `find_nodes`) and look at what's actually on screen | ||
| before deciding what kind of blocker this is. | ||
|
|
||
| ## Classifying what's there | ||
|
|
||
| | Kind | What it looks like | What you do | | ||
| |---|---|---| | ||
| | `nag` | "App isn't responding" (ANR), an in-app review prompt, an update nag | Dismiss it (the safe default action — usually "Not now"/"Later"/close) and re-check the screen; this isn't a real obstacle. | | ||
| | `unknown_modal` | A modal you don't recognize and can't confidently classify | Tap an explicit **Close/X** if one is visibly present. If there isn't one, **stop and tell the user** what's on screen — never scroll it or blind-`back()` out of it. | | ||
| | `permission_grantable` | Camera / microphone / storage / media / notifications / calendar, etc. | Grant it **only if the user's actual task explicitly needs it** (a scan task justifies camera). Tap "While using the app"/"Allow". If the need isn't obvious from the task, treat it like a sensitive scope below — ask instead of guessing. | | ||
| | `permission_sensitive` | **Contacts / SMS / call log / location** | **Always ask the user first — never auto-grant, regardless of the task.** This is a hard floor, not a judgment call. | | ||
|
|
||
| ## The one judgment call: `permission_grantable` | ||
|
|
||
| Decide only against what the user actually asked for, not what would be | ||
| convenient. "Scan the QR code" justifies camera; it does not justify | ||
| location. If the link between the permission and the stated task isn't | ||
| obvious, treat it as sensitive: ask one short question and wait — offer | ||
| concrete options (grant / deny / let the user handle it on the device) — | ||
| the same pattern `core/credentials` uses for anything gated. Granting a | ||
| permission is consent on the user's device; when in doubt, ask rather than | ||
| assume. | ||
|
|
||
| ## Never | ||
|
|
||
| - Never scroll a modal (it typically won't respond the way a normal screen | ||
| does) or `back()` out of one you haven't identified — you can dismiss the | ||
| wrong thing or leave the flow entirely without realizing it. | ||
| - Never tap "Allow" on a sensitive scope, or on a permission the task didn't | ||
| actually call for. | ||
| - Never treat a permission prompt or ANR as a task failure in itself — clear | ||
| it (or surface it to the user) and continue; report failure only if the | ||
| underlying task still can't proceed afterward. | ||
|
|
||
| Related: `core/credentials` (anything beyond a runtime permission — login, | ||
| payment, OTP, consent), `core/debugging` (what to do when the *cause* of a | ||
| stall turns out not to be a blocker after all). |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I already said, or add memory/ into local or remove local/. In general, I agree that scripts/ and tests/ could be removed from .gitignore but don't add same script/test files you added in previous commits