Yizhou Liu1, Zhengxi Lu1, Fei Tang1, Yiwen Qiu1, Yong Du1, Tongbo Chen1, Bofan Chen1, Tao Jiang2,
Wenhao Xu2, Yunfeng Shu1, Wenqi Zhang1, Weiming Lu1, Jun Xiao1, Yueting Zhuang1, Yongliang Shen1,β
1Zhejiang University, Β 2Ant Group
β Corresponding Author
Overview Β· Key Features Β· Metrics Β· Installation Β· Quick Start Β· Custom Methods Β· Citation
MemU-Bench is a benchmark for evaluating how well autonomous GUI agents can leverage user history (memory) to complete ambiguous tasks on Android devices. MemU-Bench introduces a memory suite where task instructions contain ambiguous references that can only be resolved by recalling the user's past interactions.
Related benchmark. KnowU-Bench (project page, paper) evaluates whether mobile agents know the user β inferring preferences, asking for clarification and acting proactively β while MemU-Bench evaluates whether they can recall the user's long interaction history.
- Nov 2026 β Project page released: https://zjureal.com/MemU-Bench/.
- Oct 2026 β Paper released on arXiv.
- π§ Memory-driven tasks: Tasks are intentionally ambiguous β agents must retrieve relevant information from user history to succeed
- π± User interaction trajectories: Includes real user operation logs (screenshots + actions) as history context, with 118 tasks divided into short (71 tasks, β€400 steps) and long (47 tasks, β₯700 steps) based on history length
- π Triple evaluation metrics: SR (Success Rate), SPL (Success weighted by Path Length), and IE (Interaction Efficiency)
| Metric | Description |
|---|---|
| SR | Success Rate |
| SPL | Success weighted by Path Length β penalizes inefficient solutions |
| IE | Interaction Efficiency β measures how efficiently the agent interacts with the environment |
βββ batch_restore_and_snapshot.py # Emulator initialization & snapshot creation
βββ move_data/ # Data restoration scripts for emulator setup
βββ run_memubench.py # Main evaluation runner
βββ apks/ # App APK files for emulator setup
βββ user_database/ # Per-user app data for emulator initialization
βββ user_history/ # User operation logs (screenshots + processed.jsonl)
βββ memory_generation/
β βββ generate_tasks_map.py # Generate user β task goal mapping
β βββ tasks_map.json # Pre-generated task mapping
β βββ memory/
β βββ generate_memory.py # Generate memory from operation logs (AppMem)
βββ retrieve_information/
β βββ retrieve_information.py # Retrieve relevant info for each task
βββ eval/
β βββ eval_checkpoint.py # Extract checkpoint results to JSON
β βββ eval_metrics.py # Calculate SR, SPL, IE from two JSON files
βββ eval_results/
β βββ human.json # Pre-generated human reference results
βββ android_world/ # Core environment (based on AndroidWorld)
β βββ agents/ # Agent implementations
β βββ task_evals/fixed_tasks/ # Task definitions per user/history-length
β βββ env/ # Android environment & ADB utilities
-
Set up the Android Emulator
Download Android Studio and create an Android Virtual Device (AVD): select Pixel 6, System Image Tiramisu, API Level 33, and AVD name MemUBenchAvd. Emulator setup instructions can also be found in AndroidWorld.
-
Create a conda environment
conda create -n memu_bench python=3.11 conda activate memu_bench
-
Install dependencies
pip install -r requirements.txt python setup.py install
-
Install
ffmpeg(if not already installed)# macOS brew install ffmpeg
The full evaluation pipeline consists of the following steps:
Download the dataset from https://huggingface.co/datasets/vagrt/MemU-Bench-data and place it in the project root:
apks/β app APK files for emulator setupuser_database/β contains app data for each user, used for emulator initialization. The dataset is distributed asuser_*.tar.zstarchives; extract them in the project root before running the pipeline:Installmkdir -p user_database for archive in user_database/*.tar.zst; do zstd -dc "$archive" | tar -xf - -C user_database done
zstdfirst if it is unavailable (for example,brew install zstdon macOS). After extraction, the directory should containuser_database/user_0/throughuser_database/user_11/.user_history/β contains user operation logs (screenshots +processed.jsonl), used for memory generation and retrieval
Launch the emulator from the command line with the gRPC flag:
EMULATOR_NAME=MemUBenchAvd
~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554Run the batch restore script to install apps, restore user data, and create snapshots. This step only needs to be done once β the snapshots are reusable across multiple evaluation runs.
python batch_restore_and_snapshot.py --install-apps --apk-dir ./apksAfter this, the emulator will have snapshots like snapshot_0_short, snapshot_0_long, etc. Subsequent evaluations can load these snapshots directly without re-initializing.
Create a .env file in the project root specifying the model and API credentials:
MODEL=your_model_name
API_KEY=your-api-key
BASE_URL=your-base-urlThis configures the model used across the entire pipeline β memory generation, retrieval, and online execution will all use this model.
Additionally, update fixed.env with the judger and grounding model configurations. Expected models:
JUDGER_MODEL: Qwen3.5-35B-A3BGROUNDING_MODEL: Qwen3-VL-32B-Instruct
JUDGER_API_KEY=your-judger-api-key
JUDGER_BASE_URL=your-judger-base-url
JUDGER_MODEL=Qwen3.5-35B-A3B
VLM_JUDGER_API_KEY=your-judger-api-key
VLM_JUDGER_BASE_URL=your-judger-base-url
VLM_JUDGER_MODEL=Qwen3.5-35B-A3B
GROUNDING_API_KEY=your-grounding-api-key
GROUNDING_BASE_URL=your-grounding-base-url
GROUNDING_MODEL=Qwen3-VL-32B-InstructThis step generates structured memory from raw user operation logs, then retrieves relevant information for each task.
Generate task mapping (if memory_generation/tasks_map.json does not exist, run this to generate it):
python memory_generation/generate_tasks_map.pyGenerate memory from user operation logs:
python memory_generation/memory/generate_memory.py \
--session_paths ./user_history \
--output_path ./user_information/memory/${MODEL_NAME}.json \
--env_file .envRetrieve relevant information from memory for each task:
python retrieve_information/retrieve_information.py \
--method memory \
--path ./user_information/memory/${MODEL_NAME}.json \
--output ./relevant_information/memory/${MODEL_NAME}.json \
--log_path ./retrieve_log/memory/${MODEL_NAME}.txt \
--env_file .envRun the agent on the memory suite tasks, using the retrieved information:
python run_memubench.py \
--suite_family memory_suite \
--checkpoint_dir ./checkpoint_results/${MODEL_NAME}/memory/run1 \
--method with_history \
--user_information_path ./relevant_information/memory/${MODEL_NAME}.json \
--eval_setting unclear_goal \
--env_file .envFirst, extract checkpoint results into a JSON file:
python eval/eval_checkpoint.py \
--checkpoint_dir ./checkpoint_results/${MODEL_NAME}/memory/run1 \
--output ./eval_results/${MODEL_NAME}_memory.jsonThen, calculate SR, SPL, IE by comparing with the provided human reference:
python eval/eval_metrics.py \
--ref ./eval_results/human.json \
--eval ./eval_results/${MODEL_NAME}_memory.json \
--output ./metric_results/memory.jsonMemU-Bench is designed to be modular β you can plug in your own memory processing and retrieval methods while reusing the rest of the benchmark pipeline.
You can replace memory_generation/memory/generate_memory.py with your own script to process user operation logs. The output format is flexible and can be whatever structure your method requires.
To use your own retrieval method, implement a retrieval module in retrieve_information/ (e.g., retrieve_information/your_method.py) with a get_filtered_history(goal, path, ...) function, then register it in retrieve_information/__init__.py by adding a new elif branch for your method name. When running retrieval, specify your method name via --method:
python retrieve_information/retrieve_information.py \
--method your_own_method \
--path /path/to/your/generated/data \
--output ./relevant_information/your_own_method/${MODEL_NAME}.json \
--env_file .envThen in online evaluation, simply use --method with_history and point --user_information_path to your retrieval output:
python run_memubench.py \
--suite_family memory_suite \
--method with_history \
--user_information_path ./relevant_information/your_own_method/${MODEL_NAME}.json \
--eval_setting unclear_goal \
--env_file .envThe only requirement is that the output JSON follows this format, since downstream task execution reads the retrieved information from it:
{
"results": [
{
"goal": "the ambiguous task instruction",
"subdir": "short",
"user_history": "the retrieved information for this task"
}
]
}As long as the retrieval output format is consistent, the downstream steps (online evaluation, metric calculation) remain unchanged.
If you find MemU-Bench useful, please cite:
@article{liu2026memubench,
title = {MemU-Bench: Benchmarking GUI Agents on Long-Horizon User Interactions},
author = {Liu, Yizhou and Lu, Zhengxi and Tang, Fei and Qiu, Yiwen and Du, Yong and Chen, Tongbo and Chen, Bofan and Jiang, Tao and Xu, Wenhao and Shu, Yunfeng and Zhang, Wenqi and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
journal = {arXiv preprint arXiv:2611.00191},
year = {2026}
}
@misc{chen2026knowubench,
title={KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation},
author={Tongbo Chen and Zhengxi Lu and Zhan Xu and Guocheng Shao and Shaohan Zhao and Fei Tang and Yong Du and Kaitao Song and Yizhou Liu and Yuchen Yan and Wenqi Zhang and Xu Tan and Weiming Lu and Jun Xiao and Yueting Zhuang and Yongliang Shen},
year={2026},
eprint={2604.08455},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2604.08455},
}This codebase is built upon AndroidWorld. For emulator configuration details, see the AndroidWorld documentation.


