Awesome Fine-Grained Multimodal Perception
A curated collection of the latest research and resources on Fine-Grained Multimodal Perception . This repository encompasses datasets, benchmarks, research papers, and practical methods for enhancing fine-grained perception capabilities in Multimodal Large Language Models (MLLMs).
🚀🚀🚀 Contributions are welcome! If you find any missing papers, datasets, or tools, feel free to open an issue or submit a pull request.
Date
News
2026-06-04
Added Attentive-CoT and Beyond Visual Memory to Training-Based Methods (SFT & Distillation)
2026-06-03
Updated iVGR venue to ICML 2026; Reorganized structure with What is, 📚 TOC, and time-based sorting
📜 View More Updates
What is Fine-Grained Multimodal Perception?
Multimodal Large Language Models (MLLMs) excel at broad visual understanding but still struggle with fine-grained perception , where decisive evidence is small and easily overwhelmed by global context. Recent advancements have shifted towards a "Thinking-with-Images" paradigm, where MLLMs actively acquire local information during inference rather than relying solely on a global image encoding.
This awesome list categorizes methods into four paradigms:
Data-Centric Methods : Improving perception through data quality, quantity, and synthetic generation
Training-Free Methods : Enhancing perception at inference time via prompting, chain-of-thought, and attention strategies
Agentic: Thinking with Images : Actively interacting with images through tools (zoom, crop, search, edit) during inference
Training-Based Methods : Improving perception through supervised fine-tuning, distillation, and reinforcement learning
⬆ Back to Top
Task Legend: FG: Fine-Grained Perception | OCR: Text Recognition | Cnt: Counting | Grd: Grounding | Rea: Reasoning
Benchmark
Paper
Venue & Year
Task
Highlights
Download
Preprints
GeoBrowse
GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces
Arxiv 2026
FG, Rea
Geolocation benchmark, agentic tool use, expert reasoning traces
GitHub
CountQA
CountQA: How Well Do MLLMs Count in the Wild?
Arxiv 2025
Cnt
Visual counting with reasoning
HuggingFace
CoCoT
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
Arxiv 2024
Grd
Visual grounding with reasoning
N/A
Rexverse
ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Arxiv 2024
FG
Fine-grained visual understanding
GitHub
2026
VisuRiddles
VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning
AAAI 2026
FG
Visual riddles, fine-grained perception evaluation
Github
GroundingME
GroundingME: Grounding Multi-Modal Evaluation
CVPR 2026
Grd
Multi-modal grounding evaluation
GitHub
RTV-Bench
RTV-Bench: Benchmarking MLLM Continuous Perception
ICLR 2026
FG, Rea
Real-time video analysis, 552 videos, 4,608 QA pairs, multi-timestamp reasoning
GitHub
SFE
Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning
ICLR 2026
FG, Rea
Scientific cognitive evaluation, three interconnected levels
GitHub
Thyme-SFT/RL
Thyme: Thinking with Images for Visual Reasoning
ICLR 2026
FG, Rea
Visual reasoning with tool use
GitHub
Visual Probe
Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
ICLR 2026
FG
Visual probing for perception
GitHub
TreeBench
Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology
ICLR 2026
Rea
Traceable reasoning evaluation
GitHub
ZoomBench
Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception
ICML 2026
FG, Rea
845 VQA samples, 6 perceptual dimensions, dual-view evaluation
GitHub
2025
HR-Bench
Divide and Conquer: A Comprehensive Benchmark for High-Resolution Image Understanding
AAAI 2025
FG
High-resolution image understanding
GitHub
MME-RealWorld
MME-RealWorld: Evaluating Real-World Perception in MLLMs
ICLR 2025
FG, Grd
Real-world perception benchmark
GitHub
ColorBench
ColorBench: Can VLMs See and Understand the Colorful World? A Benchmark for Color Perception
NeurIPS 2025
FG
Color perception evaluation
GitHub
2024
VStar
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
CVPR 2024
FG
Visual search, attribute recognition
GitHub
CV-Bench
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
CVPR 2024
FG, Rea
Computer vision-centric benchmark
HuggingFace
MMVP
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
CVPR 2024
FG
Visual shortcomings evaluation
GitHub
2017
Visual Genome
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
IJCV 2017
Grd, FG
Dense image annotations, grounding
HuggingFace
⬆ Back to Top
This category focuses on methods that improve fine-grained perception through data quality, quantity, and synthetic data generation pipelines.
⬆ Back to Top
This category covers inference-time approaches that enhance fine-grained perception without additional training, leveraging prompting strategies, chain-of-thought reasoning, and attention mechanisms.
Title
Venue & Year
Highlights/Keywords
Code
Preprints
Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
Arxiv 2026
Query-Aware, Adaptive Perception, Efficient Zoom
Github
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Arxiv 2026
Training-Free, Iterative, Visually-Grounded Reasoning
GitHub
SvfEye: A Semantic-Visual Fusion Framework with Multi-Scale Visual Context for Multimodal Reasoning
Arxiv 2026
Semantic-Visual Fusion, Multi-Scale Context
GitHub
Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
Arxiv 2025
Focusing Strategy, Chain Reasoning
GitHub
Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild
Arxiv 2025
Self-guided Reasoning, Socratic Method
GitHub
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
Arxiv 2024
Image-of-Thought, Reasoning Refinement
N/A
Chain of Thought Prompt Tuning in Vision Language Models
Arxiv 2023
Prompt Tuning, VLM CoT
N/A
Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings
Arxiv 2023
Multimodal Infillings, Visual CoT
GitHub
2026
ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
ICLR 2026
Efficient High-Resolution, Visual Understanding
GitHub
Hide: Rethinking the zoom-in method in high resolution mllms via hierarchical decoupling
ICML 2026
Attention-based Detection
GitHub
2025
ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Search
EMNLP 2025
Tree Search, Attention Mapping
GitHub
Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
ICML 2025
Spatial Visualization, Imaginative Reasoning
GitHub
2024
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
AAAI 2024
Knowledge Augmentation, CoT Reasoning
N/A
Visual Chain-of-Thought Prompting for Knowledge-based Visual Reasoning
AAAI 2024
Interactive Prompting, Knowledge-based Reasoning
N/A
TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding
ACM TOMM
Text-rich Images, Zoom-in Strategy
GitHub
Compositional Chain-of-Thought Prompting for Large Multimodal Models
CVPR 2024
Compositional Prompting
GitHub
Cantor: Inspiring Multimodal Chain-of-Thought of MLLM
MM 2024
CoT Inspiration, MLLM Reasoning
GitHub
Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
NeurIPS 2024
Visualization-of-Thought, Spatial Reasoning
GitHub
Multimodal Chain-of-Thought Reasoning in Language Models
TMLR 2024
Multimodal CoT, Two-stage Framework
GitHub
2023
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
NeurIPS 2023
Compositional Reasoning, Tool Integration
GitHub
⬆ Back to Top
Agentic: Thinking with Images
This category includes models that actively interact with images during inference using tools (zoom, crop, search, edit, sketch), enabling iterative visual exploration and dynamic focus on regions of interest. These methods reduce interference by isolating micro-crops but typically incur higher latency due to multi-pass inference.
Title
Venue & Year
Highlights/Keywords
Code
Preprints
S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images
Arxiv 2026
Scientific Reasoning, Thinking-with-Images
N/A
Test-time Scaling over Perception: Resolving the Grounding Paradox in Thinking with Images
Arxiv 2026
Test-time Scaling, Grounding Paradox
N/A
Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization
Arxiv 2026
Agentic Policy Optimization, Reasoning-Action Gap
N/A
CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
Arxiv 2025
Tool-integrated, Executable Visual Reasoning
GitHub
DeepEyesV2: Toward Agentic Multimodal Model
Arxiv 2025
Scaled Tool Learning, Visual Search
GitHub
DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning
Arxiv 2025
Visual Manipulation, Sketching, Internalized
GitHub
VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
Arxiv 2025
Region Refinement, Recursive Focusing
N/A
VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning
Arxiv 2025
Video Reward Models, Thinking-with-Image
GitHub
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
Arxiv 2024
Sketching, Visual Chain of Thought
GitHub
2026
Let's Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts
AAAI 2026
Interleaved-Modal CoT, Dynamic Visual Thoughts
GitHub
Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
CVPR 2026
Video Generation, Multimodal Reasoning Paradigm
GitHub
AGILE: Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in VLMs
ICLR 2026
Agentic Interaction, Jigsaw Puzzle, Visual Perception
GitHub
DeepEyes: Incentivizing "Thinking with Images" in Vision-Language Models via Reinforcement Learning
ICLR 2026
Reinforcement Learning, Zoom/Crop Tools, Long CoT
GitHub
Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
ICLR 2026
Visual Probing, Agentic Tools
GitHub
Thyme: Think Beyond Images
ICLR 2026
Pixel-space Operations, Code Generation
GitHub
Visual Planning: Let's Think Only with Images
ICLR 2026 Oral
Visual Planning, Image-only Reasoning
GitHub
2025
Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
CVPR 2025
Visual CoT, Attention Grounding
Project
ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding
ICML 2025
Visual Editing, Structured Understanding
GitHub
PixelReasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning
NeurIPS 2025
Pixel-level Operations, Dynamic Visual Manipulation
GitHub
2024
Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models
CVPR 2024
Interactive Spot Identification, Zoom-in Reasoning
GitHub
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
CVPR 2024
Visual Search, Search-based Perception
GitHub
⬆ Back to Top
This category covers methods that improve fine-grained perception through training procedures, including supervised fine-tuning, knowledge distillation, and reinforcement learning.
Supervised Fine-Tuning & Distillation
Title
Venue & Year
Highlights/Keywords
Code
Preprints
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Arxiv 2026
Attention-Guided CoT-SFT, Delayed Answer Commitment, Visual-Token Access
N/A
Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning
Arxiv 2026
Latent Token Decomposition, Boundary Markers, Mechanistic Analysis
N/A
LanteRn: Latent Visual Structured Reasoning
Arxiv 2026
Latent Visual Reasoning, Structured Reasoning
N/A
Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
Arxiv 2025
Image Generation CoT, Verification
GitHub
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?
Arxiv 2025
Long-CoT, Scaling
N/A
URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics
Arxiv 2025
Mathematical Reasoning, Verification
GitHub
AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning
Arxiv 2024
Slow Thinking, Mathematical Reasoning
GitHub
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models
Arxiv 2024
Efficient Fine-Grained Perception, Text-rich
GitHub
2026
Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology
ICLR 2026
Traceable Evidence, Visual Grounded Reasoning
GitHub
Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception
ICML 2026
Region-to-Image Distillation, ZoomBench, Single-Pass Perception
GitHub
2025
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
CVPR 2025
Long-Chain Reasoning, Insightful Analysis
GitHub
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
CVPR 2025
Perception Tokens, Visual Reasoning
Project
SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning
NeurIPS 2025
Depth Perception, Spatial Reasoning, Structured Rationales
GitHub
2024
Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition
ICML 2024
Video Reasoning, Perception to Cognition
GitHub
⬆ Back to Top
Title
Venue & Year
Highlights/Keywords
Code
2026
iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning
ICML 2026
Visually Grounded Reasoning, Internalization, RL
N/A
PAPO: Perception-Aware Policy Optimization for Multimodal Reasoning
ICLR 2026
PAPO, Perception-Aware RL, Implicit Perception Supervision
GitHub
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
ICLR 2026
Visual Perception Reward, RLVR, GRPO
GitHub
RAPID: Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
ICLR 2026
Perception-Decoupling, Reasoning Alignment, Two-stage Pipeline
GitHub
Spotlight on Token Perception for Multimodal Reinforcement Learning
ICLR 2026
Token Perception, Visual Dependency, RLVR Optimization
GitHub
ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Models
ICLR 2026
Self-Evolution, Fine-Grained Perception, RL
GitHub
VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning
ICLR 2026
Unified Perception-Reasoning, RL, Multi-task
GitHub
VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
ICLR 2026
Reinforcement Learning, Tool Use, Thinking-with-Images
GitHub
Preprints
Boosting Multimodal Reasoning with MCTS-Automated Structured Thinking
Arxiv 2025
MCTS, Structured Thinking
GitHub
Boosting the Generalization and Reasoning of VLMs with Curriculum Reinforcement Learning
Arxiv 2025
Curriculum RL, Generalization
N/A
LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
Arxiv 2025
Two-Stage RL, Rule-based
GitHub
MedVLM-R1: Incentivizing Medical Reasoning Capability of VLMs via Reinforcement Learning
Arxiv 2025
Medical Reasoning, RL
GitHub
MM-Eureka: Exploring Visual Aha Moment with Rule-based Large-scale Reinforcement Learning
Arxiv 2025
Visual Aha Moment, Rule-based RL
GitHub
OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning via Iterative Self-Improvement
Arxiv 2025
Self-Improvement, Iterative Training
GitHub
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
Arxiv 2025
Emotion Recognition, Explainable RL
GitHub
R1-OneVision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
Arxiv 2025
Cross-Modal, Generalized Reasoning
GitHub
Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement
Arxiv 2025
Segmentation, Cognitive Reinforcement
GitHub
Training Multi-Image Vision Agents via End2End Reinforcement Learning
Arxiv 2025
Multi-Image, End2End RL, Vision Agents
N/A
Virgo: A Preliminary Exploration on Reproducing o1-like MLLM
Arxiv 2025
O1-like MLLM, Reasoning Exploration
GitHub
VisualPRM: An Effective Process Reward Model for Multimodal Reasoning
Arxiv 2025
Process Reward Model
Project
VisualThinker-R1-Zero: R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
Arxiv 2025
R1-Zero, Visual Reasoning
GitHub
VisRL: Intention-Driven Visual Perception via Reinforced Reasoning
Arxiv 2025
Intention-Driven, Reinforced Reasoning
GitHub
2025
LlamaV-o1: Rethinking Step-by-Step Visual Reasoning in LLMs
ACL-Findings 2025
Visual Reasoning, O1-style
GitHub
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
ICCV 2025
GRPO, Step-wise Reasoning
GitHub
Diving into Self-Evolving Training for Multimodal Reasoning
ICML 2025
Self-Evolving, RL Training
GitHub
Mulberry: Empowering MLLM with O1-like Reasoning and Reflection via Collective Monte Carlo Tree Search
NeurIPS 2025 Spotlight
MCTS, O1-like Reasoning
GitHub
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
NeurIPS 2025
Visual Hallucination Critic, RL Proxy Task, Fine-Grained Perception
GitHub
⬆ Back to Top
This category covers application domains where fine-grained perception is critical.
Title
Venue & Year
Domain
Highlights/Keywords
Code
Preprints
EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs
Arxiv 2026
Emotion
Omni-Modal, Emotional Understanding
N/A
Q-DeepSight: Incentivizing Thinking with Images for Image Quality Assessment and Refinement
Arxiv 2026
Image Quality
Thinking-with-Images for IQA, Refinement
N/A
Q-Probe: Scaling Image Quality Assessment to High Resolution via Context-Aware Agentic Probing
Arxiv 2026
Image Quality
Agentic Probing, High-Resolution IQA
N/A
VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning
Arxiv 2026
Robotics
VLA, Thinking-with-Image, Action Models
N/A
CoT-Drive: Efficient Motion Forecasting for Autonomous Driving with LLMs and Chain-of-Thought Prompting
Arxiv 2025
Autonomous Driving
Motion Forecasting, CoT
N/A
MedVLM-R1: Incentivizing Medical Reasoning Capability of VLMs via Reinforcement Learning
Arxiv 2025
Medical
Medical Reasoning, RL
GitHub
DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving
Arxiv 2024
Autonomous Driving
End-to-End Driving, CoT
Github
Robotic Control via Embodied Chain-of-Thought Reasoning
Arxiv 2024
Robotics
Embodied CoT, Control
GitHub
2026
Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessment
ICLR 2026
Image Quality
Grounding-IQA, Multimodal Referring, Quality Assessment
GitHub
2024
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
CVPR 2024
Robotics
Object-centric, Manipulation
GitHub
Dolphins: Multimodal Language Model for Driving
ECCV 2024
Autonomous Driving
Driving-specific, Multimodal
GitHub
Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
ECCV 2024
Autonomous Driving
Interpretable, Chain Reasoning
GitHub
MedCoT: Medical Chain of Thought via Hierarchical Expert
EMNLP 2024
Medical
Hierarchical Expert, Medical CoT
GitHub
⬆ Back to Top
In an era where synthetic media is increasingly sophisticated and pervasive, our research serves as a critical line of defense. By advancing fine-grained perception technologies, we aim to:
Enhance Visual Understanding: Enable MLLMs to perceive fine-grained details without expensive inference-time operations.
Bridge the Zooming Gap: Internalize the benefits of "zooming in" during training for efficient single-pass perception.
Industrial Application & Impact: We provide robust, scalable perception solutions for Ant Group's diverse content platforms.
For questions or collaborations, please contact:
⬆ Back to Top