Skip to content

ant-research/Awesome-Fine-Grained-Multimodal-Perception

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Awesome Fine-Grained Multimodal Perception Awesome

Overview

A curated collection of the latest research and resources on Fine-Grained Multimodal Perception. This repository encompasses datasets, benchmarks, research papers, and practical methods for enhancing fine-grained perception capabilities in Multimodal Large Language Models (MLLMs).

🚀🚀🚀 Contributions are welcome! If you find any missing papers, datasets, or tools, feel free to open an issue or submit a pull request.


📢 Latest Updates

Date News
2026-06-04 Added Attentive-CoT and Beyond Visual Memory to Training-Based Methods (SFT & Distillation)
2026-06-03 Updated iVGR venue to ICML 2026; Reorganized structure with What is, 📚 TOC, and time-based sorting
📜 View More Updates
Date News

What is Fine-Grained Multimodal Perception?

Multimodal Large Language Models (MLLMs) excel at broad visual understanding but still struggle with fine-grained perception, where decisive evidence is small and easily overwhelmed by global context. Recent advancements have shifted towards a "Thinking-with-Images" paradigm, where MLLMs actively acquire local information during inference rather than relying solely on a global image encoding.

This awesome list categorizes methods into four paradigms:

  • Data-Centric Methods: Improving perception through data quality, quantity, and synthetic generation
  • Training-Free Methods: Enhancing perception at inference time via prompting, chain-of-thought, and attention strategies
  • Agentic: Thinking with Images: Actively interacting with images through tools (zoom, crop, search, edit) during inference
  • Training-Based Methods: Improving perception through supervised fine-tuning, distillation, and reinforcement learning

Back to Top


📚 Table of Contents


Benchmarks & Datasets

Task Legend: FG: Fine-Grained Perception | OCR: Text Recognition | Cnt: Counting | Grd: Grounding | Rea: Reasoning

Benchmark Paper Venue & Year Task Highlights Download
Preprints
GeoBrowse GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces Arxiv 2026 FG, Rea Geolocation benchmark, agentic tool use, expert reasoning traces GitHub
CountQA CountQA: How Well Do MLLMs Count in the Wild? Arxiv 2025 Cnt Visual counting with reasoning HuggingFace
CoCoT CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs Arxiv 2024 Grd Visual grounding with reasoning N/A
Rexverse ChatRex: Taming Multimodal LLM for Joint Perception and Understanding Arxiv 2024 FG Fine-grained visual understanding GitHub
2026
VisuRiddles VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning AAAI 2026 FG Visual riddles, fine-grained perception evaluation Github
GroundingME GroundingME: Grounding Multi-Modal Evaluation CVPR 2026 Grd Multi-modal grounding evaluation GitHub
RTV-Bench RTV-Bench: Benchmarking MLLM Continuous Perception ICLR 2026 FG, Rea Real-time video analysis, 552 videos, 4,608 QA pairs, multi-timestamp reasoning GitHub
SFE Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning ICLR 2026 FG, Rea Scientific cognitive evaluation, three interconnected levels GitHub
Thyme-SFT/RL Thyme: Thinking with Images for Visual Reasoning ICLR 2026 FG, Rea Visual reasoning with tool use GitHub
Visual Probe Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search ICLR 2026 FG Visual probing for perception GitHub
TreeBench Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology ICLR 2026 Rea Traceable reasoning evaluation GitHub
ZoomBench Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception ICML 2026 FG, Rea 845 VQA samples, 6 perceptual dimensions, dual-view evaluation GitHub
2025
HR-Bench Divide and Conquer: A Comprehensive Benchmark for High-Resolution Image Understanding AAAI 2025 FG High-resolution image understanding GitHub
MME-RealWorld MME-RealWorld: Evaluating Real-World Perception in MLLMs ICLR 2025 FG, Grd Real-world perception benchmark GitHub
ColorBench ColorBench: Can VLMs See and Understand the Colorful World? A Benchmark for Color Perception NeurIPS 2025 FG Color perception evaluation GitHub
2024
VStar V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs CVPR 2024 FG Visual search, attribute recognition GitHub
CV-Bench Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs CVPR 2024 FG, Rea Computer vision-centric benchmark HuggingFace
MMVP Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs CVPR 2024 FG Visual shortcomings evaluation GitHub
2017
Visual Genome Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations IJCV 2017 Grd, FG Dense image annotations, grounding HuggingFace

Back to Top


Research Papers

Data-Centric Methods

This category focuses on methods that improve fine-grained perception through data quality, quantity, and synthetic data generation pipelines.

Title Venue & Year Highlights/Keywords Code
2025
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct ACL 2025 Evolution, Instruction Tuning GitHub
Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis ICCV 2025 Autonomous Synthesis, Open-source GitHub
First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training NeurIPS 2025 Unsupervised Pre-training GitHub
Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning NeurIPS 2025 Minimally Edited, Difference Detection GitHub
2024
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator ECCV 2024 VQA Generation, Unlabeled Images GitHub
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model EMNLP 2024 Self-Instruct, Synthetic Data GitHub

Back to Top


Training-Free Methods

This category covers inference-time approaches that enhance fine-grained perception without additional training, leveraging prompting strategies, chain-of-thought reasoning, and attention mechanisms.

Title Venue & Year Highlights/Keywords Code
Preprints
Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models Arxiv 2026 Query-Aware, Adaptive Perception, Efficient Zoom Github
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs Arxiv 2026 Training-Free, Iterative, Visually-Grounded Reasoning GitHub
SvfEye: A Semantic-Visual Fusion Framework with Multi-Scale Visual Context for Multimodal Reasoning Arxiv 2026 Semantic-Visual Fusion, Multi-Scale Context GitHub
Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs Arxiv 2025 Focusing Strategy, Chain Reasoning GitHub
Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild Arxiv 2025 Self-guided Reasoning, Socratic Method GitHub
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models Arxiv 2024 Image-of-Thought, Reasoning Refinement N/A
Chain of Thought Prompt Tuning in Vision Language Models Arxiv 2023 Prompt Tuning, VLM CoT N/A
Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings Arxiv 2023 Multimodal Infillings, Visual CoT GitHub
2026
ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models ICLR 2026 Efficient High-Resolution, Visual Understanding GitHub
Hide: Rethinking the zoom-in method in high resolution mllms via hierarchical decoupling ICML 2026 Attention-based Detection GitHub
2025
ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Search EMNLP 2025 Tree Search, Attention Mapping GitHub
Imagine while Reasoning in Space: Multimodal Visualization-of-Thought ICML 2025 Spatial Visualization, Imaginative Reasoning GitHub
2024
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning AAAI 2024 Knowledge Augmentation, CoT Reasoning N/A
Visual Chain-of-Thought Prompting for Knowledge-based Visual Reasoning AAAI 2024 Interactive Prompting, Knowledge-based Reasoning N/A
TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding ACM TOMM Text-rich Images, Zoom-in Strategy GitHub
Compositional Chain-of-Thought Prompting for Large Multimodal Models CVPR 2024 Compositional Prompting GitHub
Cantor: Inspiring Multimodal Chain-of-Thought of MLLM MM 2024 CoT Inspiration, MLLM Reasoning GitHub
Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models NeurIPS 2024 Visualization-of-Thought, Spatial Reasoning GitHub
Multimodal Chain-of-Thought Reasoning in Language Models TMLR 2024 Multimodal CoT, Two-stage Framework GitHub
2023
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models NeurIPS 2023 Compositional Reasoning, Tool Integration GitHub

Back to Top


Agentic: Thinking with Images

This category includes models that actively interact with images during inference using tools (zoom, crop, search, edit, sketch), enabling iterative visual exploration and dynamic focus on regions of interest. These methods reduce interference by isolating micro-crops but typically incur higher latency due to multi-pass inference.

Title Venue & Year Highlights/Keywords Code
Preprints
S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images Arxiv 2026 Scientific Reasoning, Thinking-with-Images N/A
Test-time Scaling over Perception: Resolving the Grounding Paradox in Thinking with Images Arxiv 2026 Test-time Scaling, Grounding Paradox N/A
Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization Arxiv 2026 Agentic Policy Optimization, Reasoning-Action Gap N/A
CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning Arxiv 2025 Tool-integrated, Executable Visual Reasoning GitHub
DeepEyesV2: Toward Agentic Multimodal Model Arxiv 2025 Scaled Tool Learning, Visual Search GitHub
DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning Arxiv 2025 Visual Manipulation, Sketching, Internalized GitHub
VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought Arxiv 2025 Region Refinement, Recursive Focusing N/A
VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning Arxiv 2025 Video Reward Models, Thinking-with-Image GitHub
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models Arxiv 2024 Sketching, Visual Chain of Thought GitHub
2026
Let's Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts AAAI 2026 Interleaved-Modal CoT, Dynamic Visual Thoughts GitHub
Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm CVPR 2026 Video Generation, Multimodal Reasoning Paradigm GitHub
AGILE: Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in VLMs ICLR 2026 Agentic Interaction, Jigsaw Puzzle, Visual Perception GitHub
DeepEyes: Incentivizing "Thinking with Images" in Vision-Language Models via Reinforcement Learning ICLR 2026 Reinforcement Learning, Zoom/Crop Tools, Long CoT GitHub
Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search ICLR 2026 Visual Probing, Agentic Tools GitHub
Thyme: Think Beyond Images ICLR 2026 Pixel-space Operations, Code Generation GitHub
Visual Planning: Let's Think Only with Images ICLR 2026 Oral Visual Planning, Image-only Reasoning GitHub
2025
Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought CVPR 2025 Visual CoT, Attention Grounding Project
ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding ICML 2025 Visual Editing, Structured Understanding GitHub
PixelReasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning NeurIPS 2025 Pixel-level Operations, Dynamic Visual Manipulation GitHub
2024
Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models CVPR 2024 Interactive Spot Identification, Zoom-in Reasoning GitHub
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs CVPR 2024 Visual Search, Search-based Perception GitHub

Back to Top


Training-Based Methods

This category covers methods that improve fine-grained perception through training procedures, including supervised fine-tuning, knowledge distillation, and reinforcement learning.

Supervised Fine-Tuning & Distillation

Title Venue & Year Highlights/Keywords Code
Preprints
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning Arxiv 2026 Attention-Guided CoT-SFT, Delayed Answer Commitment, Visual-Token Access N/A
Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning Arxiv 2026 Latent Token Decomposition, Boundary Markers, Mechanistic Analysis N/A
LanteRn: Latent Visual Structured Reasoning Arxiv 2026 Latent Visual Reasoning, Structured Reasoning N/A
Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step Arxiv 2025 Image Generation CoT, Verification GitHub
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems? Arxiv 2025 Long-CoT, Scaling N/A
URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics Arxiv 2025 Mathematical Reasoning, Verification GitHub
AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning Arxiv 2024 Slow Thinking, Mathematical Reasoning GitHub
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models Arxiv 2024 Efficient Fine-Grained Perception, Text-rich GitHub
2026
Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology ICLR 2026 Traceable Evidence, Visual Grounded Reasoning GitHub
Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception ICML 2026 Region-to-Image Distillation, ZoomBench, Single-Pass Perception GitHub
2025
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models CVPR 2025 Long-Chain Reasoning, Insightful Analysis GitHub
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models CVPR 2025 Perception Tokens, Visual Reasoning Project
SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning NeurIPS 2025 Depth Perception, Spatial Reasoning, Structured Rationales GitHub
2024
Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition ICML 2024 Video Reasoning, Perception to Cognition GitHub

Back to Top


Reinforcement Learning

Title Venue & Year Highlights/Keywords Code
2026
iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning ICML 2026 Visually Grounded Reasoning, Internalization, RL N/A
PAPO: Perception-Aware Policy Optimization for Multimodal Reasoning ICLR 2026 PAPO, Perception-Aware RL, Implicit Perception Supervision GitHub
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward ICLR 2026 Visual Perception Reward, RLVR, GRPO GitHub
RAPID: Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning ICLR 2026 Perception-Decoupling, Reasoning Alignment, Two-stage Pipeline GitHub
Spotlight on Token Perception for Multimodal Reinforcement Learning ICLR 2026 Token Perception, Visual Dependency, RLVR Optimization GitHub
ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Models ICLR 2026 Self-Evolution, Fine-Grained Perception, RL GitHub
VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning ICLR 2026 Unified Perception-Reasoning, RL, Multi-task GitHub
VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use ICLR 2026 Reinforcement Learning, Tool Use, Thinking-with-Images GitHub
Preprints
Boosting Multimodal Reasoning with MCTS-Automated Structured Thinking Arxiv 2025 MCTS, Structured Thinking GitHub
Boosting the Generalization and Reasoning of VLMs with Curriculum Reinforcement Learning Arxiv 2025 Curriculum RL, Generalization N/A
LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL Arxiv 2025 Two-Stage RL, Rule-based GitHub
MedVLM-R1: Incentivizing Medical Reasoning Capability of VLMs via Reinforcement Learning Arxiv 2025 Medical Reasoning, RL GitHub
MM-Eureka: Exploring Visual Aha Moment with Rule-based Large-scale Reinforcement Learning Arxiv 2025 Visual Aha Moment, Rule-based RL GitHub
OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning via Iterative Self-Improvement Arxiv 2025 Self-Improvement, Iterative Training GitHub
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning Arxiv 2025 Emotion Recognition, Explainable RL GitHub
R1-OneVision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization Arxiv 2025 Cross-Modal, Generalized Reasoning GitHub
Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement Arxiv 2025 Segmentation, Cognitive Reinforcement GitHub
Training Multi-Image Vision Agents via End2End Reinforcement Learning Arxiv 2025 Multi-Image, End2End RL, Vision Agents N/A
Virgo: A Preliminary Exploration on Reproducing o1-like MLLM Arxiv 2025 O1-like MLLM, Reasoning Exploration GitHub
VisualPRM: An Effective Process Reward Model for Multimodal Reasoning Arxiv 2025 Process Reward Model Project
VisualThinker-R1-Zero: R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model Arxiv 2025 R1-Zero, Visual Reasoning GitHub
VisRL: Intention-Driven Visual Perception via Reinforced Reasoning Arxiv 2025 Intention-Driven, Reinforced Reasoning GitHub
2025
LlamaV-o1: Rethinking Step-by-Step Visual Reasoning in LLMs ACL-Findings 2025 Visual Reasoning, O1-style GitHub
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization ICCV 2025 GRPO, Step-wise Reasoning GitHub
Diving into Self-Evolving Training for Multimodal Reasoning ICML 2025 Self-Evolving, RL Training GitHub
Mulberry: Empowering MLLM with O1-like Reasoning and Reflection via Collective Monte Carlo Tree Search NeurIPS 2025 Spotlight MCTS, O1-like Reasoning GitHub
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs NeurIPS 2025 Visual Hallucination Critic, RL Proxy Task, Fine-Grained Perception GitHub

Back to Top


Applications & Domains

This category covers application domains where fine-grained perception is critical.

Title Venue & Year Domain Highlights/Keywords Code
Preprints
EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs Arxiv 2026 Emotion Omni-Modal, Emotional Understanding N/A
Q-DeepSight: Incentivizing Thinking with Images for Image Quality Assessment and Refinement Arxiv 2026 Image Quality Thinking-with-Images for IQA, Refinement N/A
Q-Probe: Scaling Image Quality Assessment to High Resolution via Context-Aware Agentic Probing Arxiv 2026 Image Quality Agentic Probing, High-Resolution IQA N/A
VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning Arxiv 2026 Robotics VLA, Thinking-with-Image, Action Models N/A
CoT-Drive: Efficient Motion Forecasting for Autonomous Driving with LLMs and Chain-of-Thought Prompting Arxiv 2025 Autonomous Driving Motion Forecasting, CoT N/A
MedVLM-R1: Incentivizing Medical Reasoning Capability of VLMs via Reinforcement Learning Arxiv 2025 Medical Medical Reasoning, RL GitHub
DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving Arxiv 2024 Autonomous Driving End-to-End Driving, CoT Github
Robotic Control via Embodied Chain-of-Thought Reasoning Arxiv 2024 Robotics Embodied CoT, Control GitHub
2026
Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessment ICLR 2026 Image Quality Grounding-IQA, Multimodal Referring, Quality Assessment GitHub
2024
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation CVPR 2024 Robotics Object-centric, Manipulation GitHub
Dolphins: Multimodal Language Model for Driving ECCV 2024 Autonomous Driving Driving-specific, Multimodal GitHub
Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving ECCV 2024 Autonomous Driving Interpretable, Chain Reasoning GitHub
MedCoT: Medical Chain of Thought via Hierarchical Expert EMNLP 2024 Medical Hierarchical Expert, Medical CoT GitHub

Back to Top


Why We Do It

In an era where synthetic media is increasingly sophisticated and pervasive, our research serves as a critical line of defense. By advancing fine-grained perception technologies, we aim to:

  • Enhance Visual Understanding: Enable MLLMs to perceive fine-grained details without expensive inference-time operations.
  • Bridge the Zooming Gap: Internalize the benefits of "zooming in" during training for efficient single-pass perception.
  • Industrial Application & Impact: We provide robust, scalable perception solutions for Ant Group's diverse content platforms.

✉️ Contact Us

For questions or collaborations, please contact:

Back to Top


Star History

Star History Chart

About

A collection of the latest research and resources on Fine-Grained Multimodal Perception

Resources

License

Stars

30 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors