HoloCount: A Holistic Visual Counting Benchmark for MLLMs

Jinhong Deng, Limeng Qiao, Guanglu Wan
Meituan
Code Dataset arXiv
2,480
QA Pairs
20
Fine-grained Subsets
1,481
Visual Concepts
20+
Models Evaluated

Abstract

Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios.

Benchmark Taxonomy

Taxonomy overview of the HoloCount benchmark

Taxonomy overview of the HoloCount benchmark. The dataset features a three-level hierarchical taxonomy containing 2,480 QA pairs across 20 fine-grained counting tasks.

HoloCount is organized around a three-level hierarchical taxonomy that shifts the focus from mere superficial perception to the advanced cognitive processing and environmental resilience required for true numerical grounding:

  • Semantic Counting (6 subsets) — assesses foundational skills through atomic counting and property-based filtering, testing the model's ability to isolate target objects by fine-grained attributes such as chromatic properties, material, scale, action & state, and general semantics.
  • Analytical Counting (7 subsets) — examines the interplay of vision and logic through spatial-based reasoning (Visual-Prompt Region Grounding, Coordinate-Prompt Region Grounding, Relative Canonical Orientation) and set-based logic reasoning (Differential Comparison, Joint-Set Aggregation, Complementary Exclusion, Categorical Cardinality).
  • Robustness Testing (7 subsets) — probes the systemic limits of model reliability under adverse perceptual domains (high-density scenes, aerial perspectives, small-scale objects, occlusion) and deceptive counter-priors (null-target prompting, linguistic prior conflict, visual distractor illusion).

Example Tasks

Example tasks from the HoloCount benchmark

Example tasks from the HoloCount benchmark, which comprises 20 fine-grained subtasks spanning Semantic Counting, Analytical Counting, and Robustness Testing.

Data Curation Pipeline

The general data curation pipeline of the HoloCount benchmark

The general data curation pipeline of the HoloCount benchmark.

To ensure diversity, difficulty, and diagnostic depth, we implement a multi-stage data curation pipeline with rigorous human-in-the-loop verification:

  1. Data Collection/Generation — We adaptively collect images from the internet, detection datasets, specialized benchmarks, and manual synthesis for specific purposes.
  2. Data Cleaning and Filtering — We exclude images with extreme sizes or aspect ratios, and images where target objects occupy extreme scales.
  3. QA Generation and Filtering — We use advanced MLLMs to generate QA pairs and employ another model to verify correctness.
  4. Rigorous Human Verification — Each sample undergoes manual review by expert annotators to correct numerical inaccuracies and ensure linguistically natural, unambiguous queries.

Main Results

Performance comparison of state-of-the-art MLLMs on HoloCount. Bold and underline denote the best and second-best results.

Main results on HoloCount benchmark

Key Findings

Overall Performance: The most advanced open-source models now rival or even surpass proprietary systems. The top open-source entry, Qwen3.5-397B-A17B, achieves a macro average of 76.9%, while leading closed-source models Gemini-3-Flash-Preview and Gemini-3.1-Pro-Preview reach 74.8% and 74.7%, respectively.
Catastrophic Failure in High-Density Scenes: Counting in dense scenarios remains a universal weakness. Even Qwen3.5-27B achieves only 20.0% on this subset while exceeding 80% on most others. Most models score below 10%, revealing that current vision encoders cannot maintain fine-grained spatial distinctions when objects heavily overlap.
Logical Execution vs. Perception Disconnect: A clear dissociation exists between atomic counting and higher-order analytical tasks. Nearly all models exceed 85% on Atomic Counting, but drop sharply on tasks requiring set operations or comparisons. For instance, InternVL3.5-38B falls from 79.7% to 35.0% on Differential Comparison.
Inverse Performance Paradox in Null-Target: Smaller models often outperform advanced proprietary engines when counting absent objects. Qwen3-VL-8B achieves 96.4%, while Gemini-3.1-Pro-Preview collapses to 55.2%, suggesting that aggressive optimization for scene description inadvertently increases over-confidence.
Resisting Linguistic Priors: Most models perform poorly (20-40%) when language priors conflict with visual evidence. The Gemini-3 series stands out with 69.9-73.6%, implying a stronger vision-first arbitration mechanism.
Impact of Thinking Mode: Extended thinking consistently improves counting accuracy across all model scales, with gains of 10.6 to 15.4 absolute percentage points. Smaller models benefit more (4B: +15.4), suggesting counting tasks require deliberate step-by-step reasoning.
Thinking vs Non-thinking comparison

Comparison of Qwen3.5 models in thinking versus non-thinking (instruct) mode on HoloCount. Thinking mode consistently improves counting accuracy across all model scales.

Dataset Composition

SubsetNumberSubsetNumber
Semantic Counting
  Atomic Counting (Atom.)182  Scale Attribution (Scale)100
  Chromatic Attribution (Chrom.)202  Action & State Attribution (Act.&St.)102
  Material Attribution (Mater.)101  General Semantic Attribution (Gen.)101
Analytical Counting
  Visual-Prompt Region Grounding (Vis.)102  Differential Comparison (Diff.)100
  Coordinate-Prompt Region Grounding (Coord.)78  Joint-Set Aggregation (Aggr.)100
  Relative Canonical Orientation (Orient.)84  Complementary Exclusion (Excl.)101
  Categorical Cardinality (Categ.)101
Robustness Testing
  High-Density Enumeration (Dense)100  Null-Target Prompting (Null.T.)250
  Aerial Perspective Shift (Aerial)109  Linguistic Prior Conflict (Ling.P.)163
  Small-Scale Enumeration (Small)101  Visual Distractor Illusion (Vis.D.)200
  Partial Object Occlusion (Occl.)103
Total2,480

Comparison with Existing Benchmarks

Benchmark # Samples # Visual Concepts Taxonomy Tasks Analytical Robustness
ShanghaiTech1,1981No1××
NWPU5,1091No1××
JHU-CROWD++4,3721No1××
CARPK1,1481No1××
FSC-1476,135147No1××
CountBench488231No1××
PixMo-Count54060No1××
CountQA1,528534No1××
HoloCount (Ours)2,4801,481Yes20✓✓

BibTeX

@inproceedings{deng2026holocount,
  title={HoloCount: A Holistic Visual Counting Benchmark for MLLMs},
  author={Deng, Jinhong and Qiao, Limeng and Wan, Guanglu},
  year={2026}
}