Machine Theory of Mind in naturalistic social videos
Existing ToM benchmarks are often clean but artificial. WildToM moves evaluation to in-the-wild interactions where social intent, emotion, and knowledge must be inferred from multimodal evidence under perspective constraints.
A hybrid human-AI pipeline converts naturalistic videos into diagnostic ToM evaluation samples with explicit reasoning order labels.
Clip selection favors socially rich interactions rather than scripted prompts.
Character-centered prompts recover latent states from situated perspectives.
Samples are rewritten into consistent MC and open-ended formats with order labels.
Annotators refine factual grounding, plausibility, and category/order correctness.
WildToM-Bench balances scale with social diversity across dimensions, scenes, and relationship types.
| Category | Count | Ratio |
|---|---|---|
| Home | 91 | 31.4% |
| Outdoor | 65 | 22.4% |
| Workplace | 57 | 19.7% |
| Restaurant | 38 | 13.1% |
| Hospital | 17 | 5.9% |
| Store | 12 | 4.1% |
| School | 10 | 3.4% |
| Category | Count | Ratio |
|---|---|---|
| Friends | 79 | 27.2% |
| Family | 57 | 19.7% |
| Strangers | 50 | 17.2% |
| Colleagues | 50 | 17.2% |
| Romantic | 30 | 10.3% |
| Authority | 24 | 8.3% |
Performance gaps reveal that nested social reasoning is still a major bottleneck for current multimodal systems.
In the current draft, WildToM-Reasoner reaches 72.7% MC accuracy, compared with 62.1% for the strongest baseline Qwen3-VL-32B. The first-to-second order gap is not uniform: it is small for Belief/Desire but expands sharply for Intention and Knowledge.
Curated examples from selected Social-IQ clips. Browse one case at a time by mental dimension.
@misc{wildtom2026,
title = {WildToM: Benchmarking Machine-Theory-of-Mind in the Wild},
author = {Author list to be inserted},
year = {2026}
}