TimeScope is an open-source benchmark designed to measure how well vision-language models understand long videos. By adding short “needle” clips into videos ranging from 1 minute to 8 hours, it evaluates three skills:
• localized retrieval,
• information synthesis,
• fine-grained temporal perception. Timescope reveals that many state-of-the-art models still struggle with true temporal comprehension. (View Highlight)
Recent advances in multimodal AI have produced models claiming to understand hour-long videos. This trend mirrors progress in long-context language models, which excel at reasoning over lengthy text. Following this, vision-language systems now advertise context windows that can handle thousands of frames. But these claims require a closer look: do these models truly demonstrate understanding of the sequence of events? Are they limited to surface-level retrieval \ recognition? It’s crucial to ask if their capabilities are being overstated. (View Highlight)
his measurement gap leaves us wondering: What does it really mean for a model to “understand” long videos? To address this, we’re excited to introduce TimeScope, a new open-source benchmark hosted on Hugging Face. TimeScope probes the limits of long-video capabilities by inserting several short (~5-10 second) video clips—our “needles”—into base videos ranging from 1 minute to 8 hours. With three distinct task types, it evaluates not just retrieval but synthesis, localization, and fine-grained motion analysis, providing a more holistic view of temporal comprehension. (View Highlight)
TimeScope flips the script by emphasizing three pillars of long-video understanding:
Localized Retrieval: Can the model spot and answer questions about a specific short segment within a vast video?
Information Synthesis: Can it gather and order details from multiple points across the timeline?
Fine-Grained Temporal Perception: Can it analyze motion and events in needles that demand dense, multi-frame sampling? (View Highlight)
Localized Retrieval
This tests basic retrieval and understanding of a localized event. Questions are put so that sampling a relevant frame from the needle should suffice—like asking about a shorter part in a longer video.
Example:
What mode of transportation is shown in the video? (View Highlight)
Information Synthesis
Here, we embed multiple text-based needles (e.g., 2-4 short clips displaying “secret words” via on-screen text) at different points in the video. The model must identify all words and report them in chronological order, simulating tasks like extracting timestamps or key facts from dispersed scenes. This requires scanning the full timeline and understanding relative positioning (View Highlight)
Fine-Grained Temporal Perception
For questions focusing on motion or sequences within a short clip, single-frame sampling won’t cut it—the model needs to perceive dynamics across frames. This probes whether long-context handling preserves temporal fidelity. (View Highlight)
Model size isn’t everything. Qwen 2.5-VL 3B and 7B, as well as InternVL 2.5 models at 2B, 4B, and 8B parameters, exhibit nearly indistinguishable long-video curves to their smaller counterparts. All of them plateau at roughly the same context length, showing that simply scaling parameters does not automatically grant a longer temporal horizon. (View Highlight)
Gemini 2.5-Pro is in a league of its own. It is the only model that maintains strong accuracy on videos longer than one hour. (View Highlight)
Trade-offs across tasks matter. Qwen 2.5-VL shines in the Information-Synthesis (OCR) task—identifying and ordering dispersed text snippets—yet it falls behind on Fine-Grained Temporal Perception, where precise motion counting is required. (View Highlight)
