VST: Video LLMs That Think While Watching (Not After)

Here’s the problem with current Video LLMs: they wait.

You show the model a 10-minute surgical procedure. It watches silently, accumulating visual tokens. Then you ask “When did the surgeon nick the artery?” and it has to reason through everything it just saw — with you waiting.

This isn’t how humans watch video. We think while watching. When something important happens, we note it mentally. When the surgeon’s hand slips, we don’t wait until the video ends to process that information.

Video Streaming Thinking (VST) brings this capability to Video LLMs. Instead of passive perception followed by expensive post-hoc reasoning, VST thinks continuously during playback — producing intermediate “streaming thoughts” that make final responses both faster and more accurate.

The Core Innovation: Amortized Reasoning

The key insight is clever: if you know you’ll need to reason about a video eventually, why wait?

VST shifts the LLM from passive waiting to active intermittent reasoning during video consumption. As each clip arrives, the model processes it immediately and generates intermediate thoughts. By the time a user asks a question, most of the reasoning work is already done.

This “front-loads and amortizes” the reasoning cost across video playback. The result:

• 15.7x faster responses than Video-R1 (a strong reasoning-focused baseline)
• +5.4% better accuracy on VideoHolmes reasoning benchmark
• Real-time responsiveness preserved even with deep reasoning

This isn’t a speed-quality tradeoff. The continuous thinking approach actually improves comprehension because the model maintains sustained attention rather than trying to reconstruct context after the fact.

How It Works

The architecture implements a “thinking-while-watching” mechanism inspired by human neural coupling:

Streaming Thoughts: As video clips arrive, VST produces intermediate reasoning tokens that capture evolving understanding. These aren’t just cached features — they’re actual thoughts about what’s happening.

Causal Streaming: Unlike offline models that can look ahead, VST operates causally. At any moment, it only knows what it’s seen so far, just like a human viewer.

Multi-Turn Interaction: The model handles ongoing dialogue while video plays. You can ask questions mid-stream and get immediate, contextually-grounded responses.

Training Pipeline

VST uses a two-stage post-training approach:

1. VST-SFT: Structurally adapts an offline VideoLLM to causal streaming reasoning. The model learns to produce streaming thoughts that are useful for downstream QA.

2. VST-RL: Reinforcement learning through self-exploration in a multi-turn video interaction environment. The model improves its streaming thoughts based on how well they support final answers.

The training data is generated using video knowledge graphs — extracting entities and relations from videos to create high-quality streaming QA pairs. An “entity-relation grounded streaming Chain-of-Thought” enforces multi-evidence reasoning and sustained attention.

Benchmark Performance

VST-7B delivers strong results across both streaming and offline benchmarks:

Benchmark	VST-7B	Notes
StreamingBench	79.5%	Primary streaming benchmark
OVO-Bench	59.3%	Online video understanding
VideoMME	64.9%	General video comprehension
LongVideoBench	58.0%	Long-form video
VideoHolmes	41.9%	Complex reasoning (+5.4% vs Video-R1)

The larger VST-32B pushes further: 80.7% on StreamingBench and 45.1% on VideoHolmes.

Why This Matters: Real Applications

The streaming thinking paradigm unlocks use cases that were previously impractical:

Surgical Assistance

A VST-powered copilot watching a live procedure could track anatomical landmarks, note when critical structures come into view, and flag potential complications — all in real-time. When the surgeon asks “Is that the common bile duct?”, the answer is immediate because the model has been reasoning about anatomy throughout.

Current systems require the surgeon to pause, upload a clip, and wait for analysis. VST operates at the speed of thought.

Intraoperative Decision Support

During surgery, timing is everything. A model that thinks while watching could:

• Track instrument positions relative to critical structures
• Monitor for bleeding or tissue damage in real-time
• Correlate current views with pre-operative imaging
• Flag when technique deviates from standard protocols

The continuous reasoning means these insights are available the moment they’re relevant, not seconds later.

Surgical Education and Debriefing

For training residents, VST could provide real-time annotation of what’s happening and why. “Notice how the surgeon is approaching the hilum from this angle” — delivered as it happens, not reconstructed after the fact.

Post-operative review becomes richer when the model has genuine streaming thoughts to share, not just frame-by-frame analysis generated after the video ends.

Live Sports Coaching

A VST system watching a basketball game could track player positioning, identify developing plays, and predict outcomes — all while the action unfolds. Coaches could query “Why did that fast break fail?” and get immediate analysis grounded in real-time observation.

Security and Surveillance

Monitoring feeds for anomalies becomes more effective when the model maintains continuous situational awareness rather than processing frames independently. VST can track evolving situations across time.

Quality Control

Manufacturing inspection often requires understanding sequences of operations. VST could watch assembly processes and flag deviations in real-time — “That weld didn’t fully penetrate” — rather than discovering issues in post-hoc review.

Emergency Response

First responders reviewing drone footage or body camera streams could get real-time situation analysis. “Three people visible in the northeast corner, one appears injured” — immediate, not after scrubbing through footage.

Robotic Telepresence

Remote operators controlling robots benefit from AI that thinks alongside them. VST could maintain spatial awareness and flag hazards while the operator focuses on the task at hand.

The Human Analogy

The paper draws inspiration from “neural coupling” — the way human brains synchronize processing with ongoing experience rather than buffering everything for later analysis.

When you watch a movie, you don’t passively record visual data and then reason about the plot at the end. You’re constantly building mental models, forming predictions, updating understanding. VST brings this cognitive style to Video LLMs.

Available Now

The team has released everything:

Models (HuggingFace):

• VST-3B
• VST-7B
• VST-32B

Training Data:

• vst_sft_data: Video-text pairs from multiple sources
• vst_rl_data: RL training data for multi-turn interactions

Code: Full training and evaluation at github.com/1ranGuan/VST

The Bottom Line

Video Streaming Thinking represents a genuine paradigm shift in how we build Video LLMs. The insight — that models should think during video consumption, not after — seems obvious in retrospect. But implementing it required careful architectural choices and a novel training pipeline.

The results speak for themselves: faster responses and better accuracy. For applications requiring real-time video understanding — especially high-stakes domains like surgical assistance — this is exactly the capability that’s been missing.

VST doesn’t just make Video LLMs faster. It makes them think the way humans do.

Paper: arXiv:2603.12262
Code: github.com/1ranGuan/VST
Project Page: 1ranguan.github.io/VST