What is Director by VideoDB?

Director is an open-source framework for building AI video agents that can reason through complex video tasks like search, editing, compilation, and generation. Natural language commands orchestrate multiple agents to complete workflows automatically.

How does Director compare to VST (Video Streaming Thinking)?

They're complementary. VST solves video understanding (thinking while watching), Director solves video doing (orchestrating actions). VST could power Director's comprehension agents, while Director handles the workflow orchestration and output generation.

What agents come pre-built with Director?

20+ agents including video summarization, media search, clip editing, voiceover generation, language translation, subtitle creation, video script generation, highlight extraction, and text-to-movie generation.

Can I add custom agents to Director?

Yes. Copy the sample_agent.py template, implement your logic, register it in the ChatHandler, and it's available. The framework handles context, progress reporting, and output streaming automatically.

What infrastructure does Director use?

Built on VideoDB's video-as-data infrastructure for storage, indexing, and streaming. Connects to LLMs, databases, and GenAI APIs. Includes a chat-based UI, video player, and collection browser.

Is Director open source?

Yes, fully open source on GitHub (video-db/Director). One-click deploy to Render or Railway. Self-host or run locally with Python 3.9+ and Node.js 22.8+.

Director: The Missing Orchestration Layer for AI Video Agents

By Prahlad Menon Published 2026-08-10 5 min read

Last week I wrote about VST — Video LLMs that think while watching. The core insight: models should reason during video playback, not after. It’s a breakthrough in video understanding.

But understanding is only half the problem.

Once an AI comprehends your video, what does it do with that knowledge? How do you go from “find the funniest moment” to actually having a clip ready to share on Slack?

Enter Director — VideoDB’s open-source framework that turns video understanding into video action.

The Orchestration Gap

Here’s the workflow most people suffer through today:

Upload video to transcription service
Read transcript, manually find interesting moments
Open video editor, scrub to those timestamps
Cut clips, export
Maybe add subtitles (another tool)
Upload somewhere for sharing
Finally share the link

Director reduces this to: “Upload this video and send the highlights to my Slack.”

That’s it. The framework’s reasoning engine figures out the rest — which agents to invoke, in what order, with what parameters. Upload agent → scene detection agent → highlight extraction agent → clip creation agent → Slack sharing agent. All orchestrated automatically.

What Director Actually Is

Think of Director as ChatGPT for video — a chat interface where natural language commands trigger complex multi-step workflows:

“Summarize this 2-hour lecture in 5 minutes”
“Find every mention of ‘revenue’ and create a compilation”
“Generate a movie from this script with voiceovers”
“Translate this video to Spanish with new subtitles”
“Extract the funniest scenes and add meme captions”

Behind the chat interface is a reasoning engine that:

Parses your intent
Identifies which agents are needed
Orchestrates them in the right sequence
Streams progress updates in real-time
Delivers playable results instantly

The key word is instantly. Built on VideoDB’s streaming infrastructure, you don’t wait for exports or downloads. Results play immediately.

20+ Pre-Built Agents

Director ships with agents for the workflows people actually need:

Category	Agents
Understanding	Video summarization, scene detection, transcript extraction
Search	Media search, moment finding, semantic clip retrieval
Editing	Clip creation, compilation, overlay addition, frame extraction
Audio	Voiceover generation, dubbing, audio extraction
Localization	Language translation, subtitle creation, caption generation
Generation	Text-to-movie, script-to-video, highlight reels

Each agent is a focused unit that does one thing well. The reasoning engine composes them into workflows.

The VST Connection

This is where it gets interesting.

VST (Video Streaming Thinking) represents the cutting edge of video comprehension — models that maintain continuous reasoning during playback, not post-hoc analysis.

Director represents the cutting edge of video action — orchestrating agents to do useful things with video.

They’re complementary layers of the same stack:

┌─────────────────────────────────────┐
│         User Intent                 │
│   "Find the surgery complication"   │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│     Director (Orchestration)        │
│   Reasoning engine, agent routing   │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│     VST (Understanding)             │
│   Real-time streaming comprehension │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│     VideoDB (Infrastructure)        │
│   Storage, indexing, streaming      │
└─────────────────────────────────────┘

A VST-powered comprehension agent inside Director would be formidable. Real-time understanding feeding into automated workflows. “Watch this surgery and flag any complications” becomes a single command that continuously monitors and acts.

Architecture Deep Dive

Director’s architecture is clean:

Reasoning Engine: The brain. Analyzes user input, maintains context across turns, dynamically selects and orchestrates agents. Handles multi-agent coordination for complex workflows.

Chat UI: Conversational interface with integrated video playback. Watch results while you chat. Based on videodb-chat.

Video Player: Not just playback — interactive tools for scrubbing, annotation, and selection. Multi-platform support via videodb-player.

Collection View: Browse and organize your media library. Search, filter, batch operations.

Agent Framework: Each agent is a Python class with a simple interface:

run() method for main logic
Content types for output (TextContent, VideoContent, ImageContent)
Progress updates via push_update()
Final publish via self.output_message.publish()

Building Custom Agents

Adding a new agent is straightforward:

# 1. Copy sample_agent.py, rename
# 2. Update class name, agent_name, description
# 3. Implement run() with your logic
# 4. Use appropriate content types for output
# 5. Register in ChatHandler

The framework handles:

Session persistence
Progress streaming
Error handling
Output formatting
External API connectivity

If your agent’s logic could be shared across multiple agents, factor it into a reusable tool instead.

Real-World Workflows

Content Repurposing: “Take my 1-hour podcast and create 10 short clips for social media with captions.”

Director: Upload → Transcribe → Identify highlight moments → Generate 10 clips → Add captions → Export with platform-specific formatting.

Video Search at Scale: “Search my entire media library for mentions of ‘product launch’ and compile a timeline.”

Director: Index library → Semantic search across all videos → Extract relevant clips → Compile chronologically → Stream result.

Automated Localization: “Translate this training video to Japanese, French, and German with native voiceovers.”

Director: Extract audio → Transcribe → Translate (3x) → Generate voiceovers (3x) → Sync with video → Export three versions.

Meeting Intelligence: “Summarize this Zoom recording and create action item clips for each person mentioned.”

Director: Transcribe → Identify speakers → Extract action items → Create per-person clip compilations → Generate summary document.

Deploy in One Click

Director is fully open source and easy to run:

Local:

git clone https://github.com/video-db/Director.git
cd Director
./setup.sh
make run

Cloud: One-click deploy buttons for Render and Railway.

Requirements: Python 3.9+, Node.js 22.8+, and API keys for your preferred LLM/GenAI services.

The Bigger Picture

We’re watching the video AI stack mature in real-time:

Understanding: VST, Video-R1, and streaming VideoLLMs that comprehend video as it plays
Orchestration: Director and agentic frameworks that turn comprehension into action
Infrastructure: VideoDB and similar platforms that treat video as queryable, streamable data

The gap between “AI can understand video” and “AI can do useful things with video” is closing fast. Director is a significant step — it’s the workflow layer that was missing.

For developers building video-powered applications, this is the framework to watch. For teams drowning in video content, it’s the automation layer you’ve been waiting for.

Links: