NVIDIA's SpatialClaw: Why Code Is the Best Action Interface for AI Agents

NVIDIA just published results that validate what every serious agent developer already suspects:

Code is the right action interface.

Not JSON schemas. Not structured tool calls. Not single-pass programs.

Actual code, running in a persistent kernel where variables survive across steps, where the agent can inspect intermediate results, and where it can revise its strategy based on what it observes.

The framework is called SpatialClaw, and while it’s focused on spatial reasoning, the implications extend far beyond that domain.

The Core Insight

Three action interfaces, three very different outcomes:

Interface	Approach	Weakness
Single-pass code	Write complete program, execute once	Can’t revise after execution starts
Structured tool-call	JSON API, fixed schemas	Can’t freely compose outputs with NumPy/SciPy
Code as action	Persistent kernel, incremental execution	✓ Compose, inspect, revise

SpatialClaw beats the prior best spatial agent by +11.2 points across 20 benchmarks. Same backbone model. Same perception tools. The only difference is the action interface.

That’s not a marginal improvement—that’s a fundamental architectural advantage.

Why This Matters for Your Agents

If you’re building coding agents, this confirms what tools like Claude Code and Codex CLI already embody:

The agent needs to:

1. Write code
2. Execute it
3. See the results
4. Revise and continue

Without step 3, you’re flying blind. Without step 4, you can’t recover from wrong assumptions.

The structured tool-call approach—{"tool": "get_depth", "params": {...}}—locks you into a fixed composition model. You can chain tools, but you can’t do this:

# Get depth for two objects
depth_1 = perception.get_depth(mask_1)
depth_2 = perception.get_depth(mask_2)

# Compose with standard scientific computing
from scipy.spatial import KDTree
tree = KDTree(points_1)
distances, _ = tree.query(points_2)

# Inspect intermediate result
print(f"Min distance: {distances.min():.3f}")

# Revise if needed
if distances.min() > threshold:
    # Try a different approach
    ...

With structured tool calls, that composition is impossible. The agent can invoke tools, but it can’t think in code.

The Numbers

SpatialClaw tested across six VLM backbones (26B to 397B parameters):

Backbone	No-tool	SpatialClaw	Gain
Qwen 3.6-27B	55.0	62.7	+7.7
Gemma 4-31B	53.4	59.9	+6.5
Gemma 4-26B-A4B	48.0	54.3	+6.3
Qwen 3.6-35B-A3B	52.6	57.2	+4.6
Qwen 3.5-122B-A10B	53.7	56.9	+3.2
Qwen 3.5-397B-A17B	57.3	60.4	+3.1

Key insight: smaller models with code-as-action beat larger models without it. Gemma 4-26B with SpatialClaw (54.3) outperforms Qwen 3.5-122B without it (53.7).

The architecture matters more than the model size.

Four Findings Worth Knowing

1. Utility wrappers don’t matter much

Removing all utility functions (tools.Mask, tools.Geometry, etc.) and keeping only core perception + scientific libraries barely affects performance (56.4 vs 56.9).

The persistent kernel with NumPy/SciPy compensates. The interface is the value, not the pre-built utilities.

2. The agent adapts its tool composition spontaneously

Without any category-specific prompting:

• Distance questions → KDTree search, vector norms
• Direction questions → dot products, angular ops
• Camera motion → pose composition, transformation chains

The model learns to compose appropriately from question semantics alone. Structured interfaces can’t elicit this behavior.

3. Gains concentrate where chained computation is needed

Largest improvements (+6-9pp) in:

• Camera motion
• Multi-view reasoning
• Relative direction

These require geometric computation across frames and viewpoints. One-shot approaches fail precisely because they can’t accumulate and compose results.

4. 70%+ of wins come from composition and control flow

When SpatialClaw beats structured tool-call, an LLM judge attributes:

• 52.2% to code composition (chaining multiple calls coherently)
• 19.5% to control flow (if/for over intermediate results)
• 28.3% to interface-neutral perceptual wins

The wins are architectural, not accidental.

Connection to Codebase Tools

We wrote recently about codebase-memory-mcp—a tool that indexes codebases into knowledge graphs for structural queries. Same principle applies:

SpatialClaw	codebase-memory
Perception modules in persistent kernel	Graph queries in persistent session
Compose depth + segmentation + scipy	Compose trace_path + detect_changes + filter
Inspect intermediate 3D reconstructions	Inspect intermediate call graphs
Revise based on visual feedback	Revise based on query results

Both benefit from code as the action interface. The agent can chain operations, inspect state, and refine its approach.

If you’re using codebase-memory (or similar tools like SocratiCode), SpatialClaw’s results suggest an enhancement path: expose graph queries as functions in a persistent kernel, let the agent compose them with standard Python, and watch the quality improve.

Real-World Test: Codebase Analysis

I tested this on a real repository—a multi-provider AI agent container with 60+ markdown files, shell scripts, and dynamic config generation.

Traditional approach (file-by-file reading):

• Would consume ~25,000 tokens reading core files
• MEMORY.md alone is 64KB
• Most of that content irrelevant to architectural questions

SpatialClaw-style approach (compose-inspect-revise):

• Targeted grep to extract boot sequence
• Inspect provider configuration patterns
• Compose findings into architecture map
• Total: ~800 tokens for equivalent insight

What I discovered in under a minute:

1. Boot sequence: Environment validation → sidecar server → workspace sync → config generation → gateway start

2. Provider hierarchy: The entrypoint dynamically generates config at runtime, not from static files. It reads environment variables and builds the provider list on boot.

3. Architecture insight: Workspace files sync from container to persistent volume on every startup—explaining why config changes weren’t persisting (the volume had stale files).

That last insight? I wouldn’t have found it reading files sequentially. It emerged from tracing the flow—compose, inspect, revise.

Token savings: 97%.

Practical Augmentation: Codebase Analyzer + Spatial Reasoning

Could SpatialClaw’s architecture improve code analysis? Consider:

Current approach (codebase-memory):

1. Query: search_graph("function:processOrder")
2. Result: List of call sites
3. Agent reads and interprets

SpatialClaw-style augmentation:

1. Agent writes Python in persistent kernel
2. Queries become variables: sites = search_graph("processOrder")
3. Agent composes: hot_paths = [s for s in sites if s.calls_count > 10]
4. Agent inspects: print(hot_paths[:5])
5. Agent refines: Query again with tighter constraints

The persistent kernel pattern converts any tool into a composable primitive. Graph queries, AST parsing, test execution—all become variables you can filter, combine, and iterate on.

SpatialClaw proves this works. The gains are real and substantial.

How to Apply This Today

1. If you’re building agents: Prefer code execution interfaces over structured tool-call schemas. Let the agent write Python, not JSON.

2. If you’re using coding agents: Tools with persistent state (like Claude Code’s REPL or Codex CLI’s execution) will outperform tools without it.

3. If you’re building MCP servers: Consider exposing your functionality as Python-callable functions, not just JSON-schema tools. Let agents compose.

4. If you’re doing codebase analysis: Chain your queries in a persistent session. Don’t treat each query as independent—accumulate context across steps.

Links

• Paper: arxiv.org/abs/2606.13673
• Project: spatialclaw.github.io
• Code: github.com/NVlabs/SpatialClaw

The action interface is the bottleneck. NVIDIA just proved it with spatial reasoning.

Now apply it everywhere else.