soul.py + ai_query(): Persistent AI Memory From SQL

What if your LLM could remember things between queries?

Yesterday we covered ai_query() for PII redaction — Databricks’ SQL function that calls LLMs inline with your data. Today: making those LLM calls stateful by connecting soul.py’s persistent memory layer.

soul.py provides persistent memory for AI agents using markdown files and hybrid RAG+RLM retrieval. This post shows how to expose that memory layer via Databricks’ ai_query() — so data engineers can build memory-aware pipelines without leaving SQL.

Why This Matters

Every LLM call starts fresh. The model doesn’t remember:

• What you asked last week
• Decisions made in previous pipeline runs
• Patterns discovered in earlier batches

soul.py fixes this with persistent memory — markdown files and hybrid RAG+RLM retrieval that give agents continuity across sessions. But until now, you needed Python to use it.

By wrapping soul.py in a Databricks Model Serving endpoint, we can call it via ai_query() — giving SQL users access to agents that remember.

The Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Your SQL Query                                │
│   SELECT ai_query('soul-memory-agent', 'What patterns did we see?') │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Serverless SQL Warehouse                          │
│                      ai_query() function                             │
└─────────────────────────────────────────────────────────────────────┘
                              │ calls
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                  Model Serving Endpoint                              │
│              Custom Agent (soul.py HybridAgent)                      │
│                                                                      │
│    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐           │
│    │  SOUL.md    │    │ MEMORY.md   │    │ Vector Idx  │           │
│    │  (Identity) │    │   (Log)     │    │  (Search)   │           │
│    └─────────────┘    └─────────────┘    └─────────────┘           │
│                                                                      │
│    Router → RAG (fast, ~90%) or RLM (thorough, ~10%)                │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     Unity Catalog                                    │
│              (Delta tables for persistence)                          │
└─────────────────────────────────────────────────────────────────────┘

The key insight: soul.py’s memory files (SOUL.md, MEMORY.md) become Delta tables in Unity Catalog. The agent reads and writes to these tables, maintaining state across sessions.

Memory Tables

First, create the backing tables in Unity Catalog:

-- Identity (SOUL.md equivalent)
CREATE TABLE IF NOT EXISTS catalog.soul_memory.identity (
    agent_id STRING NOT NULL,
    soul_content TEXT NOT NULL,
    updated_at TIMESTAMP DEFAULT current_timestamp(),
    version INT DEFAULT 1
) USING DELTA;

-- Memory log (MEMORY.md equivalent)  
CREATE TABLE IF NOT EXISTS catalog.soul_memory.memory_log (
    agent_id STRING NOT NULL,
    timestamp TIMESTAMP DEFAULT current_timestamp(),
    role STRING NOT NULL,  -- 'user', 'assistant', 'system'
    content TEXT NOT NULL,
    metadata MAP<STRING, STRING>
) USING DELTA
PARTITIONED BY (agent_id);

-- Decisions (explicit agent decisions)
CREATE TABLE IF NOT EXISTS catalog.soul_memory.decisions (
    agent_id STRING NOT NULL,
    decision TEXT NOT NULL,
    context TEXT,
    rationale TEXT,
    created_at TIMESTAMP DEFAULT current_timestamp(),
    confidence DOUBLE DEFAULT 1.0
) USING DELTA;

These tables mirror soul.py’s file-based storage:

• identity → SOUL.md (who the agent is)
• memory_log → MEMORY.md (timestamped conversation history)
• decisions → Explicit choices with rationale

The Model Endpoint

Wrap soul.py’s HybridAgent as an MLflow model:

import mlflow
from mlflow.pyfunc import PythonModel

class SoulMemoryAgent(PythonModel):
    """MLflow model that exposes soul.py memory operations."""
    
    def load_context(self, context):
        from soul.hybrid_agent import HybridAgent
        from delta_backend import DeltaMemoryBackend
        
        self.backend = DeltaMemoryBackend(
            catalog="main",
            schema="soul_memory",
            agent_id="default"
        )
        self.agent = HybridAgent(
            provider="anthropic",
            memory_backend=self.backend
        )
    
    def predict(self, context, model_input):
        if isinstance(model_input, dict):
            operation = model_input.get("operation", "ask")
            query = model_input.get("query", "")
            
            if operation == "ask":
                result = self.agent.ask(query)
                return {"answer": result["answer"], "route": result.get("route")}
            
            elif operation == "remember":
                self.backend.log_memory(query, role="user")
                return {"status": "stored"}
            
            elif operation == "recall":
                memories = self.backend.search(query, k=5)
                return {"memories": memories}
            
            elif operation == "decide":
                decision = model_input.get("decision", "")
                rationale = model_input.get("rationale", "")
                self.backend.log_decision(decision, query, rationale)
                return {"status": "decision_logged"}
        
        result = self.agent.ask(str(model_input))
        return result

Deploy this to a Model Serving endpoint, and it becomes callable via ai_query().

SQL Operations

Ask with Memory Context

-- Agent remembers previous interactions
SELECT ai_query(
    'soul-memory-agent',
    '{"operation": "ask", "query": "What patterns have we seen in customer complaints?"}'
) AS response;

The agent retrieves relevant memories before answering. If you asked about customer complaints last week, it remembers.

Store a Memory

-- Explicitly store something to remember
SELECT ai_query(
    'soul-memory-agent',
    '{"operation": "remember", "query": "Customer ID 4521 prefers email over phone contact"}'
) AS status;

Recall Relevant Memories

-- Semantic search over memory
SELECT ai_query(
    'soul-memory-agent',
    '{"operation": "recall", "query": "customer contact preferences"}'
) AS memories;

Log a Decision

-- Record a decision with rationale
SELECT ai_query(
    'soul-memory-agent',
    '{
        "operation": "decide",
        "query": "VIP escalation policy",
        "decision": "Route VIP tickets to senior support within 15 minutes",
        "rationale": "VIP customers have SLA requirements and higher churn risk"
    }'
) AS status;

Pipeline Patterns

Pattern 1: Memory-Aware Ticket Processing

SELECT 
    ticket_id,
    customer_id,
    ai_query(
        'soul-memory-agent',
        concat(
            '{"operation": "ask", "query": "Given customer history, how should we handle: ',
            issue_text,
            '"}'
        )
    ) AS recommended_action
FROM support_tickets
WHERE status = 'new';

The agent considers all previous interactions with that customer, previous similar issues, and decisions made about handling patterns.

Pattern 2: Learning Loop

-- After resolving tickets, teach the agent
INSERT INTO catalog.soul_memory.memory_log (agent_id, role, content)
SELECT 
    'support-agent' as agent_id,
    'system' as role,
    concat(
        'Resolved: ', issue_text, 
        ' | Solution: ', resolution, 
        ' | Satisfaction: ', satisfaction_score
    ) as content
FROM resolved_tickets
WHERE resolved_at > current_timestamp() - INTERVAL 1 DAY;

The agent learns from resolutions. Tomorrow, when similar issues arise, it recommends what worked.

Pattern 3: Cross-Pipeline Memory

-- Pipeline A (ETL) stores a finding
SELECT ai_query('soul-memory-agent', 
    '{"operation": "remember", "query": "Detected 15% data quality issues in vendor feed on 2026-06-18"}'
);

-- Pipeline B (Analytics, later) asks about patterns
SELECT ai_query('soul-memory-agent',
    '{"operation": "ask", "query": "What data quality issues have we seen in vendor feeds this month?"}'
);

Different pipelines share memory. The analytics pipeline knows what ETL discovered.

Pattern 4: Agent Specialization

-- Create a specialized agent identity
INSERT INTO catalog.soul_memory.identity (agent_id, soul_content)
VALUES (
    'compliance-agent',
    '# Compliance Agent

You are a healthcare compliance specialist. You:
- Know HIPAA regulations thoroughly
- Flag potential PHI exposure
- Recommend remediation steps
- Never provide medical advice

When unsure, recommend consulting legal.'
);

Now queries to compliance-agent get compliance-focused responses.

How soul.py Routing Works

soul.py’s HybridAgent routes queries automatically:

• ~90% go to RAG — Fast vector search, sub-second, good for direct recall (“What did we decide about X?”)
• ~10% go to RLM — Recursive synthesis, thorough, good for complex reasoning (“Analyze all customer complaints and identify themes”)

The router is a small LLM call that classifies query complexity. You don’t configure it — it just works.

Delta vs File-Based Memory

Feature	soul.py (Files)	soul.py + Delta
Storage	MEMORY.md, SOUL.md	Delta tables
Query interface	Python	SQL (ai_query)
Collaboration	Git	Unity Catalog RBAC
Scale	Single machine	Distributed
Versioning	Git history	Delta time travel
Search	Local Qdrant	Databricks Vector Search

The Delta backend is better for team environments where multiple pipelines share memory. The file backend is better for local development and Git-versioned agents.

What You Get

1. Agents that learn from data pipelines — Every batch run can teach the agent something
2. Cross-session memory — The agent remembers last week’s decisions today
3. SQL-native interface — Data engineers don’t need Python notebooks
4. Hybrid retrieval — Fast for simple queries, thorough for complex ones
5. Unity Catalog governance — RBAC on who can read/write agent memory

The Bigger Picture

Yesterday’s post showed ai_query() as a stateless LLM call — transform data, move on. Today’s pattern makes it stateful. The agent accumulates knowledge over time.

This is the difference between a calculator and a colleague. A calculator gives you the same answer to the same question every time. A colleague remembers what you discussed last week and builds on it.

---

Code: The full skill with Delta backend implementation is available in menonpg/databricks-contrib

soul.py: github.com/menonpg/soul.py

ai_query() Docs: Databricks SQL Functions