RouterRetriever: Why Your RAG System Needs Multiple Embedding Experts

By Prahlad Menon 5 min read

Your RAG system probably uses a single embedding model — maybe OpenAI’s text-embedding-3-large, Cohere’s embed-v3, or an open-source model like BGE or E5. These models work reasonably well across domains, but they consistently underperform specialized models when retrieving from domain-specific corpora.

Here’s the painful truth about RAG: if you want to use a different embedding model, you have to re-embed your entire document corpus. Your vector database is married to your embedding model. Want to try a better medical embedding model? Re-index everything. Want domain-specific embeddings for legal vs. technical docs? Maintain multiple vector databases.

RouterRetriever solves this elegantly: keep your document index frozen, and route queries through domain-specific adapters instead.

The Problem with General-Purpose Embeddings

Most retrieval systems train on MSMARCO, a large general-domain dataset. This produces embeddings that work “okay” everywhere but excel nowhere. When you test these models on specialized domains — legal documents, medical literature, scientific papers — they consistently lose to domain-specific models.

The traditional solutions have significant drawbacks:

  • Multi-task training: Combine MSMARCO + domain data. Expensive, requires full retraining when adding domains, and performance often degrades as you add more domains.
  • Domain adaptation: Still stuck with one model trying to do everything.
  • Instruction-following retrievers: Encode domain hints in prompts, but the single model bottleneck remains.
  • Multiple specialized models: Need separate vector databases for each — expensive and operationally complex.

RouterRetriever: The Key Insight

Here’s the clever part that makes this work:

Your documents are indexed using the frozen base encoder only. The vector database never changes.

Only queries are transformed — at inference time, the query goes through base encoder + selected LoRA expert, producing a domain-optimized query embedding that still matches against your existing document embeddings.

This works because the LoRA experts are trained to produce query embeddings that retrieve better from the same document corpus embedded by the base model. The document side stays frozen; the query side adapts.

Architecture

                        INDEXING (one-time)
                        ==================
Documents ──→ [Base Encoder] ──→ Vector DB
                    ↓
              (embeddings stored)
              
              
                        QUERY TIME
                        ==========
                              ┌──────────────┐
                              │ Pilot Embeds │ ← (pre-computed centroids)
                              └──────┬───────┘
                                     │ similarity
                                     ↓
Query ──→ [Base Encoder] ──→ Route ──→ Select Expert
                 │                         │
                 │                         ↓
                 └──────────────→ [Base + LoRA Expert]
                                         │
                                         ↓
                                  Query Embedding
                                         │
                              search     ↓
                Vector DB ←──────────────┘
                    │
                    ↓
              Retrieved Docs

The system has three components:

  1. Frozen base encoder: A general-purpose model (Contriever in the paper). Documents are embedded with this — never re-indexed.
  2. Domain-specific LoRA experts: Lightweight adapters (~1M params each, 0.5% of base model) trained on domain-specific query-document pairs.
  3. Pilot embedding library: Pre-computed representative embeddings used for routing.

How Routing Works

The routing uses embedding similarity — no learned classifier needed:

def route_query(query, pilot_embeddings, experts):
    # 1. Encode query with base model (same encoder used for docs)
    query_emb = base_encoder(query)
    
    # 2. Calculate similarity to each expert's pilot embeddings
    scores = {}
    for expert_name, pilot_embs in pilot_embeddings.items():
        # Average similarity to this expert's representatives
        scores[expert_name] = mean([
            cosine_sim(query_emb, pilot) 
            for pilot in pilot_embs
        ])
    
    # 3. Select best expert
    best_expert = max(scores, key=scores.get)
    
    # 4. Generate final embedding with selected expert
    # This embedding will retrieve from docs embedded by base_encoder
    return base_encoder_with_lora(query, expert=best_expert)

Why This Works (The LoRA Magic)

The key insight is how the LoRA experts are trained:

  1. Training data: Query-document pairs from a specific domain (e.g., medical queries + medical docs)
  2. Training objective: Contrastive loss — make query embedding close to its relevant document embedding
  3. Document embeddings: Generated by the frozen base encoder (same as production index)
  4. Query embeddings: Generated by base encoder + LoRA adapter

The LoRA adapter learns to transform queries such that they match better against documents embedded by the plain base encoder. It’s learning “how should I ask this question so the existing index understands me better for this domain?”

Pilot Embeddings: Deciding Which Expert to Use

The pilot embeddings are computed once during setup:

  1. For each training instance, find which expert performs best on it
  2. Group instances by their best-performing expert (not by original domain!)
  3. Compute centroid embedding for each group using the base encoder
  4. Store these centroids as “pilot embeddings”

At inference time, the query embedding (from base encoder, no LoRA) is compared against pilot embeddings to route to the right expert. Then the query is re-encoded with that expert’s LoRA.

Results: +3.2 Points Over Multi-Task Training

On the BEIR benchmark (the standard for evaluating retrieval across domains):

ApproachnDCG@10
MSMARCO-only (baseline)—
RouterRetriever vs MSMARCO+2.1
RouterRetriever vs Multi-task+3.2
RouterRetriever vs other routing+1.8

Key Finding: Scales Better Than Multi-Task

This is important for production systems. As you add more domains:

  • Multi-task training: Performance degrades after ~4-5 domains (catastrophic forgetting)
  • RouterRetriever: Performance keeps improving with more experts

The paper shows that even on domains without a specific expert, RouterRetriever outperforms the general-purpose baseline. The routing mechanism finds the “closest” expert automatically.

Zero-Shot Generalization

Tested on 7 BEIR datasets that had no trained experts at all — RouterRetriever still outperformed both MSMARCO-only and multi-task baselines. The routing selects whichever existing expert happens to be closest to the unseen domain.

Practical Implementation

The authors released full code and model checkpoints.

Code

GitHub: github.com/amy-hyunji/RouterRetriever

HuggingFace: huggingface.co/amy-hyunji-lee/RouterRetriever

Setup

git clone https://github.com/amy-hyunji/RouterRetriever
cd RouterRetriever
pip install -r requirements.txt

Training a New Expert

# 1. Preprocess your domain data
python preprocess.py \
  --datasets your_domain \
  --beir_dir beir_datasets \
  --save_dir beir_datasets

# 2. Train LoRA expert
python train.py --config config/your_domain.json

Building Pilot Embeddings

# Score each instance with all experts
python get_scores.py \
  --dataset your_domain \
  --save_dir outputs/all_score.json \
  --gatenames expert1/expert2/expert3

# Compute pilot embeddings (centroids)
python get_pilot_embeddings.py \
  --datasets expert1/expert2/expert3 \
  --score_path outputs/all_score.json \
  --beir_dir beir_datasets \
  --save_dir outputs/pilot_embs.json

Inference

python test.py \
  --pilot_embs_path outputs/pilot_embs.json \
  --datasets test_domain \
  --beir_dir beir_datasets \
  --save_dir outputs/score.json

When Should You Use This?

Good fit:

  • Multi-domain retrieval (legal + medical + financial in one system)
  • Systems that add new domains over time (just train new LoRA, no reindexing!)
  • When you have domain-specific training data (query-document pairs)
  • Enterprise RAG with diverse document types
  • When re-embedding millions of documents is too expensive

Probably overkill:

  • Single-domain applications (just train one domain-specific model)
  • No domain-specific training data available
  • Extremely latency-sensitive (routing adds one extra forward pass)

The Bigger Picture: Query-Side Adaptation

RouterRetriever is part of a broader insight: you can often adapt queries instead of documents.

This same principle appears in:

  • HyDE (Hypothetical Document Embeddings): Transform query into a fake document before searching
  • Query2Doc: Expand queries with LLM-generated context
  • Instruction-following retrievers: Modify query behavior via prompts

RouterRetriever takes this further with parametric adaptation (LoRA weights) rather than input modification. The document index stays frozen; the query transform adapts.

For RAG systems handling diverse domains, this is worth exploring. The code is open source, the approach is well-validated (AAAI 2025), and the architecture solves a real pain point: no more re-indexing when you want domain-specific retrieval.

Paper: arXiv:2409.02685

Code: github.com/amy-hyunji/RouterRetriever

Models: huggingface.co/amy-hyunji-lee/RouterRetriever

Base model: Contriever

Benchmark: BEIR