Gigatoken: The 1000x Faster Tokenizer That Makes HuggingFace Look Like Dial-Up

If you’ve ever tokenized a large corpus for RAG indexing or fine-tuning, you know the pain. HuggingFace tokenizers crawl at tens of MB/s. That 12 GB dataset? Go make coffee. Maybe lunch.

Gigatoken just dropped, and the numbers are absurd: 24.53 GB/s on a dual-socket EPYC server. That’s not a typo. That same 12 GB OpenWebText corpus that takes HuggingFace tokenizers ~8 minutes? Under half a second.

The Benchmarks

On an AMD EPYC 9565 (144 cores):

Tokenizer	Gigatoken	HuggingFace	Speedup
GPT-2	24.53 GB/s	24.8 MB/s	989×
Llama 3/3.1/3.2	22.15 GB/s	48.5 MB/s	457×
Qwen 3	22.16 GB/s	34.2 MB/s	648×
DeepSeek V3/R1/V4	19.69 GB/s	26.2 MB/s	750×

Even on consumer hardware (M4 Max, 16 cores), you’re looking at 8+ GB/s—still 500-1000× faster than HuggingFace depending on the tokenizer.

At these speeds, you could tokenize the entirety of Common Crawl (130 trillion tokens) in about 6.5 hours.

How It Works

The secret sauce is SIMD optimization and aggressive caching:

1. SIMD pretokenization — Instead of farming out regex to a general-purpose engine, Gigatoken uses hand-tuned SIMD (AVX512/AVX2/NEON) to blast through the pretokenization step
2. Pretoken cache hierarchy — If a word has been seen before, look up its tokens instantly. The hard part is managing cache growth on long-tailed distributions
3. Minimal Python overhead — Rust reads data directly, skipping Python’s GIL bottleneck
4. Zero inter-thread communication — Each thread works independently

Drop-In Replacement

The API is designed to swap in without rewriting your pipeline:

import gigatoken as gt

# Wrap your existing HuggingFace tokenizer
hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()

# Use exactly like before, but 100-1000x faster
tokens = tokenizer.encode_batch(["Your text here"])

Or go native for maximum speed:

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(["corpus.txt"], separator=b"<|endoftext|>")
tokens = tokenizer.encode_files(file_source)

Why This Matters

For RAG pipelines: Chunking and tokenizing documents is often the bottleneck before embedding. A 1000× speedup means you can re-index your entire knowledge base in the time it used to take to process a single batch.

For fine-tuning: Dataset prep on large corpora goes from “overnight job” to “grab a coffee.”

For token budgeting: When you need to count tokens across millions of documents to optimize context windows, slow tokenizers make iteration painful.

The Caveats

• SentencePiece tokenizers (Gemma, older Llama) are slower—still faster than HuggingFace, but only 10-20× instead of 500-1000×
• Windows isn’t well-tested; use WSL
• WordPiece not yet supported

Install

pip install gigatoken

Or try without installing:

uvx --with tokenizers gigatoken bench 'openai-community/gpt2' your_data.txt \
  --validate --doc-separator "<|endoftext|>"

---

GitHub: marcelroed/gigatoken
License: MIT
Stars: 2.7k and climbing

This is one of those tools that makes you wonder why it took until 2026. If you’re doing any serious work with text at scale, swap it in.