Stop Taking a Helicopter
to the Grocery Store

Here's a pattern we see constantly: a team builds a RAG pipeline, and every step — chunking, embedding, reranking, entity extraction — gets routed through an LLM API. GPT-4o for chunking. OpenAI embeddings for vectors. Another LLM call for query rewriting. Maybe Cohere for reranking.

It works. But it's the equivalent of taking a helicopter to the grocery store. You'll get there, sure — but you're burning jet fuel for a task that a bicycle handles better.

LLMs are general-purpose reasoning engines. They're extraordinary at understanding context, generating text, and handling ambiguous instructions. But most of the steps in an AI pipeline aren't ambiguous. Chunking is chunking. Embedding is embedding. Reranking is reranking. These are well-defined, narrow tasks — and there are models specifically trained to do each one faster, cheaper, and often better than a general-purpose LLM.

We're talking about models in the 30 million to 300 million parameter range — roughly 100x to 1,000x smaller than models like GPT-4o or Claude. They're typically trained for a single task: generating embeddings, splitting text into semantic chunks, re-scoring search results, or extracting named entities.

What makes them powerful isn't their size — it's their specificity. A 128 MB embedding model that was trained on billions of sentence pairs for one job (producing good vectors) will outperform a 70B parameter model that's trying to do everything. It's the same reason a Formula 1 car beats a helicopter on a racetrack.

Let's walk through each step of a typical AI pipeline and see what happens when you swap the LLM for a purpose-built model.

Embedding models convert text into dense vectors that capture semantic meaning. This is arguably the most important step in any search or RAG pipeline — and it's also the most wasteful place to use an LLM.

Models like bge-small-en-v1.5 (128 MB, 384 dimensions) or nomic-embed-text-v1.5 (1.2 GB, 768 dimensions) are specifically trained for producing high-quality embeddings. They consistently score at or near the top of the MTEB leaderboard — the standard benchmark for embedding quality — despite being a fraction of the size of general LLMs.

For multimodal use cases, Termite also ships clip-vit-base-patch32 (584 MB) for unified image and text embeddings, and clap-htsat-unfused (2 GB) for audio.

Most teams either chunk by fixed token counts (which breaks mid-sentence) or use an LLM to identify split points (which costs $2.50 per million tokens on GPT-4o). There's a better option: small transformer models trained specifically for document segmentation.

Termite's chonky-mmbert-small-multilingual-1 (570 MB) understands document structure and splits at natural paragraph and topic boundaries. It supports multiple languages and runs in 2-5ms per document — fast enough to chunk millions of documents in hours on a single machine.

Reranking is the single highest-leverage improvement you can make to a RAG pipeline. A reranker takes your initial search results and re-scores each one by looking at the full query-document pair together (cross-attention), catching relevance signals that embeddings alone miss.

Cohere's Rerank API costs $2.00 per 1,000 searches. Termite's mxbai-rerank-base-v1 (713 MB) provides comparable quality at zero marginal cost. At 100K searches per month, that's $200/month saved.

The real advantage isn't just cost — it's latency. A local reranker adds 5-10ms to your pipeline. An API call adds 100-300ms plus network variability. For user-facing search, that difference is the line between "snappy" and "sluggish."

NER extracts structured information (people, organizations, locations, dates) from unstructured text. This is incredibly useful for building faceted search, populating knowledge graphs, or enriching documents with metadata before indexing.

Instead of prompting GPT-4o with "extract all entities from this text" at $2.50/M tokens, Termite's gliner2-base-v1 (798 MB) handles it locally with customizable label sets. For relationship extraction (who works at what company, what event happened where), rebel-large (2.9 GB) maps entities and their relationships in a single pass.

When a user searches for "quarterly results," they might mean financial reports, academic grades, or sports scores. Query rewriting generates multiple variants to improve recall. Termite's flan-t5-small-squad-qg (569 MB) handles question generation and paraphrasing locally, while pegasus-paraphrase (4.5 GB) offers higher-quality paraphrasing for more demanding use cases.

Let's put real numbers on it. Here's what a typical RAG pipeline costs when every step runs through an LLM API — versus running the same pipeline with task-specific models on Termite.

Termite is a local ML inference server that runs ONNX models with an Ollama-compatible API. Think of it as Ollama, but for all the other models in your AI pipeline — the ones that aren't LLMs. Embeddings. Chunking. Reranking. NER. OCR. Rewriting.

It ships with a curated model garden of task-specific models, all optimized for CPU inference with FP16 and INT8 quantization options. One command to pull a model, one command to run it.

Every model in the Termite garden runs locally via ONNX Runtime, with optional XLA and Go backends. Browse the full catalog at antfly.io/termite/models.

Termite is designed to slot into existing workflows. If you're already using Antfly, small models are built directly into the database — just configure your index with an embedder, chunker, and reranker. If you're using Termite standalone, it exposes an Ollama-compatible API that any HTTP client can call.

Small models run automatically as part of your index configuration. No separate service to manage.

Run Termite as an independent service and call it from any language or framework.

This isn't an argument against LLMs. It's an argument for using the right tool for the job. LLMs are unmatched for tasks that require reasoning, creativity, and handling ambiguity.

Termite is a single binary. Install it, pull the models you need, and start serving. No Docker, no GPU, no configuration files.