I cut a RAG pipeline from 90s to 4s without changing the model

TL;DR: A RAG pipeline serving research questions was taking 90 seconds per query. The obvious fix is to swap the model or scale the infrastructure. Neither was the problem. Three pipeline bugs (over-retrieval, a dead cache, and uncontrolled chunk bloat) accounted for 95% of the latency. Each had a one-line fix. Here’s how to find them in your own pipeline.

Key takeaways:

• Vector search returns chunks, not documents. Deduplicate by source before you send to the LLM. It is a one-line fix that cuts token count by 30-50%.
• A semantic cache that uses different KNN scoring from the main vector store is dead code. Align the query construction and log hits at Info level, not Debug.
• Chunk count and latency are not linearly related. Going from 10 chunks to 3 cuts time-to-first-token by 60-80% because attention cost scales quadratically with input length.
• Debug in one order: log token count before synthesis, count LLM calls per request, check cache hit rate, then profile retrieval.

How does RAG latency accumulate?

A RAG pipeline has five stages. Every stage adds time.

Query -> Embed -> Search vector DB -> Assemble context -> LLM generates answer

Stage five gets all the blame. It’s slow, expensive, and everyone assumes it is the bottleneck. But the four stages before it routinely account for 60-80% of total latency.

The reason is straightforward: every stage multiplies work. A query that returns 10 chunks sends 10x the tokens to the LLM than a query that returns 3 chunks. Those extra tokens do not add linearly to generation time. Attention cost scales roughly quadratically with input length. Double the context and you roughly quadruple the compute.

That’s the wrong place to start.

A kitchen works the same way. If the head chef (the LLM) is slow, you assume she needs more training or better tools. But check the prep line first. If four line cooks are all peeling the same carrot, the bottleneck is not the chef. It’s the prep workflow.

What is the first latency bug: over-retrieval of redundant chunks?

This is the most common bug and the easiest to fix.

Here’s how it works:

Vector search returns individual chunks, not whole documents. If your corpus uses paragraph-level chunking and a page has 50 paragraphs, the vector store happily returns 5 top-scoring chunks from the same page. They all have identical or near-identical relevance scores because they come from the same source and match the same query.

The LLM then reads 5 variations of the same content. It pays attention cost on every one. The responses get no better from the redundancy, but the latency grows with each extra chunk.

An engineer named Jamie Maguire hit exactly this in a .NET MCP documentation search. Top-8 chunks retrieved, 5 from the same document URI. The token count before synthesis was 19,700. Latency: 18-25 seconds per query.

The fix:

# Before: top N chunks, possibly all from the same source
chunks = vector_search(query).take(8)

# After: one chunk per source document, then cap
chunks = vector_search(query) \
    .group_by(lambda c: c.document_uri) \
    .select_many(lambda g: g.take(1)) \
    .take(8)

One pipeline operation. Token count dropped from 19,700 to 12,266. The LLM now saw 3 distinct sources instead of 5 paragraphs of the same page.

Why it works:

The deduplication forces the LLM to work with diverse evidence. Each chunk represents a different source, so the synthesis covers more ground. The redundancy was not helping answer quality. It was only adding latency.

The catch:

If your documents have genuinely distinct sections (API reference, quickstart, tutorial all in the same file), take(1) per source might drop useful diversity. Use take(2) per group for multi-section documents. The exact number matters less than the principle: deduplicate before you pay attention cost.

What is the second latency bug: semantic cache alignment?

Many RAG pipelines add a semantic cache. Store query and answer pairs. When a new query is similar to a cached one, return the cached answer. Skip the entire pipeline.

Here’s how it works:

Many RAG pipelines add a semantic cache. They store query and answer pairs. When a new query matches a cached one, the cache returns the stored answer. Skip the entire pipeline.

The cache uses a similarity threshold, say 0.92. A new query gets embedded, compared against cached entries, and if the score exceeds the threshold, the cache returns the answer.

The problem is that the cache’s search and the main vector store’s search use different scoring methods. In Jamie’s pipeline, the main search used a raw kNN JSON API call. The cache used knn() nested inside query() which applies different score normalization. The 0.92 threshold was mathematically impossible to hit. The cache returned zero hits for every query.

It was dead code. Running on every request. Adding latency. Never skipping a single pipeline execution.

The fix:

Two things.

1. Align the KNN query construction. Both paths must use the same API call so scores are comparable.
2. Log cache hits at Information level, not Debug. A cache that only logs at Debug cannot be monitored in production. You cannot tune what you cannot see.

Why it works:

Once the scoring is aligned, the cache starts hitting. For workloads with repeated query patterns (FAQ bots, customer support, documentation search), semantic caching can skip the pipeline entirely for 30-70% of requests. It’s sub-100ms instead of multiple seconds.

The catch:

Semantic caching only helps if your users repeat themselves. Open-ended research queries where every question is unique will never hit the cache regardless of alignment. If your workload has low query repetition, skip caching and focus on the other two bugs.

What is the third latency bug: chunk count management?

Every chunk you send to the LLM costs tokens and attention. The relationship is not linear.

Here’s how it works:

The LLM computes attention between every pair of tokens in its input. Double the input length and you roughly quadruple the attention matrix size. That means the generation step’s cost grows much faster than the chunk count.

A Towards AI case study showed what happens when you control for this. Going from 10 chunks to 3 dropped time-to-first-token from 1.2 seconds to 0.28 seconds. Total latency from 2.3 seconds to 0.45 seconds. Token count down 60%.

Three techniques manage chunk count without losing answer quality:

1. Score threshold filtering. Only keep chunks above a relevance score, say 0.75. It’s a one-line filter that drops noisy chunks before they enter the LLM.

2. Metadata pre-filtering. Before vector search, filter by document type, date range, or category. This reduces the search space so the vector store returns better candidates. A chunk about deployment config should not compete with a chunk about pricing when the query is about deployment.

3. Adaptive chunk count. Not every query needs the same number of chunks. A simple factual question needs 2-3 chunks. A complex research question might need 5-8. A query classifier (a lightweight classifier, not an LLM) chooses the count based on query complexity.

Why it works:

Queries rarely need 10 chunks. They need 3-4. The extra chunks add noise, not signal. Noise forces the LLM to spend attention on irrelevant content, which adds latency without improving answer quality. Reducing chunks to what the query needs is the single biggest latency optimization because it addresses all five pipeline stages at once.

The catch:

Too few chunks and the LLM hallucinates. It does not have enough evidence to ground its answer. The right number depends on your use case. Start with a generous cap (8-10 chunks), log what gets used per query type, then tighten over time. You should never guess the number. Measure it.

How do you debug a slow RAG pipeline?

The three bugs above share a common methodology. It works regardless of your stack.

Step 1: Log token count before the synthesis call.

This is the highest-value diagnostic metric. In every case study examined, the token count was the smoking gun. Jamie’s 19,700 tokens. The Towards AI case study’s 10 chunks. The number tells you immediately whether you have an over-retrieval problem or a generation problem.

logger.info("Context for synthesis: %d chunks, %d tokens, %d sources",
    len(chunks), token_count, source_count)

Step 2: Count LLM calls per request.

A single user query should trigger one LLM call. Jamie’s pipeline had three. One code path bypassed the chunk assembler entirely and sent 62,860 tokens straight to the model. This was invisible without per-call logging.

Add a unique request ID. Log every LLM call with its parent request ID. Count them.

Step 3: Check cache hit rate.

If you have a semantic cache and you do not know its hit rate, assume it is zero. Add a counter metric at Information level and watch it over a day of production traffic. If it stays at zero, your scoring is misaligned (bug 2) or your queries are too diverse (in which case, remove the cache).

Step 4: Profile retrieval diversity.

Run a sample of queries and log how many unique source documents contribute to the top-N chunks. If the average is below half of N (e.g. 3 unique sources out of 8 chunks), you have a deduplication problem (bug 1).

FAQ

What is the biggest cause of RAG pipeline latency? Over-retrieval of redundant chunks. Vector search returns chunks, not documents. When multiple chunks from the same document hit the top-N results, the LLM pays attention cost on duplicate content. Deduplicating by source is a one-line fix that cuts 30-50% of token count.

Can caching fix RAG latency? Only if the cache uses the same scoring as your main vector store. Most semantic caches use different KNN query constructions, making the threshold unreachable. Fix the alignment first, then measure hit rate. If query diversity is too high, caching will not help.

Do I need a smaller or faster model to reduce RAG latency? Usually not. The pipeline stages before the model (embedding, search, chunk assembly, context prep) account for 60-80% of total latency. Fixing those is free. Swapping the model costs API migration work and may reduce answer quality.

How many chunks should I send to the LLM? Start at 8-10, log what gets used, and tighten. Most queries need 3-4. A 10 to 3 reduction cuts time-to-first-token by 60-80% because attention cost scales quadratically with input length.

What is the first thing I should debug? Log the token count before the synthesis call. Then count LLM calls per request. Then check cache hit rate. Then profile retrieval diversity. This order catches all three bugs in sequence.

Related Posts

• Cache your agent’s reasoning, not its responses. Pipeline-aware caching at intermediate checkpoints hits 83% reuse rate.
• Your agent’s memory is making it a yes-man. When the problem is not retrieval speed but what you do with what you retrieved.
• Multi-agent LLMs can’t explore each other. Another case where the bottleneck is not the model but the coordination layer around it.

---

This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]