--- title: "How DeepSeek cuts agent response latency by 60% with speculative decoding" canonical: "https://agenticup.dev/posts/dspark-speculative-decoding/" pubDate: "2026-06-27T00:00:00.000Z" description: "DeepSeek's DSpark framework uses semi-autoregressive generation and confidence-scheduled verification to accelerate LLM inference by 60-85% with zero quality loss. The training code is now open source." tags: [speculative-decoding, inference-optimization, deepseek, latency, agent-architecture, dspark] --- **TL;DR:** Speculative decoding accelerates LLM inference by having a small draft model propose tokens and a large target model verify them in parallel. DeepSeek's DSpark framework adds semi-autoregressive generation (fixes suffix decay) and confidence-scheduled verification (avoids wasted compute). In production on DeepSeek-V4, it gives 60-85% per-user speedup with zero quality loss. The training code is now open source in the [DeepSpec repository](https://github.com/deepseek-ai/DeepSpec). I was optimizing an agent loop last month. Each turn required around four tool calls. Each tool call needed a 200-token generation. At that depth, the user felt every millisecond. The fix was speculative decoding. Not a new model. Not a smaller model. A technique that makes the same model generate tokens faster without changing the output distribution. DeepSeek open-sourced their training code for it. > **Key takeaways:** > - Speculative decoding generates multiple tokens per forward pass with zero quality loss > - DSpark's semi-autoregressive architecture fixes suffix decay in parallel drafters > - Confidence-scheduled verification prunes low-confidence tokens to save compute > - 60-85% per-user speedup on DeepSeek-V4 production traffic > - The DeepSpec repo lets you train draft models for your own deployment ## The latency bottleneck in agent loops Every agent loop has a fundamental rhythm: perceive, decide, act. The decide step is a generation call. In a multi-turn agent, that call happens multiple times per user request. Each generation call is autoregressive. One token at a time. One forward pass per token. If the output is 200 tokens, that is 200 sequential passes through a model with hundreds of billions of parameters. The GPU use during this process is terrible. The model is memory-bandwidth-bound for most of the generation. The compute hardware sits idle while waiting for the next token to move through the memory bus. The kitchen equivalent is a single chef cooking one dish at a time from start to finish. The stove is hot. The pans are ready. But the chef plates each dish entirely before starting the next. The throughput limits by the slowest sequential path. Speculative decoding changes this. > **Tradeoff:** Speculative decoding works best when the draft model is a good approximation of the target. If the draft is too weak, most tokens get rejected and you are slower than baseline. The draft-to-target quality gap is the binding constraint. ## How speculative decoding works The core insight is simple. A small draft model proposes a block of candidate tokens. The large target model verifies the entire block in a single parallel forward pass. Tokens that match the target distribution get accepted. The first mismatch triggers rejection of all subsequent tokens. One bonus token appends for free. The math is clean. Because verification uses rejection sampling with the target model's own distribution, the output distribution stays exactly. There is no quality loss. This is not quantization or pruning. It is a pure latency optimization. The speedup depends on three levers: - Draft faster , lower the draft model's latency per block - Draft better , increase the acceptance rate so more tokens survive verification - Verify smarter , skip verification for tokens that will likely fail rejection DSpark addresses all three. ## Problem 1: Suffix decay in parallel drafters Early drafters were autoregressive. They generated tokens one at a time, conditioning each on the previous. This gave high acceptance rates but the drafting latency grew linearly with block size. Fast drafters forced small blocks. Parallel drafters changed this. DFlash produces all draft tokens in a single forward pass, making latency nearly independent of block size. You can draft 16 tokens as fast as 2. But parallel drafters have a blind spot. Each position predicts independently. When the context has multiple plausible continuations , "of course" and "no problem" , a parallel drafter can produce "of problem" or "no course." Each position marginalizes over all possible predecessors not conditioning on the one sampled. Acceptance rate decays rapidly at later positions. DFlash wastes its long blocks. DSpark's fix is a semi-autoregressive architecture. A parallel backbone (DFlash) does the heavy compute. A lightweight sequential head injects dependency between draft tokens. The sequential head has two variants. A Markov head uses a low-rank bigram matrix (256-dim) to model the transition from the previous token. An RNN head maintains a recurrent state that captures the full prefix history within a block. Both are cheap. The sequential module adds minimal latency because it operates on one token at a time in a tight loop with a small embedding table. The parallel backbone continues to dominate the draft latency. The result: DSpark improves macro-average accepted length over Eagle3 by 30% and over DFlash by 18% across Qwen3-4B, 8B, and 14B. > **Tradeoff:** The RNN head captures more context than the Markov head but adds more sequential latency. For small blocks (5 tokens or fewer), the Markov head is sufficient. For larger blocks, the RNN head pays off. ## Problem 2: Wasted verification on low-confidence tokens Drafting more tokens does not automatically translate to speedup. Every token submitted for verification occupies batch capacity in the target model. If trailing tokens have a high rejection risk, verifying them is pure waste. The waste depends on two factors. First, structured text like code has higher acceptance rates than open-ended chat. Second, under light load, extra verification is nearly free. Under heavy load, every rejected token steals capacity from other active requests. DSpark's fix is confidence-scheduled verification. A confidence head predicts per-position acceptance probability. A hardware-aware scheduler prunes low-confidence suffix tokens based on current system load. The confidence head is a linear projection with a sigmoid, trained end-to-end. It takes the backbone hidden state and the previous token's Markov embedding. The output is a scalar estimating the conditional probability that the draft token at position k survives verification, given that all preceding tokens passed. The scheduler uses these probabilities to compute expected acceptance length for each active request. It greedily allocates verification budget to the most promising prefixes. Under light load, verify everything. Under heavy load, drop trailing tokens with high rejection risk. The scheduler adapts to both data domain and system load in real time. In DeepSeek-V4 production deployment, this mechanism is what prevents the throughput cliff. Under strict SLA (120 tok/s/user for Flash, 50 for Pro), the MTP-1 baseline collapses. DSpark maintains robust throughput. ## What this means for agent latency An agent loop that makes 4 tool calls at 200 tokens each generates 800 tokens per user turn. At autoregressive speeds, that is 800 sequential forward passes. With DSpark at an average acceptance of 5 tokens per cycle (conservative for production), each forward pass produces 5 tokens. The same 800 tokens require 160 verification passes plus drafting overhead. The total per-user speedup is 60-85%. Now apply this to SPAgent, a system that specifically targets agent search latency. SPAgent reduces LLM inference time by 23.8% and action execution time by 29.4% using speculative decoding. The combination of DSpark's draft quality and SPAgent's agent-aware scheduling would be even stronger. For a production agent, speculative decoding is not exotic. It is a deployment prerequisite. If the user is waiting for tokens, you are losing them. ## The open-source ecosystem DeepSeek released the DSpark checkpoints for V4-Flash and V4-Pro. More importantly, they open-sourced DeepSpec, a training repository that includes Eagle3, DFlash, and DSpark. The repo contains training code, evaluation benchmarks, and data preparation utilities. This means you can train draft models for your own target model. The technique is not locked to DeepSeek. The broader ecosystem is moving fast. LMSys published a next-gen speculative decoding survey. Google released speculative cascades. vLLM and SGLang both support speculative decoding in production. The speculators ecosystem on Red Hat includes DFlash support. This is the year speculative decoding becomes a standard deployment layer, not a research experiment.

Agent mode: Speculative decoding decouples draft generation from target verification using a small draft model. DSpark's semi-autoregressive architecture fixes parallel drafter suffix decay with a lightweight sequential head. For agent loops, this means 60-85% faster generation per turn with zero output quality change. The DeepSpec repository provides training code adaptable to any target model.

## FAQ > **Does speculative decoding work with any model?** > It works with any model pair where the draft model approximates the target distribution. DeepSeek's DSpark works for V4-Flash and V4-Pro. The DeepSpec repo lets you train draft models for other targets. > > **Can speculative decoding make things slower?** > Yes. If the draft model is too weak, most tokens get rejected and the overhead of drafting plus verification exceeds the baseline. The draft-to-target quality gap is the binding constraint. > > **How does this compare to quantization or pruning?** > Quantization and pruning change the model weights and can reduce quality. Speculative decoding preserves the exact output distribution. They are complementary , you can quantize the target model and use speculative decoding on top. > > **Is DSpark available in open-source inference engines?** > DSpark specifically is new and tied to DeepSeek-V4. But the broader speculative decoding ecosystem (Eagle3, DFlash) runs in vLLM, SGLang, and the LMSys stack. The DeepSpec training code gives you the tools to create your own. ## Related Posts - [How to host an AI agent: a beginner's guide](/posts/how-to-host-ai-agent-beginners-guide/) , Speculative decoding in production deployment - [The Vertical Agent Method: ship AI agents in 14 days](/posts/the-vertical-agent-method-framework/) , Framework for scoping and shipping production agents --- ## References - Cheng, X. et al. (2026). [DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation](https://arxiv.org/abs/2603.09906). DeepSeek-AI / Peking University. - Chen et al. (2026). DFlash: Parallel speculative decoding. DeepSeek-AI. - Li et al. (2026b). Eagle3: Autoregressive speculative decoding with TTT. - Huang, Z. et al. (2025). [SPAgent: Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design](https://arxiv.org/abs/2511.20048). Up to 1.65x end-to-end speedup for agent search. - [DeepSpec Repository](https://github.com/deepseek-ai/DeepSpec) , MIT-licensed training code for Eagle3, DFlash, and DSpark. - [SpecForge](https://arxiv.org/abs/2603.18567) , Training speculative decoding models with SGLang. - [LMSys Blog](https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/) , Next-generation speculative decoding survey. - [Google Research: Speculative Cascades](https://research.google/blog/speculative-cascades-a-hybrid-approach-for-smarter-faster-llm-inference/) , Hybrid approach for faster LLM inference. - [The Vertical Agent Method: ship AI agents in 14 days](/posts/the-vertical-agent-method-framework/) , Framework for scoping and shipping agents. --- This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at hello@agenticup.dev