The Real Meaning of AI Inference Acceleration: Can We Make It 10x Faster Without Changing the Model’s Intelligence?
From the Mechanics of Accelerated Inference Serving to DSpark, vLLM, and LithosAI—Technical Truths That Matter More Than Token Generation Speed
Introduction: Making AI Smarter and Making AI Faster Are Two Different Problems
While comparing AI models recently, I noticed an intriguing phenomenon.
The output speeds of various API providers serving the same family of models differed substantially. While some services generated dozens of tokens per second, others reached hundreds per second, and in certain benchmarks, even exceeded 1,000 tokens per second.
Does this disparity arise simply because their GPU performance differs?
Or is it because altering the very method of executing the model makes significantly faster inference possible without sacrificing the model’s intelligence?
The technology discussed here falls under the domain commonly referred to as accelerated inference serving or inference acceleration.
The core principle is not shrinking the model or replacing it with a lower-capability alternative. It is about keeping the weights of the already trained model intact while optimizing the inference engine and the way hardware is utilized.
Of course, not all acceleration techniques guarantee the exact same quality as the original. Lowering precision through quantization or approximating certain computations may boost speed, but it introduces the possibility that output results will deviate.
Conversely, certain inference acceleration techniques utilizing proper verification algorithms can accelerate speed while preserving the output probability distribution of the original model.
So how much acceleration is actually achievable in practice? And how are inference providers such as LithosAI, Baseten, and DeepInfra leveraging these technologies?
This article examines these questions from an engineering perspective.
1. What You Need to Know First: Why Are LLMs Slow?
Large language models typically generate the next token one by one.
For example, suppose the model outputs the following sentence:
Optimizing the inference engine can increase the model’s response speed.
The model computes the first token, and based on that result, computes the next token. By repeating this process, it completes the sentence.
This generation method is known as autoregressive generation.
The problem is that every single time a token is generated, the massive model’s weights must be read repeatedly.
Technical Block ①: Memory Bandwidth Bottleneck
In LLM inference, it is not just compute capability that matters, but how quickly the model weights can be read from GPU memory.
Particularly during the token generation phase with small batch sizes, the GPU often cannot fully utilize its available compute capacity, and memory bandwidth becomes the primary bottleneck.
Simplified, this can be expressed as follows:
Approximate Maximum Generation Speed Under Memory-Bound Conditions
Tokens generated per second ≈ Memory bandwidth ÷ Weight reads required per token
For instance, if 1 trillion bytes worth of weights must be read entirely for every token, even in an environment providing 8 TB/s of memory bandwidth, a simple calculation yields a theoretical limit of roughly 8 tokens per second.
In reality, of course, models are distributed across multiple GPUs, and various factors come into play—including caching, batching, compute intensity, and memory access patterns. The formula above is not a predictive equation for actual real-world performance, but an approximation model to help conceptualize memory bottlenecks.
Here, an important question arises:
If weights must be read repeatedly just to generate a single token each time, could we process multiple tokens in a single execution of the model?
This question is the essential starting point for inference acceleration technology.
2. Accelerated Serving Optimizes Execution, Not the Model
To understand inference acceleration, one must distinguish between three approaches:
① Modifying the Model Itself
Reducing the model size or computational workload through quantization, distillation, pruning, and related methods.
This approach can reduce memory usage and accelerate execution. However, the computational results may diverge from the original model.
② Optimizing the Inference Engine
While retaining the identical model weights, optimizing GPU memory access, compute kernels, batching, cache management, and scheduling.
A representative inference engine is vLLM. vLLM supports PagedAttention, continuous batching, various quantization schemes, and speculative decoding techniques.
Official documentation: https://docs.vllm.ai/en/latest/features/speculative_decoding/
③ Accelerating the Inference Algorithm Itself
Where models previously generated tokens one at a time, newer algorithms propose multiple candidate tokens in advance and verify them in parallel.
As long as only results ultimately permitted by the original model are accepted, generation speed can be boosted while preserving the output distribution.
These three approaches are not mutually exclusive. Production services often combine multiple methods.
However, if maintaining the original model’s quality is the goal, one must evaluate methods that simply lower precision separately from algorithms that preserve output distributions.
3. The Cornerstone Technology: Speculative Decoding
Speculative decoding is one of the most critical concepts in understanding inference acceleration.
In Korean, it is translated as 추측 디코딩 or 투기적 디코딩.
While the name may sound somewhat unfamiliar, the underlying principle is intuitive.
While the larger model generates tokens one by one, a separate smaller model or dedicated draft generator proposes multiple upcoming tokens in advance. The larger target model then verifies these candidates in a single forward pass.
When candidates are correct, multiple tokens can be accepted at once. From the point of the first mismatch, the output is corrected according to the original model’s rules, and generation proceeds.
Technical Block ②: Draft → Verify → Accept
For example, suppose the original model intends to generate the following sentence:
The inference engine can improve generation speed
The draft generator proposes the following candidate tokens in advance:
The inference engine can increase generation speed
When the original model verifies the candidate sequence, the tokens up to the discrepancy can be accepted as-is, while the point where the divergence between improve and increase occurs is handled according to the original model’s distribution.
Repeating this process produces an output of the same length with fewer sequential execution steps than having the original model generate tokens one by one.
Note that this example is intended for conceptual clarity. In real speculative decoding, the system does not merely compare string values; it relies on token probability distributions and rigorous verification criteria.
How, Then, Is Quality Preserved?
In properly designed speculative sampling, tokens proposed by the draft model are not accepted unconditionally.
Candidates are verified against the original model’s probability distribution, and if necessary, tokens are resampled from an adjusted probability distribution.
When this verification procedure is implemented rigorously, the probability distribution of the final output remains identical to that of the original model, even if the draft model is small or imperfect.
This is the core takeaway.
The draft generator is used solely to accelerate computation, while the authority to determine the final output remains strictly with the original model.
Consequently, not every token proposed by the draft model is accepted. The acceleration efficiency hinges on how many candidate tokens pass verification.
A caveat is worth noting here: having the same output distribution means that the statistical and probabilistic characteristics of the output are preserved under identical sampling configurations. It does not mean that two independently executed runs will output character-for-character identical sentences every single time. Obtaining bitwise identical outputs under deterministic settings requires accounting for random seed handling and precise execution conditions.
4. DSpark: Making Drafting and Verification More Efficient
Now let us examine DSpark.
DSpark is an inference acceleration technology originating from DeepSeek research that introduces two key components to the draft generation architecture of the earlier DFlash lineage:
- Markov head
- Confidence head
The associated public implementation can be found in the vLLM project’s Speculators repository.
Technical documentation: https://github.com/vllm-project/speculators/blob/main/docs/user_guide/algorithms/dspark.md
4-1. Markov Head: Compensating for Inter-Token Dependencies
Proposing multiple tokens in parallel introduces a specific challenge.
While the prediction for the first token may be relatively accurate, subsequent tokens suffer from weakening dependency on which tokens were actually selected preceding them.
For example, even if the first word is predicted well, the probabilities of the second and third words depend directly on that word.
The Markov head mitigates this issue by incorporating information from preceding tokens to adjust subsequent candidates.
In plain terms, it preserves the speed advantages of proposing multiple tokens at once while better capturing the transitional dependencies between tokens.
4-2. Confidence Head: Shortening Verification for Low-Confidence Candidates
Suppose the draft generator proposes five tokens.
The first three tokens may have a high likelihood of acceptance, whereas the trailing two tokens might be uncertain.
In this scenario, blindly verifying all five tokens is inefficient.
The confidence head estimates the probability that each candidate will pass verification. Using this information, the system can dynamically adjust to propose longer spans during high-confidence intervals and verify shorter sequences when uncertainty is high.
Such adaptive verification is particularly crucial in multi-tenant environments where numerous requests are processed concurrently.
4-3. Does DSpark Automatically Make Every Model 10x Faster?
It does not.
DSpark is an algorithm designed to reduce the cost of draft generation and verification; it is not a silver-bullet accelerator that guarantees a fixed speedup multiplier across all models and hardware configurations.
Actual performance varies depending on the following factors:
- Size and architecture of the target model
- Speed and accuracy of the draft generator
- Number of tokens that pass verification
- GPU memory bandwidth and compute capability
- Number of concurrent requests
- Prompt length and generation length
DSpark’s open-source release provides a pre-trained draft model for GLM-5.2 and training examples for the Qwen family. That is, the existence of an algorithm must be distinguished from whether a specific commercial API actually employs it.
Furthermore, bearing the DSpark name does not mean every implementation delivers identical performance. Compatibility between the draft model and target model, configuration parameters, and the hardware environment remain critical.
5. How to Read Acceleration Performance Numbers
The most common mistake when evaluating inference acceleration is comparing a single TPS figure in isolation.
TPS stands for Tokens Per Second, representing the number of tokens processed per second. However, depending on which type of TPS is being measured, its meaning changes entirely.
| Metric | Meaning | Why It Matters |
|---|---|---|
| TTFT | Time to first token | How fast the response begins |
| Per-request TPS | Generation speed within a single request | Streaming speed perceived by the user |
| Overall throughput | Total tokens per second across the entire server | Capacity to serve many concurrent users |
| p95 latency | Threshold within which 95% of requests complete | Service quality under traffic congestion |
| Task completion time | Time elapsed until the actual job finishes | Real-world productivity of an agent |
For example, even if an entire server processes 1,000 tokens per second in aggregate, that does not mean an individual request receives 1,000 tokens per second.
Conversely, even if an individual user experiences an output speed of 1,000 tokens per second, there is no guarantee that a complex task will finish 10 times faster. External tool calls, builds, and test executions can easily consume more time than the model takes to generate text.
Technical Block ③: Calculating the Actual Acceleration Multiplier
If, under identical conditions, the target model’s baseline per-request speed is 100 TPS and the accelerated endpoint achieves 500 TPS, the speedup multiplier based on output rate is calculated as:
Acceleration multiplier = 500 ÷ 100 = 5x
For this calculation to be meaningful, however, variables such as model variant, output length, prompt length, inference settings, and hardware environment must be properly controlled across both measurements.
Especially when comparing APIs across different vendors, one must verify whether the actual underlying model variant being served is truly identical.
A 5x output speedup number serves as a useful starting point, but on its own, it does not prove a pure engine-level performance improvement multiplier.
6. How Fast Are Real-World Service Providers?
Let us now look beyond theory at real-world API services.
As of October 2026, Artificial Analysis compares multiple providers serving DeepSeek V4.1 Flash.
The public benchmark page reports the following output speeds:
- LithosAI ULTRA CHAT: approx. 1,396 TPS
- Inco FAST: approx. 1,234 TPS
- Baseten FAST: approx. 694 TPS
- LithosAI FAST: approx. 540 TPS
These figures reflect results obtained under specific measurement conditions in that benchmark. They may vary depending on provider-specific configurations and time of day, and identical speeds are not guaranteed across all requests.
Source: https://artificialanalysis.ai/models/deepseek-v4-1-flash-non-reasoning/providers
These results demonstrate an important reality:
Output speeds can diverge dramatically even among APIs serving the same model family. The optimization capabilities of inference providers are therefore genuinely worth evaluating.
However, one cannot conclude from these numbers alone that a specific provider achieved its reported speed purely through engine optimization while preserving identical quality to the original model.
This is because several confounding factors may be intertwined:
- Divergent precision levels or model variants
- Differing GPU and memory configurations
- Degree of inference engine optimization
- Adoption of speculative decoding or other acceleration techniques
- Request handling policies and server load
Therefore, when encountering a fast API, one must ask:
“Exactly which model, at what precision, and on what inference engine is this provider running?”
And how did they verify quality parity with the original model?
Only when these two questions can be answered can raw speed figures be evaluated as meaningful technical achievements.
7. Quantization and Lossless Acceleration Must Be Strictly Distinguished
Just because a technology increases speed does not make it equivalent to others.
Quantization
A method that lowers the precision of model weights or computations to reduce memory footprint and computational overhead.
For instance, utilizing low-precision formats such as FP8 or FP4 instead of FP16 or BF16 can deliver faster compute and higher throughput depending on hardware support.
However, quantization is mathematically not the identical computation to the original model. While the difference in output quality may be minimal, specific capabilities may degrade depending on the task.
Consequently, while using FP4 does not inherently mean performance is poor, asserting full parity with the original model requires independent empirical validation.
Speculative Decoding
In contrast, properly implemented speculative decoding verifies candidates through the original model and applies exact correction steps to preserve the output probability distribution.
The goal of this approach is not to obtain an approximate answer quickly, but to execute the sequential computations required by the original model more efficiently.
This distinction is critical.
Quantization can alter the numerical representation and precision of computations. Speculative decoding restructures execution order efficiently while retaining the original model’s outputs.
In production services, both techniques can certainly be deployed in tandem. In such cases, the quality impact from quantization and the correctness guarantees of speculative decoding must be evaluated independently.
8. Are Accelerated Serving Providers Actually Using DSpark?
A more rigorous standard is necessary here.
It can be verified that DSpark is open-sourced and supported by vLLM.
Yet that fact alone does not warrant the conclusion that all APIs from LithosAI or Baseten rely on DSpark.
Commercial providers may combine proprietary kernels, custom batching, cache management, speculative decoding, and model-specific optimizations. A provider might deploy DSpark for certain models while opting for alternative methods on others.
The following factors must therefore be verified:
- The exact model ID and variant deployed on the actual endpoint
- Actual weight precision, such as FP16, BF16, FP8, or FP4
- Whether speculative decoding is employed and its specific implementation method
- The relationship between the draft model and the target verification model
- Output distribution preservation or quality verification results relative to the original model
- Discrepancies between public benchmarks and proprietary internal measurements
That a provider supports DSpark, that it has enabled DSpark for a specific service, and that the service delivers quality identical to the original model are three distinct claims.
Any technically sound evaluation must separate these three aspects.
9. What Matters Most to Developers Is Not Tokens, but Completed Tasks
The business value of accelerated serving does not lie solely in boasting high TPS numbers.
Suppose, for example, a coding agent analyzes 100,000 tokens of context and generates modified code.
If an existing API delivers 100 TPS and a new API provides 500 TPS, the text generation time can theoretically be reduced to one-fifth.
However, the total elapsed task time does not necessarily shrink to one-fifth. Time spent reading files, running tests, executing builds, invoking tools, and handling retries remains constant or dominant.
Furthermore, if the accelerated API triggers more retries, the expected cost savings can quickly evaporate.
Hence, the metric that truly matters in real-world products is:
Total Cost Per Successful Task
Total inference cost + tool call cost + retry cost + failure recovery cost divided by the number of successfully completed tasks.
This metric reflects business feasibility far more accurately than raw per-token pricing.
Accelerated serving delivers immense value when it reduces turnaround time while strictly maintaining quality. This holds especially true for domains where latency dictates the entire experience—such as real-time speech processing, coding agents, large-scale codebase analysis, and multi-agent workflows.
10. How Should You Verify It Yourself?
When evaluating accelerated serving APIs for adoption, the following experimental protocol is recommended:
Step 1: Fix the Baseline Model
Use an identical model version and precision. Even if model names match, checkpoints or quantization schemes may differ, so verify the actual underlying variant.
Step 2: Align Inference Settings
Keep temperature, top-p, maximum output length, reasoning configurations, and system prompts as identical as possible.
Step 3: Repeat with Identical Prompts
Prepare prompts representative of production workloads, such as short queries, extensive code generation, mathematical reasoning, and long-context analysis.
Step 4: Measure Performance
Record TTFT, per-request TPS, overall throughput, p95 latency, total task completion time, and overall costs.
Step 5: Compare Output Quality
Verify whether the serving stack uses an algorithm that preserves the original model’s output probability distribution. If this cannot be proven, independently evaluate benchmark accuracy, code test pass rates, and consistency across reasoning tasks.
Do not conclude that quality is identical simply because two responses look similar at a glance.
Additionally, verifying that outputs match under deterministic inference settings is a distinct test from proving that the output probability distribution itself remains identical.
Conclusion: The True Competitive Edge Is Speed Without Sacrificing Intelligence
As AI model capabilities continue to advance, the optimization of inference infrastructure has emerged as another pivotal competitive arena.
Techniques that boost speed by reducing weight precision or miniaturizing models remain valuable. However, being able to infer faster while strictly preserving the original model’s quality carries distinct, exceptional engineering value.
Speculative decoding and DSpark demonstrate that very possibility. Inference engines like vLLM provide the foundation for deploying these techniques in production environments.
Real-world API benchmarks indeed reveal significant output speed discrepancies among providers serving the same model family. Yet determining which technologies underpin a provider’s ultra-high speeds—and whether quality parity with the original model is preserved—demands independent verification.
Thus, when evaluating accelerated serving, the primary question to ask is not merely: “How many tokens does it generate per second?”
“While preserving the inference quality provided by the original model, how much faster and more cost-effectively can it complete real-world tasks?”
Only when this question can be answered does inference acceleration transcend a superficial speed race and become a transformative technology that reshapes the unit economics of real-world products and services.
References
- vLLM — Speculative Decoding
- vLLM Speculators — DSpark Implementation Documentation
- vLLM — DSpark Adaptive Verification
- Artificial Analysis — DeepSeek V4.1 Flash Performance Comparison by Provider
Note: This article explains the technical mechanics of inference acceleration and public benchmarks. Whether an individual provider actually employs a specific acceleration algorithm and delivers quality parity with the original model must be verified through the technical specifications of that endpoint and independent validation results.