Mercury 2.5 LLM hits 770 tokens per second
An inference-speed claim led a cluster of serving and memory-efficiency work spanning AMD hardware, vLLM and KV-cache research.
TL;DR
- An Artificial Analysis page shared to Hacker News listed Mercury 2.5 at 770 tokens per second.
- Nunchux said it generated 5-second MiniMax-H3 videos in 1.3 seconds on AMD's MI355X, and a PyTorch post described hardware-agnostic models in vLLM.
- Two arXiv papers targeted the KV cache: one shares global KV with layer-specific local history, the other sets eviction by risk targets instead of memory budgets.
A Hacker News submission linking to Artificial Analysis said the Mercury 2.5 LLM hits 770 tokens per second. [1]
Hardware portability was a parallel theme. Nunchux said it produced 5-second MiniMax-H3 videos in 1.3 seconds on AMD MI355X accelerators, and a PyTorch blog post described hardware-agnostic models in vLLM. [2] [3]
On the research side, new arXiv papers proposed a shared global KV cache with layer-specific local history and a risk-controlled approach to KV-cache eviction that moves from memory budgets to risk targets. [4] [5]
Why it matters
Serving cost is where competition is shifting: faster tokens per second, credible non-Nvidia paths and leaner KV caches all lower the price of inference and loosen dependence on a single chip supplier.
Editor's note
The 770 tokens-per-second figure comes from a third-party benchmark page and was not reproduced; the AMD result is a vendor blog claim. arXiv items are cited at the title level.