Tokens per second is the number every LLM provider puts front and center, and it's the wrong number to pick a model by if what you're building is a voice agent. A caller never experiences throughput. They experience the pause before anything happens, then whatever comes after. A model can be the fastest thing on a benchmark leaderboard once it starts generating and still make every call feel laggy, because "once it starts" is doing a lot of work in that sentence, and how long it takes to get there isn't what tokens/sec measures at all.
Tokens per second and time-to-first-token are not the same axis
Time-to-first-token (TTFT) is how long a model takes to produce the very first token of its reply. Tokens per second is how fast it produces every token after that. These sound related. In practice, they're close to independent: a model can rank near the top of the field on one and near the bottom on the other, and nothing about a high tokens/sec number tells you where it lands on TTFT.
Two ways a model can be "fast": only one of them helps a caller
The reason is architectural, not incidental. Reasoning-style models that generate hidden "thinking" content before their visible answer can post excellent steady-state throughput once the visible reply starts, while the caller sits through everything generated before that point, work that never shows up in a transcript and isn't counted by a benchmark that only times the generation phase. Large context windows add the same kind of tax: a model re-processing a long system prompt or growing conversation history pays a fixed cost before the first token exists, regardless of how fast it streams afterward. Model architecture matters too: some routing schemes and serving setups are tuned to maximize sustained throughput under batched load, which is exactly the wrong thing to optimize for a single caller waiting on a single reply in real time.
The practical upshot: benchmark TTFT specifically, on the exact prompt shape and context length your voice agent actually sends, not the vendor's published throughput figure and not a synthetic long-generation test. A model that looks mediocre on a "tokens per second" chart can be the more responsive choice in an actual call, and a model that tops the chart can be the one that makes every reply start with an audible pause.
Fast TTFT is wasted if the pipeline doesn't stream
Getting a fast first token is only half the win: it only reaches the caller if the rest of the pipeline is built to use it. A voice agent has to pass the LLM's output to a TTS engine, and there are two very different ways to wire that handoff. One waits for the full LLM response to finish, then sends the complete text to TTS in one shot. The other streams: it starts sending completed clauses or sentences to TTS the moment they're available, while the LLM is still generating the rest of the reply.
Buffering the full response throws away most of the TTFT advantage you just paid for, because the caller's actual wait becomes TTFT plus the entire generation time of the whole reply, before any audio starts. Streaming caps that wait at TTFT plus the time to generate just the first sentence: the LLM keeps producing the rest of the reply in the background while TTS is already speaking. This is also where a lot of otherwise-good latency work quietly fails: it's easy to build a pipeline that streams on paper and buffers in practice, because a JSON-mode response, a post-processing step, or a moderation pass sitting between the LLM and the TTS call can silently force the whole reply to materialize before anything moves forward. If you can't point to the exact line where the first sentence leaves for TTS before the rest of the reply exists, the pipeline probably isn't streaming, whatever the architecture diagram claims.
The TTS side has its own version of this trap: real-time factor
Once text reaches the TTS engine, the same throughput-vs-first-byte distinction shows up again, under a different name. Real-time factor (RTF) is the ratio of how long a TTS engine takes to generate audio versus the duration of that audio: an RTF of 0.3 means it takes 0.3 seconds of compute to produce 1 second of speech. RTF under 1.0 is the baseline requirement for anything conversational: the engine has to generate audio faster than it plays back, or a long enough reply will start outrunning its own synthesis and the call falls behind in real time, not just at the start of the turn.
Generating faster than playback is the whole game
Time to generate 10 seconds of speech, at RTF 0.3 vs. RTF 1.2
RTF is easy to get a falsely good number for and easy to miss the number that actually matters, for two separate reasons. First, most published RTF figures come from a single request with no concurrent load, and the number that matters for a live product is RTF under whatever concurrency your actual traffic produces: an engine that's comfortably under 1.0 alone can slip past it once several calls are being synthesized on the same hardware at once. Second, RTF describes sustained generation speed, not the first-audio latency a caller notices at the start of a reply: a TTS engine can have an excellent RTF and still have a slow time-to-first-audio (TTFA) if it doesn't start streaming audio until it's processed a full sentence or paragraph of input text. Evaluating a TTS vendor on RTF alone without also checking TTFA is the same mistake as evaluating an LLM on tokens/sec alone without checking TTFT, the same blind spot, one stage further down the pipeline.
What this adds up to
None of this argues for ignoring throughput on either side of the pipeline: throughput still matters for long replies, the point is that it is not sufficient on its own, not that it is irrelevant. An LLM with a fast first token that then trickles out the rest of the reply slowly can still stall mid-sentence on a longer reply, and a TTS engine with fast first-audio but an RTF creeping toward 1.0 will still fall behind on long utterances under real load. The point isn't that tokens/sec and RTF don't matter; it's that they're not the first thing to check, and a spec sheet that only reports them is missing exactly the numbers that decide whether a call feels instant: LLM TTFT on your real prompt shape, whether the LLM-to-TTS handoff genuinely streams sentence-by-sentence rather than buffering, and TTS RTF measured under realistic concurrent load alongside its own TTFA. A voice agent is only as responsive as the slowest of those checks, and none of them are the number most comparisons lead with.
We've already burned that time at Dissolve Labs, so our clients don't have to: we benchmark the LLM, streaming architecture, and TTS layer against real latency behavior before a voice agent is built, not after it starts feeling slow in production.