TL;DR: A local dictation app is a systems project, not a model call. Most of the perceived wait is the model decode, so you optimize when and on what you invoke it. Whisper-family models translate and hallucinate in ways you build a harness around rather than fix, and staying truly offline means explicitly disabling the ML ecosystem's habit of phoning home.
Cloud dictation is a solved problem. You post audio to an API, text comes back, someone else worries about the models. So why build local? Three reasons kept coming up while I built mine: privacy (dictation hears everything, including client names and half-written emails), cost (a heavy dictation user generates hours of audio a day, forever), and latency floors (a round trip to a datacenter has a minimum price that a local GPU doesn't pay). Also a fourth, honestly: it's just satisfying that it works on a plane.
What I didn't appreciate going in is that local dictation is not "run Whisper in a loop." The model is maybe a third of the system. Here's the shape of the rest.
The latency is in the decode, not the recording
My naive assumption was that responsiveness would hinge on audio capture and how fast I could shuttle buffers around. Wrong. When I finally profiled an end-to-end dictation, over four fifths of the wait was the model decoding. Everything else was noise.
That reframes the whole design. You don't optimize the pipeline around the model; you optimize how often and on what you invoke the model. Two things fell out of that for me. First, decode each piece of audio once. It's embarrassingly easy to build a pipeline that transcribes overlapping chunks and throws most of the work away. Second, don't transcribe silence. A decent voice activity detector in front of the model is worth more than any flag on the model itself.
Chunking strategy matters more than it looks. Transcribe in small blocks while the user is still speaking and the perceived latency at the end drops enormously, because most of the work is already done when they stop. The final block is all they're waiting on. Same total compute, completely different feel.
Whisper-family models have opinions
Open speech models are genuinely good now, but they come with behaviors you have to design around rather than fix.
They want to translate. If you speak a language the model is shaky on, it will sometimes confidently hand you an English translation instead of a transcription, and its own language-detection confidence won't warn you. The detector can be certain and wrong at the same time. You end up needing a second signal, like sanity-checking the decode quality against the claimed language, before trusting the output.
They hallucinate on silence and truncate on odd inputs. Feed a model near-silence and it may produce a plausible sentence from nothing. Certain input combinations just misbehave, and the fix is often not a parameter but a different input: retry, or swap out part of the context, rather than turning knobs.
Context carryover cuts both ways. Some local inference servers carry text context between requests to improve continuity. Great for a stream of related sentences, weird when your dictations are unrelated and a previous snippet biases the next one. Whether to keep it is a real tradeoff: disable it and you can lose punctuation quality along with the bias.
None of these are bugs you fix once. They're properties you build a harness around: a corpus of your own recordings you can regression-test against, so that when you change a model or a flag you find out what actually got better.
"Offline" is a discipline, not a default
Here's the one that genuinely annoyed me. You build an offline app, you test it offline, it works. Then a user on a fresh machine, on a bad connection, hits a code path where some ML library helpfully decides to download a model or check a hub for updates. Your offline app now hangs on a network call you never wrote.
The ML ecosystem assumes connectivity. Hugging Face tooling, TTS engines, even utility libraries will quietly reach out unless told not to. If you're serious about local, you have to be explicit: set the offline environment flags, pre-download every model at install time, and treat any network access after installation as a bug. The payoff is an app whose behavior on day 400 matches day 1, which cloud-dependent apps can't promise.
The last mile is product work
The parts nobody blogs about took the longest. Injecting text into whatever app has focus, reliably, across the whole OS. Muting system audio while recording so the model doesn't transcribe your notification sounds. Packaging gigabytes of models into an installer people will actually download, and deciding what ships in the box versus what installs on first run. Handling the user who dictates in two languages mid-sentence, because real people do that.
If you're weighing cloud versus local for a dictation feature: cloud gets you to a demo in an afternoon. Local gets you an app that's private, free to run, and yours, at the cost of owning a small systems project. Having done it, I'd do it again. But go in knowing the model is the easy part.
Key takeaways
- Profile before designing: in my build, over four fifths of the wait was model decode, not audio plumbing.
- Decode each piece of audio once, skip silence with a VAD, and transcribe in blocks while the user is still speaking.
- Whisper-family models confidently translate, hallucinate on silence, and misbehave on odd inputs; regression-test against your own recordings.
- Offline is a discipline: set the offline flags, pre-download models at install, and treat any later network call as a bug.
- The last mile (text injection, system mute, installer size, mixed-language speech) is product work and takes the longest.