Yes, you can run a Home Assistant local voice assistant that understands two languages, keeps speech recognition and synthesis on your own hardware, and costs about $2 a month. Ours handles English and Polish. Whisper and Piper run locally over Wyoming. Home Assistant answers simple commands itself, for free. Only open-ended questions go to a small hosted LLM through OpenRouter, at a measured $0.0024 per question. A smarter model did not make it bilingual. A plainer decision did: two pipelines instead of one.
This is how we got there, including the parts that went wrong. The voice assistant was one piece of a larger build: a full smart home stood up in about a week, with an AI coding agent doing most of the research and configuration. We wrote that up in Home Assistant with Claude Code: A Full Smart Home in a Week. This article covers only voice: the language problem, the model choice, the memory budget, and where the microphones actually are.
The problem: one house, two languages, one small host
The requirement was easy to state and awkward to build. The household speaks English and Polish. Anyone should be able to ask for a light, the weather or who is home in whichever language comes naturally. They should get the answer back in the same language, in a voice that sounds right for it.
The constraints made it harder. Everything runs in Docker Desktop on a Mac mini-class desktop with 16 GB of RAM and an 8 GB Docker VM. That same VM also runs Home Assistant, Mosquitto, Zigbee2MQTT, Z-Wave JS UI and Frigate for CCTV. Every voice model we added would compete with camera detection for memory. We also did not want a monthly bill that grew quietly in the background. A voice assistant people actually use gets asked a lot of small questions, and small per-question costs add up if you never look at them.
The stack: what runs locally and what doesn’t
Speech in and speech out stay on the box
Speech-to-text is Whisper, served over the Wyoming protocol. Text-to-speech is Piper, with a British English voice and a Polish voice. Both run as containers in the same Docker Compose setup as the rest of the house. Our agent, Claude Code, added the Wyoming integrations to Home Assistant through its API with a long-lived token, not by clicking through the UI. That keeps the setup reproducible: the same compose files are meant to move to Proxmox later without changes to the voice side.
Local handling first, LLM second
The most important cost setting is a single toggle. With Home Assistant’s “prefer handling commands locally” option on, simple commands like “turn on X” are matched by Home Assistant’s built-in intent handling and never reach the LLM. They are instant and they cost nothing. The LLM only sees what the local matcher can’t handle: questions, comparisons, anything phrased loosely. In practice that means the model does the work only a model can do, and the routine traffic doesn’t touch the API.
The brain is rented by the question
The conversational “brain” is a hosted LLM accessed via OpenRouter. A hosted model suited this house for two reasons. Its cost scales with use rather than with hardware. And it adds nothing to a host whose memory, as you will see below, was already the tightest resource in the building. The trade-off is plain: anything that reaches the LLM leaves the house. That is why the local-first toggle matters for privacy as well as cost.
What broke: English questions, Polish answers
The first version used a single conversation agent for both languages. It mostly worked. Sometimes, though, the LLM answered an English question in Polish. The pipeline was still configured for English, so the British English Piper voice then read the Polish text aloud, which is about as useful as it sounds. Better prompting reduces how often this happens. It cannot make it impossible, because you are still asking the model to decide which language to use, every time.
The fix, made on 29 September, was to stop asking. We built two agents, one English-only and one Polish-only, sharing the same prompt body; the agent wrote a small prompt builder so the two never drift apart. Each language now has its own pipeline, which pairs three things: an STT language, an agent, and a Piper voice. The language is settled by which pipeline you use, before the model sees a word. The model only has to follow “reply in English” or “reply in Polish”, which is a much easier instruction to obey. In testing, 4 out of 4 answers came back in the right language.
The general lesson applies well beyond voice. If a model is making a choice you could make in code, take the choice away from it. We cover more of these failure modes in AI Automation Challenges to Plan For Before You Build. This one is a textbook case: occasional and annoying, and fixed by architecture rather than by a better prompt.
Choosing the model: from an estimated $14 to a measured $2
Our early research pointed at Claude Sonnet 5, at an estimated $14 a month for our expected usage. That is not a lot of money. But it was an estimate from desk research, not a measurement. So on 28 September the agent benchmarked 10 tool-calling models on realistic Home Assistant-style commands in English and Polish. Gemini 2.5 Flash-Lite came out on top: 4 out of 4 correct, about 0.5 seconds per response, and roughly $0.00003 per call.
Then it started making things up
Once it was live, two problems showed up that the benchmark had not tested for. The model invented presence, claiming people were home or away without evidence. It also mixed up today’s weather with tomorrow’s. Neither is a tool-calling failure. Both are the model filling gaps with plausible guesses, and a voice assistant that tells you confidently that someone is home when they are not is worse than one that says nothing.
We rewrote the shared prompt around three hard rules:
- Presence comes only from occupancy sensors, never from inference.
- Only current weather is used.
- Replies are always in the user’s language.
We then moved to Gemini 3.1 Flash-Lite. With the new prompt it produced 0 failures across 14 test questions, and its Polish reads naturally rather than as translated English. That last point matters more than any benchmark score in a bilingual household. People notice stilted phrasing within a day, and then stop using the feature.
What it actually costs
Measured on real use, a question costs $0.0024. At 30 questions a day, that comes to about $2 a month. That is well above the benchmark’s per-call figure. A real question is not a single benchmark call, which is exactly why we budget from production measurements rather than extrapolating from tests. It is still a small fraction of the original $14 estimate.
Cost control does not rely on good behaviour. The API key has a hard $50 spend limit, so a runaway loop or a leaked key has a bounded worst case. Credentials in the wider build are generated locally and stored by reference in a password vault, out of the agent’s context. Keys deserve that treatment however cheap the model is; our notes on handling them are in Data Security and Privacy in AI Projects.
The memory budget: retiring Parakeet
For English speech recognition we first ran a separate Parakeet model, about 575 MB, alongside multilingual Whisper. The reasoning seemed sound: use a dedicated English model for the dominant language. Testing showed the multilingual faster-whisper model transcribed English identically. So Parakeet was paying for nothing, and we retired it. One Whisper container now serves both pipelines.
On this host, that was not a tidy-up. It was a reliability fix. During the same week Frigate stalled because of memory pressure: at the worst point the 8 GB Docker VM had about 83 MB free and swap was full. Turning off Frigate’s semantic search fixed the immediate crisis, but stalls came back once more before the English Whisper container went. Afterwards the VM had 783 MB available, up from 485 MB. A specialised model per language is a memory tax. On a shared single host, that tax is paid by whatever else needs the RAM, which here was the cameras.
Where the microphones actually are
A voice assistant is only as good as the microphones it can reach. The Home Assistant phone app is set as the default Android assistant, which turns everyone’s phone into a bilingual microphone that is always within reach. We had hoped to use the Nest speakers too, but they can’t act as Assist microphones because Google blocks third-party AI on them. They now serve as TTS and announcement speakers instead. ESP32 satellites and Home Assistant’s own Voice hardware are the obvious next step for hands-free rooms.
The house also has Google Home voice control, through the free manual cloud-to-cloud link, with English and Polish aliases on devices. That produced one collision worth knowing about. Google’s voice matching picked up helper toggles named “…motion light” and switched them off when people asked for lights. That quietly disabled the motion automations. We renamed the helper entities so no voice command can match them. When you expose a house to voice, every entity name becomes part of the grammar, and you have to name defensively.
The last piece was not technical. The agent produced an 8-page household voice-command guide as a PDF in English and Polish. A voice assistant nobody knows how to talk to is just a container using memory.
What we would do differently
We would start with two pipelines on day one. The single-agent version taught us nothing we could not have predicted. We would also benchmark questions as well as commands from the start, including presence and weather, because those are where the hallucinations lived and the original benchmark could not have caught them. We would begin with one multilingual STT model and add a specialised one only if testing proved it was needed. And we would put the spend limit and a per-question cost measurement in place before going live, not after.
What would we do first if this were our business?
Swap the house for a business, a multilingual front desk, a shop floor or a support line, and the order of work stays the same. First, route everything deterministic away from the LLM: the local-first toggle here is the same idea as rules before models in any workflow. Second, decide language, tenant or role at the pipeline level, not inside the prompt. Third, write the hallucination rules down as hard constraints tied to real data sources, and test the questions people actually ask, not only the happy-path commands. Fourth, set a hard spend cap and measure cost per interaction on real traffic before you promise anyone a number. None of this needs a large model or a large budget. It needs a clear view of which steps genuinely need intelligence, which is the same argument we make in Practical AI for Mid-Sized Companies.
D3X Solutions LABS is an AI engineering studio in Ireland that builds AI agents, automation and orchestration systems, and advises teams putting AI into production. On a project like this we start by mapping which interactions can be handled deterministically. Then we benchmark candidate models on your real inputs, in every language you need, and put cost limits and failure tests in place before anything ships. If you are weighing up a voice or agent workflow and want a second opinion on the architecture and the running costs, book a free 30-minute call.
Frequently Asked Questions
Can Home Assistant run a voice assistant fully locally?
Speech-to-text (Whisper) and text-to-speech (Piper) can run locally over the Wyoming protocol. With "prefer handling commands locally" turned on, simple commands like "turn on X" are handled entirely by Home Assistant. Open-ended questions need an LLM. In our setup that is a hosted model via OpenRouter, so only those questions leave the house.
How much does an LLM voice assistant for Home Assistant cost to run?
Ours measured $0.0024 per question on Gemini 3.1 Flash-Lite, which is about $2 a month at 30 questions a day. Simple commands handled locally cost nothing because they never reach the LLM. We also set a hard $50 spend limit on the API key to bound the worst case.
How do I make a Home Assistant voice assistant answer in both English and Polish?
Use two pipelines rather than one multilingual agent. Each pipeline pairs its own STT language, a language-specific conversation agent and a matching Piper voice, with a shared prompt body. With a single agent, our LLM sometimes answered English questions in Polish. With two pipelines, 4 out of 4 test answers came back in the right language.
Do I need a separate English speech model alongside Whisper?
In our testing, no. We first ran a separate Parakeet model of about 575 MB for English, but the multilingual faster-whisper model transcribed English identically. Retiring Parakeet freed memory on a constrained host, and one Whisper container now serves both languages.
Can Google Nest speakers be used as Home Assistant Assist microphones?
No. Google blocks third-party AI on Nest speakers, so they can't act as Assist microphones. You can still use them as TTS and announcement speakers. For microphones, the Home Assistant phone app can be set as the default Android assistant, and ESP32 satellites or Home Assistant Voice hardware are other options.
Which LLM works best for a Home Assistant voice assistant?
We benchmarked 10 tool-calling models on English and Polish Home Assistant-style commands, and Gemini 2.5 Flash-Lite won. Once it was live, it invented presence and confused today's and tomorrow's weather. After a prompt rewrite we moved to Gemini 3.1 Flash-Lite, which had 0 failures across 14 test questions and produced natural Polish.
Want this working in your business?
D3X Solutions LABS builds AI agents and automation that pay back fast. Not sure where to start? A free 30-minute call is enough to find the first thing worth building.
Get in touch
