5 Ways I use Home Assistant with a local LLM and I wish I did it sooner

With the Home Assistant running in my home lab, I’m slowly attempting to keep most of my data local. To further that, I started using local LLMs with Home Assistant to explore the extent of achieving much without any cloud dependency. Of course, the performance of a local LLM is directly dependent on the CPU and GPU power it uses. Keeping that in mind, I attempted to integrate smart home automation in Home Assistant with local LLMs.

If you have an old computer or want to try out an SBC, it can mostly run Home Assistant and automate your smart home to a certain extent. That’s what I aimed for in my attempts, where I used a Raspberry Pi 4 and a Windows 11 desktop with an RTX 3080 graphics card to power the local LLMs. Here are some of the ways I use Home Assistant with a local LLM. Looking back, I wish I had done it sooner.

5

Engaging in Voice Control with Home Assistant

Using the conversation pipeline

After starting small, I wanted to use voice commands to control the smart devices. While the built-in Assist pipeline was good, I wanted a slightly more intuitive experience. I then integrated Home Assistant’s automations with local LLMs, such as Ollama. To make it all happen, I used Ollama integration to act as a conversation agent and the speech-to-text engine to work with the new Speech to Phrase feature. The extra sauce from the generative AI running locally helped with contextual awareness to control lights, fans, and other devices based on time, weather, previous commands, and the presence of others. I settled with a $13 M5Stack ATOM Echo smart speaker since I don’t have a Home Assistant Voice Preview Edition. I use voice control to trigger automations and chain multiple commands with some natural language in Home Assistant, all thanks to the local LLM.

4

Get short summaries for tasks and events

Gist of things to get by

I’ve always struggled with weather reports that show several numbers with icons. Home Assistant’s dashboard displays all the weather forecasts impeccably, no matter which weather integration you use. To make better clothing choices, I automated Home Assistant to compile the weather forecast data as a succinct prompt and share it with a local LLM to respond with daily tips for appropriate clothing and accessories. Also, I run a daily automation that recommends indoor and outdoor tasks based on the weather conditions. Based on that, I sometimes make the local LLM help set reminders for specific tasks. My family and I find it easy to follow these natural language summaries rather than information scattered in a tabular format.

3

Chatting with an assistant who has a custom voice personality

Chewing the fat with the LLM

After a few weeks, the default wake words and voice responses began to feel dry and repetitive whenever I requested summaries of weather or tasks from the Home Assistant. I decided to modify the prompt to request witty and cheeky responses from the local generative AI model. My experiments mostly took place with the Mistral and Llama3 models for Ollama, albeit with a refined prompt that emphasized requesting wit and humor in responses. Following my colleague, Adam Conway’s experiments, I configured the custom voice of GLaDOS from the Portal game for the voice assistant. While using his custom voice response specifically for me, I am learning to craft better and crisper prompts for the local generative AI model to process and respond easily.

2

Getting actionable notifications and alerts for events

Automating based on a custom event

Setting up automations became easy for me gradually. Over time, I optimized automations by adding a few conditions to check before relaying the final action or trigger. For example, I often forget to switch off the lamp in my room, so I have set an extended automation. The response from the local LLM in plain text arrives as a notification with actionable buttons to indicate whether I want to leave it on or turn it off. There have been times when simpler automations would turn it off or leave it on. For certain smart devices, it became necessary to check their states – whether they were on or off. In such cases, it was easier to use a local LLM to assess the scenario and, depending on that, send an actionable notification. Such actionable notifications for multiple smart devices also help in managing them and their energy consumption, to some extent.

1

Audio descriptions of images from the camera feed

Make the local LLM describe the images

I’ve been considering a video doorbell or security camera for a while, and couldn’t justify its usage completely. However, I stumbled upon a video that inspired me to make good use of the images captured from the camera feed.

Instead of a video, I preferred that the local LLM could analyze the camera image. That was very helpful in tracking the person who took my water bottle for “safekeeping”. I needed to install the LLM Vision integration from HACS and then use it with the local LLM to analyze the video images and Frigate events. Although it is a work in progress, I estimate the long-term benefits of using it.

A more natural and flexible way of dealing with smart home devices

After integrating the local LLM work with Home Assistant, I appreciated the responsiveness of this system. It worked even without an internet connection, sent no data outside my home network, and eliminated cloud dependency. While my results were acceptable and satisfactory to me, I highly recommend using better hardware (CPU and GPU) for running larger AI models for the local LLMs using LM Studio on Windows. However, you can always test a local LLM with Home Assistant on a Raspberry Pi, if you have one lying around. That’d help you understand your requirements better before buying any expensive hardware for your home lab.