The Helpful Idiot

Cleaning Up Six Years of Home Assistant with Claude

thehelpfulidiot — Mon, 05 Oct 2026 00:00:00 +0000

My Home Assistant setup has been running since at least 2020. Some of my automations still have IDs like 1588516677003, which is just the time I clicked "save" in May 2020. Since then I have added a little bit at a time and removed basically nothing.

By this fall it had 421 automations and 206 scripts. Two versions of my voice media controls (v7 and v8) were both turned on, so saying "volume up" turned the volume up twice. One automation ran every single second, all day, just to refresh a sensor. And I had a sneaking suspicion that a few things only worked by accident.

It worked, mostly. My family uses it every day. And that was exactly why I never wanted to touch it.

So I did what I did with this blog: I sat down with Claude, an AI from Anthropic, and we cleaned it up together. Claude did most of the reading, writing and testing. I made the decisions, answered a lot of questions about my house, and pasted commands into my server. Here's how that went.

The end result, after 103 commits over three days

Step Zero: Get It Into Git

Before anything else, the config had to go into Git. My Home Assistant runs in Docker, and the config folder is just a folder on the server. Turning it into a Git repo means every change becomes a commit: I can read it, ask questions about it, and roll it back.

The catch is that a Home Assistant config folder is full of things that should never leave the house: secrets.yaml, the .storage folder (which has tokens in it), the database, logs, SSH keys, camera snapshots, and whatever else integrations have dropped in there over six years. A normal .gitignore lists the things to leave out, which means anything you forgot about goes in. Mine works the other way around: it ignores everything and then opts in only the files I actually want tracked. Here it is, trimmed a little:

# ALLOWLIST: ignore everything, then opt in what we track
/*

# repo files
!/.gitignore
!/README.md
!/REVIEW.md
!/CLAUDE.md
!/.github/

# top-level config and scripts
!/*.yaml
!/*.py

# folders
!/packages/
!/blueprints/
!/themes/
!/python_scripts/
!/scripts/
!/custom_sentences/

# www: ignore its contents, allow only code
!/www/
/www/*
!/www/dashboard/
!/www/intercom-card.js
!/www/icons/

# RE-EXCLUDE anything sensitive that matches the allows above.
# These must stay below the allows.
/secrets.yaml
**/secrets.yaml
/known_devices.yaml
/ip_bans.yaml
*.backup*
*.bak
*.db
*.db-*
*.log*
.env
*.pem
*.key
id_rsa*

A few things worth pointing out:

/* comes first. Every file and folder at the top level is ignored until a ! line lets it back in.
www/ gets the same treatment one level down. That folder is where Home Assistant serves files from, so it collects camera snapshots and downloaded images. Only my dashboard code and a couple of icons and cards are tracked.
The re-excludes come last. !/*.yaml would happily let secrets.yaml in, so it's ignored again below the allow. Order matters in a .gitignore: the last matching line wins.
.storage/, the database and my ESPHome folder are never mentioned at all, so they're never tracked.
There's a secrets.example.yaml with every key my config expects and fake values. It's what lets Claude load my config without the real secrets (more on that below).

That way, if some integration drops a new file full of credentials in there next year, Git ignores it by default.

The private repo lives on GitHub, which is also where Claude works from. Claude Code has a cloud version that clones a repo into a sandbox, works on its own branch, and pushes there. That turned out to be the key to this whole project.

How Changes Got to the House

I didn't want an AI anywhere near my live Home Assistant, and it never was. Claude worked on a copy of the config in the cloud. It could install the exact same Home Assistant version I run, load my YAML into a throwaway instance with fake secrets, and test things against that. It never had a token for my house.

The loop for every change. The only step that touches the house is the one I paste.

Every round ended with a block of commands for me to paste on the server, something like this:

cd ~/docker/homeassistant/data
git pull origin claude/some-branch
docker exec homeassistant python3 -m homeassistant --script check_config -c /config
docker restart homeassistant
docker exec homeassistant python3 /config/scripts/ha_admin.py /config/scripts/plans/2026-10-automation-merge-2.json
docker exec homeassistant python3 /config/scripts/ha_admin.py /config/scripts/plans/2026-10-automation-merge-2.json --apply
git push origin master

Pull the branch, let Home Assistant check its own config, restart, apply the "plan" (more on that in a second), and push the result back to master so GitHub always matches what's actually running.

Things That Aren't in YAML

Not everything lives in YAML files. Entity names, areas, helpers, automation categories, which entities are exposed to voice: all of that is stored in Home Assistant's .storage folder, which I'm not putting in Git.

Normally that means clicking around the UI. With 180 automations getting deleted or merged, that's a lot of clicking, and a lot of chances to click the wrong thing. So Claude wrote a small script, ha_admin.py, that runs inside the Home Assistant container and applies a "plan": a JSON list of changes made through Home Assistant's own websocket API. Here's part of one:

{
  "steps": [
    {
      "op": "registry_remove_orphans",
      "note": "per-room motion light automations now merged into one per room",
      "entity_ids": [
        "automation.laundry_room_lights_on_with_motion",
        "automation.laundry_room_lights_off_when_motion_stops"
      ]
    },
    {
      "op": "automation_categories",
      "categories": [{ "name": "Lighting – Indoor", "icon": "mdi:lightbulb" }],
      "assign": { "motion_lights_laundry": "Lighting – Indoor" }
    }
  ]
}

By default it's a dry run. It prints exactly what it would change and changes nothing. Only --apply actually does it. Plans are also safe to run twice, so if something failed halfway, I just ran it again.

Letting Claude See the Running House

The YAML in Git is only half the picture. It says what an automation should do, but not what's actually out there: which entities exist, which are unavailable, which automations I'd disabled in the UI years ago, which helpers a dashboard still uses, what's in the error log. Claude had no way into my house, so it needed me to bring the house to it.

That's what a handful of small, read-only Python scripts are for. They all run on the server, inside the Home Assistant container, and none of them changes anything.

export_for_review.py is the big one. It takes a snapshot of the running system and writes it to one compressed file:

the live state of every entity, through Home Assistant's own API
the entity, device, area and category registries
every integration, helper, person, zone and voice pipeline
every dashboard's full configuration
which custom components are installed
a summary of the error log: each distinct error or warning, with how often it happened

The important part is what it leaves out. It only reads an allowlist of .storage files, and only an allowlist of fields from each one. Then it redacts anything that looks like a password, token, email address, MAC address, public IP or GPS coordinate. As a final safety net, it goes through secrets.yaml line by line and removes every one of those values from the output, then tells me how many it found. Here's what it printed the last time I ran it:

wrote review-export.json.gz (14775 KiB uncompressed)
states: 2214 from core.restore_state
entities: 7812  devices: 793  integrations: 256  dashboards: 35
secrets.yaml values found and removed from output: 0

That last zero is the one I like to see: the earlier redaction steps had already caught everything, so the safety net had nothing left to remove.

I run it, hand Claude the file, and Claude can answer questions like "is anything still using input_boolean.jon_iphone?" without ever seeing a password. A lot of the cleanup came straight out of these exports: disabled integrations nobody remembered, an old phone that was still reporting 85 entities, helpers that nothing had read in ages, dashboards with buttons that called scripts that didn't exist.

Three smaller scripts filled in the gaps:

ha_check.py prints the current state of whatever entities I give it, plus every open repair issue. It has a few extras: --homekit lists what each of my HomeKit bridges exposes (so nothing CarPlay or the Home app uses gets deleted), --z2m lists every Zigbee device, and --changes=<entity> shows how often something changed in the last 24 hours. This was the "did it work?" check after every deploy.
export_history.py exports the recorded history of a few entities, up to the 10 days my database keeps. This is how my master bath shower fan got tuned: Claude replayed 16 real showers from my humidity sensors against different rules and showed me the numbers before building anything.
recorder_noise.py reads the database and lists which entities write the most rows. More on what that found below.

First, a Review

Before changing anything, Claude read the whole thing and wrote a review: a 500-line REVIEW.md sorted by how much each problem mattered. Honestly, this alone was worth it. Some highlights from the first read:

Someone else's presence sensor was counting my phone. A copy-paste from years ago. If I was at a certain place, the house thought Mom was there too.
A safety check that could never work. Seven "sync" automations were supposed to skip when a light was unavailable. They compared a whole state object to the string 'Unavailable' (with a capital U, which Home Assistant never uses), so the check never did anything.
A voice command that called a script that didn't exist. "Call the master bedroom" would say "Calling…" and then quietly do nothing.
Five passwords and tokens sitting in plain text in configuration.yaml. They're in secrets.yaml now.

Security went first. One of my automations would unlock a door if someone arrived home and then anything moved near a door in the next ten minutes, including a cat or a delivery driver. It's been replaced with a proper "arrival window" that only unlocks the door the person is actually at, closes after the first unlock, and sends me a notification every time it does. The voice PIN unlock now only accepts the PIN from the speaker that asked for it, and locks itself out after three wrong tries.

The Rules

The most useful thing I did in this whole project was write down some rules before any of the big changes. They live in a CLAUDE.md file in the repo, so every new Claude session reads them first. The most important one:

Family first. Automations should feel helpful, never commanding or overbearing, especially for the kids. Every way a family member already interacts with the house must keep working: wall switches, Hue app, buttons, voice phrases, dashboards. Changes may only add things to learn, never take any away. If a change would alter something a person would notice, ask first and explain it in family terms.

And the second most important one: refactors have to be proven to behave exactly like the original, and real bug fixes are done separately and called out one by one. That one changed how the rest of the project went.

Merging Automations, With Proof

A lot of my automations came in pairs. "Laundry room lights on with motion" and "Laundry room lights off when motion stops." Kitchen, hallway, attic, master bath: same thing, two automations per room.

Home Assistant can do both in one automation using trigger IDs and a choose. Each original automation becomes one branch, with its triggers, conditions and actions copied over exactly.

Same logic, half the automations, and each room's motion lights are in one place

Here's the trick that made me trust it: for every merge, Claude spun up a test instance of Home Assistant with the old automations and another with the new one, with every service call stubbed out so nothing real happened. Then it fed both the same random sequence of motion and light changes (with the 10-minute delays sped up to 1 second) and compared every single service call. They had to match exactly.

The door auto-locks got the most thorough version of this. Nine automations ran the front and back door countdowns (start when the door closes, cancel when it opens, lock when the timer ends, restart at 10 minutes on arrival), and they became one per door. To check it, Claude modeled the old nine and the new two, and ran both through 40,000 random sequences of doors opening, locks changing, people arriving, Home Assistant restarting and time passing. They locked at exactly the same moments every time.

I'll admit that when I saw "40,000 simulations" go by for my back door, I laughed out loud. Is that overkill for a door lock? Absolutely. It's also the automation I least want to be wrong about, and it cost me nothing but a few minutes of waiting.

It also knew when not to merge. One of my kids' bedroom lights and the half bath both end with a long delay used as a "hold." Merged into one automation, that hold would have blocked the other branch from running, so they stayed separate. Same for a couple of automations that read their own last_triggered time.

The merges, all told:

Before	After	How it was checked
9 "light switch control" automations, one per room	1 "Light switch buttons"	every switch × every button press, identical light calls
9 front and back door auto-lock automations	2, one per door	40,000 random event sequences, identical lock decisions
22 motion-light automations	11, one per room	random motion and light sequences in every room
9 circadian and hallway-dimming automations	3	old vs new, identical calls
14 fan and speaker-volume automations	3	old vs new, identical calls
10 kids' automations (a Mira copy and a Simon copy of each)	5	old vs new, identical calls
8 outdoor light automations	1 schedule	each decision tested step by step
7 attic AC automations and 5 basement heater automations	1 each	occupancy, manual changes, a late power reading, a missed IR signal
3 TV auto-off automations and 2 TV power watchdogs	merged into the new TV setup
3 voice weather automations, 2 "where is Mom/Dad" copies	1 each	identical spoken replies to 18 weather questions and 9 "where is" situations

A few of these deserve a closer look.

The light switches

I have ESPHome wall dimmers in most rooms. Each room had its own "light switch control" automation for what the up and down buttons do, and a separate "sync" automation per switch that checked every 5 seconds whether the switch and the bulbs agreed. That's well over a dozen automations, all doing roughly the same thing with slightly different copy-paste.

Now there's one "Light switch buttons" automation. Each switch's button events are a trigger with an ID, and a small table says which lights that switch controls. The syncing moved into the switches themselves (more on that below). Claude simulated all 9 switches with all 4 button commands, old and new, and the light calls matched exactly. That one merge removed about 660 lines.

The attic AC

This one wasn't a merge so much as a rewrite, because the old automations disagreed with each other. My attic office has a window AC that only understands an IR "power toggle," which I can't tell is on or off except by looking at its smart plug's power draw. Seven automations all poked at it: some changed the thermostat's target on every presence blip, some switched the AC directly behind the thermostat's back. With a button that only toggles, anything that acts without checking can just as easily turn the AC off as on.

Now one automation, "Attic climate," is the only thing that sets the attic's target temperature: 72 when someone's up there, 80 when it's empty. It only acts when that changes, so if I set it to 70 by hand, it stays at 70 until I leave. The actual on/off goes through one script that only sends the IR toggle if the plug says the AC is in the wrong state, waits up to a minute to see it worked, and tries once more if not. The basement heater got the same treatment: five automations became one.

Voice: Media Controls

The biggest automation I have left is "Voice: Media Controls" at about 1,200 lines and 44 sentence triggers. It handles "pause," "skip," "volume up," "stop the music in the kitchen" and so on, figures out which room you meant, and asks an LLM "did you mean…?" when it can't tell. When we started, there were two copies of it running at once (v7 and v8), plus a separate "Stop All Media" automation that also fired on "stop." It's one automation now. It's big, but it's one big thing in one place, instead of three things quietly fighting.

Stop Polling Everything

This was the big one for me. A lot of my automations had a time_pattern trigger: "every 5 seconds, check if the light switch and the bulbs agree, and fix it if they don't." Twenty of them were running every few seconds. One ran every second, all day, forever.

Added up, my Home Assistant was running about 270 scheduled automations per minute, roughly 389,000 a day, mostly to check things that hadn't changed.

The fix was to react to things changing instead of constantly asking if they changed. For my ESPHome light switches, the "sync" moved into the switch firmware itself: the dimmer just follows its room's light. For the rest, a template trigger fires only when the two sides actually disagree for a few seconds. A template trigger only fires when its result changes, so there's nothing left to poll.

Now it's about 4 runs a minute, and those few are deliberate 1–5 minute safety nets. That's about 66 times fewer scheduled runs.

Faster, Too

Cleaner code is nice, but what I actually notice day to day is that the house is quicker and less busy. A few of the speed-ups:

The light switches react on their own. The "sync" between a wall switch and its room's lights used to be a round trip through Home Assistant, backed up by a 5-second poll. Now each ESPHome dimmer follows its room's light in its own firmware (follow_entity). Home Assistant doesn't have to do anything at all for the switch to catch up.
Kodi's Back and Home buttons lost a 2-second lag. The remote waited for Kodi to report which screen it was on before doing anything. It turns out Kodi answers during the request, so the wait never saw the answer and every press sat out its full 2-second timeout. In the simulation, a press went from 2.05 seconds to 0.15.
The database writes about 40% less. recorder_noise.py showed my database was writing about 1.9 million rows a day. About 420,000 of them were CPU and memory readings for every Docker container, which nothing used. Another 212,000 were my mmWave sensors reporting exactly where in the room each person was standing, constantly. Then camera snapshots, signal-strength readings, and calculated humidity averages. None of it changed what's on a dashboard or what an automation does; Home Assistant just stopped saving history nobody looked at, which means less work for the database on every single change.
The shower fan catches the shower in about a minute. Using 10 days of humidity history, the new rule caught all 16 recorded showers about a minute in. The old rule had missed one entirely and caught another 7 minutes late.
No more restart storms. Between the backed-off watchdogs below and nothing being restarted unconditionally at startup, a reboot of Home Assistant no longer kicks off a wave of service restarts.

Watchdogs That Back Off

I have a bunch of automations that restart things when they go down: the UPS monitor, the Ring bridge, my fridge's integration, cameras. They all worked the same way: if it's down, restart it. Five minutes later, if it's still down, restart it again. And again. Forever.

If something isn't coming back, hammering it every five minutes doesn't help

Now there's one shared script, script.watchdog_backoff. It restarts the thing, waits, and doubles the wait each time up to two hours. After three failed attempts it sends me one notification, and one more when it recovers. Each watchdog just hands it the details:

- action: script.watchdog_backoff
  data:
    name: Family room UPS (NUT)
    entities:
      - sensor.cyberpower1_status
    down_states:
      - unavailable
    restart_action: script.restart_nut
    first_wait_minutes: 5

While we were in there, it turned out one automation restarted a couple of services and rebooted my Frigate container every time Home Assistant started, whether they needed it or not. That's gone too.

Bugs Hiding in Plain Sight

Since everything was being compared and simulated anyway, a lot of old bugs fell out. Here are some favorites.

The weather check that only knew one kind of weather

My porch lights are supposed to turn on during the day when it's gloomy. The old check looked like this:

value_template: "{{ states('weather.home') == ('partlycloudy' or 'sunny' or 'windy') }}"

That reads fine in English. But ('partlycloudy' or 'sunny' or 'windy') isn't a list: or just returns the first thing that isn't empty, which is 'partlycloudy'. So this only ever matched "partly cloudy." On a sunny morning, the porch lights decided it was gloomy and stayed on. For years. The new version has an actual list of bright conditions, and anything else (except unknown) counts as gloomy.

Nest never knew we left

When we both left the house, Home Assistant was supposed to set the Nest thermostat to "away." But the trigger fired when the first person left. Another automation, which puts Nest back to "home" when anyone is home, immediately undid it. Then when the second person left, nothing fired at all. So Nest never found out the house was empty. It now waits until nobody is home, and since the kids can be home without us, "nobody" also means the downstairs cameras haven't seen a person for 30 minutes.

The printer protection that couldn't happen

The window AC in my attic office shares a circuit with my laser printer, so the AC is supposed to pause while the printer heats up, or the two of them trip the breaker. The old automation only paused the AC if the AC was drawing more than 600 W. Its compressor draws 400 to 500. So it never paused anything.

One space, one silently dead automation

This one was Claude's mistake, and it's my favorite lesson from the whole project. During one of the merges, a single trigger ID got indented one level too far. That turned it into a condition instead of part of the trigger. Home Assistant's check_config passed, the restart worked fine... and Home Assistant quietly disabled the whole automation with an error in the log.

It turns out check_config doesn't fail on an invalid automation. It just logs it. So now there's a validate_automations.py that loads every automation and script through Home Assistant's own validator, and nothing gets pushed until that passes. Every automation and script passes it today.

Voice Got a Lot Smarter

I have voice satellites in most rooms: Home Assistant Voice PE units and a couple of Linux boxes. Voice had the messiest problems of all, because of one thing I didn't know:

When you say something, every automation with a matching sentence fires. And they all run before Home Assistant's built-in commands, so they block those too.

My media controls automation had sentences like stop [in] [the] {area}. In a sentence trigger, {area} is a wildcard that matches any words. And since everything in brackets is optional, "stop the timer" matched it, with "timer" as the room. So did "stop the fan." "Volume up" also matched "volume {level}" and set the volume to 50%.

To find all of these, Claude took every sample sentence from every voice trigger in my config (2,823 of them) and ran each one through Home Assistant's own conversation agent to see which automations it actually reached. Then fixed them until every sentence only went where it was supposed to. Room names now have to come after a fixed word like "in," so the wildcard can't swallow the rest of the sentence.

Goodbye, 1,082 lines of custom timers

Years ago, before Home Assistant had built-in voice timers, I built my own: 9 automations, 2 scripts, 16 timers, and 32 helpers. Today every one of my satellites supports Home Assistant's built-in timers, which do everything mine did and more.

The catch was the family. Everyone was used to saying things their way, like "set a pasta timer for 5 minutes" or "how many timers do I have." Following rule #1, none of those could stop working. So the old system was replaced with a small file of extra phrasings for the built-in timers:

language: en
intents:
  HassStartTimer:
    data:
      - sentences:
          - "set [a|the] {timer_name:name} timer for <timer_duration>"
          - "(set|start) [a] <timer_duration> {timer_name:name} timer"
  HassUnpauseTimer:
    data:
      - sentences:
          - "(resume|unpause|continue) [the|my] timer"

Out of the 312 phrasings the old system understood, 310 still work. The two that don't collided with "remove 5 minutes from the timer," which seemed like a fair trade.

A Detour: Packages vs. the UI

Not everything went in a straight line. Early on, Claude moved most of my automations out of automations.yaml and into feature "packages" (one YAML file for lighting, one for media, and so on). Very tidy. But automations in packages can't be edited in Home Assistant's UI, and I still like making quick tweaks from my phone.

So they went back. Now automations.yaml and scripts.yaml stay UI-editable, organized into 19 categories in the UI (Lighting, Climate & Fans, Kids' Bedtime, Voice, and so on). Only the settled, rarely-touched stuff like locks, presence, and the voice PIN lives in packages.

Claude also rewrote both files using Home Assistant's own YAML writer, converted to the current syntax (triggers:, actions:), and checked that every automation came out identical. That sounds boring, but it matters: when I save an automation in the UI now, Git shows only the lines I actually changed, instead of Home Assistant reformatting the entire file.

By the Numbers

Since every change is a commit, Git keeps score. The first commit, my config exactly as it was, went in on a Saturday at 2 pm. The last one in this round landed Monday at 4 pm. In between: 103 commits in about 50 hours (13 on Saturday, 36 on Sunday, 54 on Monday).

Across those commits, about 46,400 lines were added and 54,000 removed. That overstates it a little, because one commit rewrote both automations.yaml and scripts.yaml in the current syntax, which touches nearly every line. Comparing just the first and last versions, 96 files changed: 22,111 lines in, 29,707 lines out. (That doesn't count the wall-tablet dashboards, which have their own post.)

	Start	Now
Automations	421	237
Scripts	206	98
Lines of YAML (config and packages)	~32,150	~21,000
`automations.yaml`	19,197 lines	12,165
`scripts.yaml`	4,892 lines	2,507
Feature packages	2	18
Scheduled automation runs	~389,000 a day	~5,900
Automations polling every few seconds	20 (one every second)	0
History rows written	~1.9 million a day	~1.1 million

Even after all of this, the house is big: today Home Assistant knows about 7,812 entities on 793 devices, across 256 integrations and helpers. Here's what went away along the way:

28 integrations and config entries: disabled leftovers (MyQ, VeSync, AdGuard, Glances, three remote Home Assistant links), two Samsung TV integrations, Waze (it had stopped working; travel times now come from my own routing server), Node-RED, an extra ChatGPT conversation agent, two old phones, and a HomeKit bridge with nothing left in it.
More than 600 orphaned entity registry entries, the ghosts automations and integrations leave behind when they're deleted. The old Pixel alone had 85.
About 100 helpers and timers that nothing read anymore, including the 48 from my old homemade timer system.
5 dashboards nobody used, including a "Test ground" whose buttons called a script that never existed.

What got added is mostly tooling: 38 plans, a validator, a side-by-side comparison script, the voice routing checker and the read-only export scripts: about 1,800 lines of new Python in all. That's the part I expect to keep using.

So, How Was Working With Claude?

Honestly? Great, with some caveats.

What it was good at:

Reading everything. I have never read all 32,000 lines of my own config. It did, more than once, and found things I'd forgotten I'd built.
Proving things. Every merge came with a simulation, every syntax conversion with a before/after comparison. That's the part I would have skipped if I were doing this alone, and it's the part that let me merge 180 automations' worth of changes without my family noticing.
Explaining. Every commit message says what changed, why, and how it was tested. Months from now, that's what I'll actually read.

What it needed from me:

Knowing the house. Which room the window AC is in, what "the green light" means to my kids, which sensors are flaky, that the basement is still being put back together after a flood. It asked a lot, and when it didn't, it was usually because the rules file already said.
Decisions. "Should the porch light stay on when it's hot out?" isn't a code question.
Running things. It never touched the live house. Every change went through me pasting a block of commands.

It also made mistakes: the indentation bug above, the packages detour, and one plan run that failed because Home Assistant's registry was bigger than the websocket library's 4 MB default message size. None of those made it past the dry run or the validator into something my family noticed, which is the whole point of building it this way.

What's Left

There's still a short list in REVIEW.md: a week of reading the log for warnings, a camera server that keeps dropping its streams, and the basement, which is still being set back up after the flood. (The last once-a-minute checks and the noisy database, which were on this list when I started writing, are done.) But the hard part is done. The config is in Git, every automation passes the validator, and the next time I want to change something, I don't have to be afraid of it.

The wall tablets got the same treatment, and I wrote that one up separately: Rewriting My Home Assistant Dashboards in Plain HTML.

If you've got a Home Assistant install that's been growing for years and you're scared to touch it, I'd really recommend this approach: get it into Git, write down your rules, and make every change prove itself before it goes anywhere near your house. Let me know in the comments if you have questions.

Thanks for reading.

Rewriting My Home Assistant Dashboards in Plain HTML

thehelpfulidiot — Sun, 04 Oct 2026 00:00:00 +0000

There are tablets on the walls all over my house. A Fire HD 10 in the family room, another one in the basement, a Lenovo StarView next to the TV that works as a remote, and one in each of the kids' rooms. (I've 3D printed mounts for these before.) They are how most of my family actually uses Home Assistant.

For years they all ran regular Home Assistant dashboards (Lovelace) in a kiosk browser. And on these tablets, Lovelace was heavy. Every one of them was loading the entire Home Assistant app just to show a few buttons. Taps would hang for a while before anything happened, and the pages kept reloading on their own, usually right when someone was trying to turn off a light.

So the dashboards got rewritten as plain HTML and JavaScript. No framework, no build step, no npm. Each tablet loads one small page that talks directly to Home Assistant.

And in the spirit of this blog's name, I should be upfront: I didn't write the code for these dashboards. Claude (the AI from Anthropic) basically wrote all of it. I told it what each tablet needed to do, tried every version on the actual tablets, and reported back what was broken or annoying. Then it fixed it. More on how that went at the end.

Here's the main one, in the family room:

The family room Fire HD 10. All of the data here is fake, from a mock Home Assistant (more on that at the end).

What's on the Screen

Each box is a "card," and most of them do a little more than they look like they do:

Weather with a five-day forecast, sunrise and sunset.
Calendar, which shows everyone's calendars together. When one of us is away from home, the same spot turns into a map showing where we are, then turns back into the calendar when we're both home.
Locks. Tap one to lock it. Each icon knows if the door is open too, so it won't try to throw a deadbolt into an open door. If the garage door is open, the whole row turns into a big red "Garage open, tap to close" button.
Thermostat, with the room temperatures from around the house underneath.
TV. While the TV is on, it's a remote with shortcuts for each input. When the TV is off, it turns into a big date and clock.
Light controls, one row per floor. More on that one below.
The kids. At night, each kid gets a face: red means they're in bed with their sound machine on, and green means their "OK to wake" light has turned on. Long-press a face to toggle that kid's sound machine. During the day this spot shows a camera instead.

The light menu counts the lights that are on per floor (2/4 means two of four). Tap the count and that whole floor turns off. Tap the name and you get the lights on that floor:

Tapping "First Floor" opens a slider for each room. It goes back to the floor list on its own after two minutes.

Here's the part I like: those floor counts come from Home Assistant's areas. I don't keep a list of lights on the dashboard. If I add a new bulb to the kitchen area in Home Assistant, the first-floor count just includes it. Lights hidden in Home Assistant (like the individual bulbs behind a light group) are left out so they aren't counted twice.

The Other Tablets

The portrait tablets all share the same building blocks. Here are the kid's room, the basement, and the family room TV remote:

A kid's room. Big buttons, nothing that can break anything important.

The basement. Same layout as the old Lovelace version, so nobody had to relearn it.

The StarView next to the TV is just a remote. The active source is highlighted.

That last one matters. My one hard rule was that nothing my family already knows how to use is allowed to change on them. So each new page started as a copy of the old Lovelace layout, button for button, and only then got cleaned up.

A Whole-House Intercom

Swipe left on any of the portrait tablets (or tap the green Calls button) and you get page two: an intercom.

Rooms show up on their own. A tablet that's asleep can be woken up from here.

It's a real voice call, right in the browser. Each room's page signs in to an Asterisk server I run, using SIP.js. During the day, calls are answered automatically with a beep (it's an intercom, after all). At night they wait quietly so nobody wakes a sleeping kid. The rooms aren't a hard-coded list either: a small monitor publishes each room to Home Assistant over MQTT, and the page finds them there.

How It Works

Every page is tiny. Here's roughly what one looks like:

<link rel="stylesheet" href="lib/theme.css?v=20261002-3">
<main class="dash">
  <div class="card" id="weather"></div>
  <div class="card" id="doors"></div>
</main>

<script src="lib/icons.js?v=20261002-3"></script>
<script src="lib/core.js?v=20261002-3"></script>
<script src="lib/cards/weather.js?v=20261002-3"></script>
<script src="lib/cards/locks.js?v=20261002-3"></script>
<script>
Dash.card('#weather', 'weather', { entity: 'weather.home' });
Dash.card('#doors', 'locks', {
  locks: [
    { name: 'Front Door', lock: 'lock.front_door' },
    { name: 'Back Door',  lock: 'lock.back_door' },
  ],
});
Dash.start({ haUrl: '' });   // blank = the Home Assistant that served this page
</script>

The files live in Home Assistant's own www folder, so Home Assistant serves them at /local/dashboard/<page>.html. There's no extra web server to run.

All the real work is in core.js, which is about 400 lines:

No polling, no refresh timer. The tablet just waits for Home Assistant to tell it something changed.

It opens one websocket to Home Assistant.
It looks through every card's settings for anything shaped like an entity ID (light.kitchen, lock.front_door...) and subscribes to only those. The tablet never hears about the other few thousand entities in my house.
When one of them changes, it redraws only the cards that use it. Everything else on the screen stays put.

Each card type is just a function in its own file that returns some HTML. Here's the "faces" card from the family room (the one that shows if the kids are asleep), simplified a little:

D.types.faces = function (el, cfg) {
  const list = cfg.list || [];
  return {
    deps: () => list.flatMap(k => [k.machine, k.green]).filter(Boolean),
    render() {
      return `<div class="faces">` + list.map(k => {
        const m = st(k.machine), color = m === 'on' ? (st(k.green) === 'on' ? 'var(--green)' : 'var(--red)') : '#e0e0e0';
        return `<button class="face" data-hold="toggle" data-e="${k.machine}" style="color:${color}">${I(k.icon)} ${esc(k.name)}</button>`;
      }).join('') + '</div>';
    },
  };
};

deps() says which entities should trigger a redraw, render() returns the HTML, and data-hold="toggle" is all it takes to make a long-press toggle the sound machine. That's basically the whole card.

How Small Is It?

The family room page, with every script and stylesheet it loads, is about 130 KB across 13 files, and around 42 KB compressed. About a third of that is the icons, which are the Material Design Icons I actually use, built into one file by a little script so the tablets don't download thousands of icons they'll never show.

Logging In

A page needs a Home Assistant long-lived access token. You open it once with the token at the end of the address:

https://<your-home-assistant>/local/dashboard/basement.html#token=eyJ...

The page saves the token in the tablet's browser storage and immediately removes it from the address bar, so it doesn't end up in history or a screenshot. After that it just works. If you build something like this yourself, make a separate, non-admin Home Assistant user just for the tablets and use that user's token. A tablet on a wall is the easiest thing in the house to walk off with.

The Boring Stuff That Makes It Reliable

Getting a dashboard to show up is easy. Getting it to keep working on a tablet that sits on a wall for months, through Wi-Fi drops, power cuts and Home Assistant restarts, is the hard part. A lot of the work with Claude went into exactly that, and each fix came from something that actually happened:

The intercom gave up after a power cut. The tablets boot faster than the server that hosts the intercom's SIP.js file. The page checked once, didn't find it, and showed "Intercom unavailable" until someone reloaded it by hand. Now it keeps retrying (5, 10, 20, 40, 60 seconds) and starts the intercom the moment the file shows up.
A websocket stuck in "connecting" forever. If the tablet's Wi-Fi drops in the middle of connecting, the socket can just sit there for minutes. Now it gets 15 seconds to log in, or it's closed and tried again.
An empty list means everything. If you ask Home Assistant to subscribe to an empty list of entities, it doesn't send you nothing. It sends you every entity in the house. A page with nothing to show now doesn't subscribe at all.
The month-long cache. Home Assistant tells browsers to keep files from /local/ for 31 days. So after I changed core.js, a tablet could keep running the old one for a month. Every page loads its scripts with a version stamp (the ?v=20261002-3 above), and a little script updates the stamp on every page at once after a change:

node tools/bump-version.js            # re-stamp every page after editing lib/
node tools/bump-version.js --check    # just report if any page is out of date

My PC stole the kids' calls. If I opened a kid's page on my computer to test something, my computer signed in to the intercom as that kid's room, and their calls could ring on my desk instead. Now adding ?intercom=dialer to the address lets a PC make calls without ever taking them, and ?intercom=off turns the intercom off completely.

There's also a screensaver: after a minute or two of no touches, the tablet shows a photo slideshow from my self-hosted Immich, and wakes back to the controls on the first tap. It's never allowed to kick in during a call.

About Those Screenshots

Every screenshot in this post came from the real dashboard code, but not from my real house. I didn't want to post my actual calendar, door states, or kids' names on the internet. So Claude wrote a small fake Home Assistant: a script that loads each page in a headless browser and answers the page's websocket messages with made-up data (a partly cloudy 68°, a back door that's unlocked, soccer practice at 5:30).

That fake turned out to be useful for more than blog posts. Most of the fixes above were tested exactly that way: a stand-in Home Assistant, and a stand-in intercom that fails to load twice before it works, to check that the page recovers on its own.

Letting Claude Write It

I'm not a web developer. I can read JavaScript well enough to follow along, but about 3,500 lines of it, plus an in-browser intercom, was never going to come out of me.

The dashboards live in their own Git repo on GitHub, and Claude worked on them the same way every time: it worked on a branch, explained what it changed and how it tested it, and I merged it and copied the folder into Home Assistant's www folder. Then I went and poked at the actual tablets.

That last part was my real job. Claude can test a page in a headless browser all day, but it can't tell that a button is too small for a kid's finger, or that the intercom stops working after the power goes out at 3 AM. Most of the fixes in the section above started with me saying "this thing did something weird," and Claude figuring out why.

Was It Worth It?

For us, yes. Taps respond right away, the random reloads are gone, the tablets come back on their own after the power goes out, and every page shows exactly what that room needs and nothing else. And because each page is just a short HTML file, changing a button is a one-line edit, not a trip through the dashboard editor on a tablet.

Would I tell everyone to do this? Probably not. Lovelace is still the right answer for most people, especially on a decent tablet or a phone, and it doesn't need any code. But if you have some older tablets on your walls that hang and reload like mine did, a page that only knows about the twenty entities it shows is a surprisingly small amount of code.

Let me know in the comments if you'd like me to go deeper on any of these cards, or the intercom.

Thanks for reading.

Goodbye WordPress, Hello Zola

thehelpfulidiot — Sat, 03 Oct 2026 00:00:00 +0000

It has been a while. My last post was the email backup update back in September 2024, and since then this blog has mostly just... sat there. Running. Asking me for updates.

That last part is the problem.

When I started this blog, it lived in a managed WordPress container on IONOS. Eventually it dawned on me that I already self host basically everything else, and I already had Traefik with Let's Encrypt certificates set up. Why was I paying someone else to host a blog? So I used a backup plugin to pack the whole site up, brought it home, and restored it into a TurnKey Linux WordPress LXC sitting behind Traefik.

That saved some money, but now I was the one keeping WordPress, Apache, PHP, MySQL, a theme, and a handful of plugins up to date, all to serve a few posts that almost never change. Every time I logged in there was a new red bubble asking me to update something.

Then I was listening to Linux Unplugged episode 683, heard them talking about Zola, and figured it was worth a shot. If you're reading this, you're looking at the result.

Same blog, three homes, a lot fewer moving parts

Why Zola

A static site generator takes a folder of Markdown files and spits out plain HTML. No database, no PHP, no admin login for bots to hammer on. The web server just hands out files.

There are a lot of these (Hugo, Jekyll, etc.), but I went with Zola. It's a single binary with no dependencies, and it builds this whole site in a few seconds. I tried it out on a practice site first before committing to moving this one.

For the theme, I went with tabi. I had been using a different theme on the practice site, but when I actually counted, this blog has 110 code blocks and 64 images. It's basically a pile of config files with some words in between. tabi comes with search, tag and category pages, and syntax highlighting with a copy button already built in. The other theme would have needed all three made by hand.

Every code block now gets a language label and a copy button. This is on my phone.

Not Doing It the Normal Way

The normal way to leave WordPress is to export everything to an XML file (WXR) and run it through a converter, or to just crawl your own site and scrape the HTML. Both of those make you un-mangle whatever the theme and the exporter did to your posts.

Since I have root on the container, I skipped all that. WP-CLI can read the raw post content straight out of the database, exactly as the editor saved it. So the whole migration came down to three scripts:

The whole migration, start to finish

I'll be honest, I did not sit down and write a block-aware WordPress converter in Python by myself. I leaned on Claude (the AI) pretty heavily for this whole project. It wrote most of the scripts, and I made the decisions, ran things, and broke things. Seems fitting for a blog called The Helpful Idiot.

Look Before You Leap

The first script, wp-recon.sh, doesn't change anything. It just surveys what's actually in there. I'm really glad we did this first, because it changed several decisions:

What I assumed	What was actually there
A mix of old and new editor content	100% Gutenberg blocks, zero classic HTML
A bunch of image galleries to untangle	No galleries at all, just 64 regular image blocks
146M of uploads to move	~95M. The other 51M was GeoIP databases from a stats plugin
A quick find-and-replace for my server's local IP	It shows up 11 times, and some of those are supposed to be there

That last one is the best example. The WordPress container's local IP had leaked into a few posts through image links. Claude's first suggestion was a database-wide search-and-replace. But recon showed that same IP is also sitting inside a few of my tutorials as a perfectly legitimate example config value. A blind find-and-replace would have quietly broken the instructions people actually copy and paste.

So instead, the converter only rewrites URLs inside image, link, and file blocks, and leaves code blocks byte-for-byte identical.

Converting the Posts

wp2zola.py is the real workhorse. Gutenberg saves each block with a little comment that says exactly what it is (paragraph, heading, code, image...), so conversion is mostly just "see the block, write the Markdown." Code blocks come out unescaped, images keep their captions, and every post gets its front matter written in TOML.

The thing I cared about most was not breaking any links. Some of these posts get linked from forums, and they're indexed by Google. So:

Posts keep their flat WordPress URLs (/some-post-name) by giving each one an explicit path
Categories and tags are named category and tag (singular!) so Zola builds /category/self-hosting/ and /tag/home-assistant/, exactly like WordPress did
Images stay at /wp-content/uploads/... so not a single image link had to change

Here's what the top of a converted post looks like now:

+++
title = "Playing a sound directly from Home Assistant Operating System"
date = 2021-10-02
path = "playing-a-sound-directly-from-home-assistant-operating-system"
description = "Playing a door chime through a speaker plugged into the Home Assistant box itself, using the VLC add-on over telnet."

[taxonomies]
category = ["Smart Home"]
tag = ["home assistant", "nest", "vlc"]
+++

Result: 19 files, zero blocks the converter didn't know how to handle.

Things That Broke Anyway

Of course it wasn't that easy. Everything below only showed up by actually running builds, not by reading documentation.

Home Assistant templates broke the build

Zola looks for its own template tags in your posts, including inside code blocks. Home Assistant uses almost the exact same {{ }} syntax, so a line like this from my remote control post made the build fall over:

value_template: "{{ trigger.to_state.attributes.button == 'KEY_POWER' }}"

The fix is to wrap any code block with curly braces in Zola's raw tags so it leaves them alone. Four posts needed it.

My series were numbered backwards

Zola sorts by date newest-first, so the first post of my email backup series proudly announced itself as "Part 4 of 4." tabi's documented fix is a little strange, but it works:

sort_by = "date"
paginate_by = 9999
paginate_reversed = true

Both series now live under /series, with automatic "Part 1 of 4" labels and next/previous links, while the posts themselves kept their old URLs.

Analytics were silently blocked

tabi ships with a strict Content Security Policy, which basically tells the browser "only run scripts that are files on this server." That's great for security, but it meant the little inline script that starts my analytics (Plausible, no cookies) just... didn't run. No error on the page, nothing. Moving it into its own .js file fixed it.

nginx doesn't like curly braces either

The old sidebar linked to WordPress's monthly archives (/2021/09/ and so on), and Google has those indexed. Zola doesn't make those pages, so they redirect to the archive page instead. My first attempt broke the nginx config, because nginx reads { as the start of a block, even inside a regex. The whole regex needs to be in quotes:

location ~ "^/[0-9]{4}(/[0-9]{2})?(/[0-9]{2})?/?$" { return 301 /archive/; }

Lesson learned (again): run nginx -t before restarting, not after.

Keeping Old Links Alive

Everything that matters (posts, categories, tags, page 2, images) has the exact same URL as before, so it needs no rules at all. The nginx config in front of the site only has to deal with the WordPress leftovers.

The most important one is the feed, since WordPress served it at /feed/ and people actually subscribe to it:

location = /feed  { return 301 /rss.xml; }
location = /feed/ { return 301 /rss.xml; }

The rest is WordPress's admin and login pages. Bots probe these constantly, and they now get a 410 Gone instead of a login page:

location ^~ /wp-admin    { return 410; }
location ^~ /wp-includes { return 410; }
location = /wp-login.php { return 410; }
location = /xmlrpc.php   { return 410; }

410 tells search engines the page is gone for good, and it's a tiny response instead of a full styled 404 page for every bot that comes knocking. Just be careful not to block all of /wp-content/, since that's where every image on this site still lives.

Cleaning Up

The last script, postfix.py, did the cleanup. It always runs as a dry run first, and it's safe to run twice. It:

Reworded a few sentences that only made sense next to WordPress comments
Pulled one workaround that only existed in a comment thread back into the post itself
Gave categories to ten posts that never had one
Built the two series
Added languages to 66 of the 110 code blocks, so they actually get syntax highlighting now

After that, there was a round of polishing: comments are now handled by giscus (they live in GitHub Discussions instead of my database), the site follows your phone's light/dark mode, the six images I had been hosting off my Synology got moved onto the site itself, and the phone menu got fixed after it refused to close on my iPhone.

Was It Worth It?

Absolutely. The whole site is now a folder of text files in a Git repo. To write a post, I write a Markdown file. To publish, I pull and rebuild. There's no database to back up, no plugins to update, and no login page for bots to brute force.

And apparently, a new platform was all it took to get me writing again.

If you're sitting on a WordPress blog that you mostly just maintain rather than write on, I'd highly recommend it. Let me know in the comments if you have any questions.

Thanks for reading.

Making an Automatic Email Backup (Updated 9/15/2024)

thehelpfulidiot — Sun, 15 Sep 2024 00:00:00 +0000

This is basically an update to the project I discussed in these three previous posts:

I have been using this method of backing up emails for the past several years and have been very please with the reliability. It has not required any modification or tinkering and has just seemed to work.

Having said that, even though it works reliably and doesn't need to be fixed, that doesn't mean I shouldn't try.

https://github.com/jon6fingrs/mbsync-dovecot

Enter this newest interation of my docker images for Mbsync and Dovecot. It uses updated versions of the software of course, but I have also made a few big changes which may be of use to some of you.

Solr

Now includes Solr integration!

Solr is an indexing and search platform. Previously, if performing a remote IMAP search through a generic IMAP client (some search their offline index), it would search each email sequentially and so could take a LONG time. Now, by deploying Solr at the same time, the search is MARKEDLY improved when I search using the Apple Mail app.

The solr docker container I have made includes the necessary setup for Dovecot, and the Dovecot container will look for the Solr container and so neither will now work without the other.

The solr container also includes cron and will write commits every 5 minutes and will optimize the database each night.

Mbsync Timing

Previously, the mbsync container would run once and then quit. Now, it will run to synchronize the mailbox, and then afterwards, it will open cron as the primary process, which will check for mail each 5 minutes. This avoids having to set a manual cronjob and keeps the container running.

If you want your own sync schedule, you can deactivate the built in cron with an environment variable and use whatever interval you would like.

Better Environment Variables

Some of the environment variables are now optional, and more intuitive. The run scripts are more robust.

What's the point?

But why did I do this? What's the point? Well, I would love to have my own mail server, but I am too worried about the administrative burden and dropped emails. This way, I feel like I am able to offload some of my dependence on the cloud by downloading and archiving every email, and just periodically purging my cloud provide inbox.

I have a complete email archive with all emails since my accounts opened. This is offline and does not require increasing my available cloud storage more and more.

Also, because I could.

So check it out if that sounds interesting and let me know how it works:

https://github.com/jon6fingrs/mbsync-dovecot

Nextcloud Docker with Nginx Proxy Manager

thehelpfulidiot — Sun, 31 Mar 2024 00:00:00 +0000

Nextcloud has become an integral part of my work flow these days. I love having a "place" to go that's totally mine. I have been running it from a docker container since before the AIO images were available and so have not had a reason to change.

The container comes in two flavors- Apache vs php-fpm. The I had been running the Apache version since it is generally easier, but always hated the idea of using Nginx Proxy Manager in front of Apache. It just seemed like a redundancy. Finally, I started to get some weird behaviors in Nextcloud which I tracked down as some webserver problem.

So now I have the perfect opportunity to fix something that really isn't breaking by doing wayyyy too much work.

I decided to try the php-fpm version and really wanted to take advantage of Nginx Proxy Manager and not have to use a second Nginx container!

Looking far and wide on the internet, most people said it was impossible, but I found really this single post that laid is out pretty nicely.

https://help.nextcloud.com/t/docker-nextcloud-fpm-behind-nginx-proxy-manager-do-i-need-another-nginx-container/133431/6

It turned out to be much easier than expected and I was essentially able to seamlessly swap out the Apache image for the php-fpm image by updating the docker compose file.

I did, however, update the nginx proxy configuration a bit from what was in that link to resolve various errors, to add the built-in collabora server, and to add push notifications.

	# set max upload size
        client_max_body_size 512M;
        fastcgi_buffers 64 4K;



	include mime.types;
	types 
	{
		application/javascript mjs;
	}

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Pagespeed is not supported by Nextcloud, so if your server is built
        # with the `ngx_pagespeed` module, uncomment this line to disable it.
        #pagespeed off;

        # HTTP response headers borrowed from Nextcloud `.htaccess`
        add_header Referrer-Policy                      "no-referrer"       always;
        add_header X-Content-Type-Options               "nosniff"           always;
        add_header X-Download-Options                   "noopen"            always;
        add_header X-Frame-Options                      "SAMEORIGIN"        always;
        add_header X-Permitted-Cross-Domain-Policies    "none"              always;
        add_header X-Robots-Tag                         "noindex, nofollow" always;
        add_header X-XSS-Protection                     "1; mode=block"     always;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        # Path to the root of your installation
        root /var/www/html;

        # Specify how to handle directories -- specifying `/index.php$request_uri`
        # here as the fallback means that Nginx always exhibits the desired behaviour
        # when a client requests a path that corresponds to a directory that exists
        # on the server. In particular, if that directory contains an index.php file,
        # that file is correctly served; if it doesn't, then the request is passed to
        # the front-end controller. This consistent behaviour means that we don't need
        # to specify custom rules for certain paths (e.g. images and other assets,
        # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
        # `try_files $uri $uri/ /index.php$request_uri`
        # always provides the desired behaviour.
        index index.php index.html /index.php$request_uri;

        # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
        location = / {
            if ( $http_user_agent ~ ^DavClnt ) {
                return 302 /remote.php/webdav/$is_args$args;
            }
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # Make a regex exception for `/.well-known` so that clients can still
        # access it despite the existence of the regex rule
        # `location ~ /(\.|autotest|...)` which would otherwise handle requests
        # for `/.well-known`.
        location ^~ /.well-known {
            # The rules in this block are an adaptation of the rules
            # in `.htaccess` that concern `/.well-known`.

            location = /.well-known/carddav { return 301 /remote.php/dav/; }
            location = /.well-known/caldav  { return 301 /remote.php/dav/; }

            location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
            location /.well-known/pki-validation    { try_files $uri $uri/ =404; }

            # Let Nextcloud's API for `/.well-known` URIs handle all other
            # requests by passing them to the front-end controller.
            return 301 /index.php$request_uri;
        }

        # Rules borrowed from `.htaccess` to hide certain paths from clients
        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }



        # Ensure this block, which passes PHP files to the PHP process, is above the blocks
        # which handle static assets (as seen below). If this block is not declared first,
        # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
        # to the URI, resulting in a HTTP 500 error response.
        location ~ \.php(?:$|/) {
            # Required for legacy support
            rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            set $path_info $fastcgi_path_info;

            try_files $fastcgi_script_name =404;

            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $path_info;
            #fastcgi_param HTTPS on;

            fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
            fastcgi_param front_controller_active true;     # Enable pretty urls
            fastcgi_pass nextcloud-nextcloud-1:9000;

            fastcgi_intercept_errors on;
            fastcgi_request_buffering off;
        }

    	location ~ \.(?:css|js|mjs|svg|gif|png|jpg|ico|wasm|tflite|map|ogg|flac)$ {
            try_files $uri /index.php$request_uri;
            expires 6M;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
    	}

        location ~ \.woff2?$ {
            try_files $uri /index.php$request_uri;
            expires 7d;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        # Rule borrowed from `.htaccess`
        location /remote {
            return 301 /remote.php$request_uri;
        }

	location ^~ /push/ {
        	proxy_pass http://nextcloud_notify:7867/;
        	proxy_http_version 1.1;
        	proxy_set_header Upgrade $http_upgrade;
        	proxy_set_header Connection "Upgrade";
        	proxy_set_header Host $host;
        	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        	}




        location / {
            try_files $uri $uri/ /index.php$request_uri;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Forwarded-Host $server_name;
        }

This has worked for Nextcloud 28.

Otherwise, the post pretty much explains everything else!

Fully Local Universal Remote Control with Home Assistant

thehelpfulidiot — Sun, 11 Feb 2024 00:00:00 +0000

Background

I have had a Logitech Harmony remote solution for almost 10 years. In my opinion, it is a very nice product. It integrates nicely into Home Assistant, and so, while it didn't behave as intuitively as I wanted out of the box, Home Assistant automations were able to modify its behavior so that it largely performed seamlessly in the way that I wanted.

Unfortunately, as with many other quality products, Logitech decided to discontinue the Harmony line of products.

Luckily, Logitech has learned from its previous mistake when it discontinued the Harmony Link years earlier. At the time, they announced that the devices would lose support and become bricks. This rustled a few jimmies and so they ultimately shipped a free Harmony Hub (the upgraded, but now current generation of IR hub) to the unhappy customers.

So even though Logitech has announced an end to the Harmony brand, they have stated that they will continue to support the devices currently in the wild. Well this is obviously a relief, but who can say how long we will get this reprieve before Logitech decides they've given us enough of their "free" computing power?

Recently, my dog ate one of my Harmony remotes. I had to purchase a used replacement from ebay. When it came, it was still paired to its previous hub, and trying to repair it set off a very difficult mission which required my downloading the official Harmony software onto a Windows Box so that I could restore my old profile.

It worked, but took several hours and was extremely frustrating to have to rely on a black box to fix something which had become integral to many automations and to my WAF. This got me thinking, how much longer do I have with Harmony? When will this next remote be destroyed? When will Harmony's recovery tools become out-dated and unusable?

This set me off on searching for a more stable, long-term solution.

Alternatives

So what are the alternatives? Well surprisingly few.

There appears to be a pretty popular Harmony replacement, called the Sofabaton. It seems to have received pretty positive reviews all around, and if I'm being honest, I didn't really look into it too much. Regardless of whether or not it behaves locally, I just saw it as trading one walled garden for another. It seemed like you would need an internet connection and the company's servers in order to add new devices. I wanted something that would be totally local.

Immediately, I was drawn to the Broadlink RM4 mini. It is a cheap IR blaser which can learn from IR remotes, or can use codes found online. It integrates very nicely into Home Assistant, and works over a local connection. I attempted to set it up normally, but there was some issue. The device was added to my wifi but not to my Broadlink account. However, this turned out to be the perfect outcome since I was able to integrate it into Home Assistant and now do not need to worry about a device calling home constantly.

The IR blaser is for my TV and soundbar which are not controllable by wifi. Otherwise, I have a kodi box, an Apple TV, and a Roku. These are all controllable over wifi.

So now, we have a way to interact with my media devices. I can use it through various automations, from my phone, a computer, etc. But I really just want a remote. There is nothing quite like controlling your media with an old-fashioned remote. It is always there and the batteries last like 10 years.

Perhaps there are bluetooth remotes available? My Home Assistant instance is in the basement while the media center is on the first floor. So if I am able to find a bluetooth remote, there needs to be a way to forward all commands to my Home Assistant instance.

Once that's done, it's just a matter of connecting the correct input with the correct output which would be fairly easy.

Solution: Remote Hardware

After much searching, I came across the Air Mouse Remote MX3 Pro with Backlit. It is not Bluetooth. I actually could not find a decent Bluetooth Remote. It does still use a 2.4ghz signal. It comes with a pre-paired USB dongle. Just plug it into a computer, and BOOM- the computer has a hardware keyboard device added.

Another nice feature is it has 5 programmable IR buttons at the bottom for backup in the event something isn't working.

There are many iterations of this remote. It seems to be a white labelled device and is just branded and repackaged by various companies. I did try several of them to try and find the best, but to be honest, they are all the same. I simply prefer the remote with the backlight versus without (although you do lose a button).

Problem: Send keypresses to Home Assistant

Ok, so the next problem. The remote is plugged into my Kodi box (Manjaro) in the family room and my Home Assistant box is in the basement. How do I get keypresses from one to the other?

I have played with USBIP in the past and, I don't like it very much. In my experience (and I am far from an expert), it works great until it doesn't. It doesn't really have a way to catch and recover from errors. It just wasn't the sort of stable, low maintenance solution I wanted (but that may have been an error on my part).

After searching far and low, I came across the git:

https://github.com/tjntomas/MI-Bluetooth-Remote-for-Home-Assistant.

It is a script for capturing a keyboard presses, and sending them to Home Assistsant so that each key_down and key_up event is converted to a Home Assistant event. From there, the hardware and "wiring" will be complete and it is a matter of using Home Assistant to create whatever series of automations for each key that your particular setup requires.

To use it, follow the instructions on the original repo. You need to make sure python is installed and that the python evtest module is installed.

The script wasn't perfect for me and so I made just a few changes. I wanted to specify the keyboard using a commandline argument since the remote I linked above creates three keyboard devices in linux. This is just easier that creating three seperate scripts. I also made sure that it would send the key code, along with the key name and type of press in the event data.

The following is a fork of the original git repo with the above mentioned and some other changes.

https://github.com/jon6fingrs/Bluetooth-Remote-for-Home-Assistant/tree/main

Here is my version of bt_remote_event.py:

"""
Script to listen to events from a bluetooth remote control and send
the events to Home Assistant.

Author: Tomas Jansson, https://github.com/tjntomas

"""

import json
import evdev # https://pypi.org/project/evdev/
import asyncio
import aiohttp
import logging
import os
import sys

# Set up logging and log levels.
logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
logging.getLogger('asyncio').setLevel(logging.CRITICAL)

BASE_API      = "http://IP_ADDRESS:8123/api/"  # URL to your HA instance.
DEV_INPUT     = "/dev/remote_"             
API_KEY       = "API_KEY_STRING"
HA_EVENT_NAME = "bt_remote" # Arbitrary name of the event that will get fired.
GRAB_DEVICE   = True # If set to True, the devices will be locked to this script and the system will not receive any events.

EVENT_PATH    = "events/" + HA_EVENT_NAME
BASE_API_URL  = BASE_API + EVENT_PATH
HEADERS       = {'content-type': 'application/json','Authorization': 'Bearer {}'.format(API_KEY)}
CMD           = "cmd"
CMD_TYPE      = "cmd_type"
CMD_NUM       = "cmd_num"

EVENT_LOG_TEMPLATE = "Fired event {} with event data{}"

async def run():
  device = evdev.InputDevice(DEV_INPUT + sys.argv[1])
  print("Using Bluetooth", str(device))
  
  if GRAB_DEVICE:
    device.grab()

  # Listen for events from the remote through the async-based evdev library.
  for event in device.read_loop():
    if event.type == evdev.ecodes.EV_KEY:   
      # Get the name of the remote key pressed, one of:
      # KEY_HOME    
      # KEY_F5      
      # KEY_COMPOSE   
      # KEY_BACK       
      # KEY_UP          
      # KEY_DOWN   
      # KEY_LEFT      
      # KEY_RIGHT    
      # KEY_ENTER      
      # KEY_VOLUMEUP   
      # KEY_VOLUMEDOWN 

      # The event string returned from the evdev library looks like this:
      # key event at 1609103448.769025, 28 (KEY_ENTER), down
      # We are only interested in the key name "KEY_ENTER" and the keypress type "down".
      cmd = str(evdev.categorize(event)).split(",")[1].split("(")[1].replace(")","")

      cmd_num = str(evdev.categorize(event)).split(",")[1].split("(")[0].replace(" ","")

      # Get the type of keypress, one of:
      # up
      # down
      # hold
      cmd_type = str(evdev.categorize(event)).split(",")[2].replace(" ","")

      # Compose the payload to send to HA when firing the event.
      payload = {CMD: cmd, CMD_TYPE: cmd_type, CMD_NUM: cmd_num}

      # Since the evdev library is async-based, we use async to send the event to HA.
      async with aiohttp.ClientSession() as session:
        await session.post(BASE_API_URL, data=json.dumps(payload), headers=HEADERS)
        logging.info(EVENT_LOG_TEMPLATE.format(HA_EVENT_NAME,payload))
        await session.close()

if __name__ == '__main__':
    # Create, start and gracefully shut down the asyncio event loop.
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(run())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

So, first we need to figure out what our keyboards are with a tool called evtest. On Ubuntu, it can be installed with:

sudo apt install evtest

This will list the name and location of all connected input devices, like the following:

No device specified, trying to scan all of /dev/input/event*
Not running as root, no devices may be available.
Available devices:
/dev/input/event0: Sleep Button
/dev/input/event1: Power Button
/dev/input/event10: Dell WMI hotkeys
/dev/input/event11: Video Bus
/dev/input/event12: DP-1
/dev/input/event13: HDA Intel PCH Headphone Mic
/dev/input/event14: HDA Intel PCH Front Line Out
/dev/input/event15: HDA Intel PCH HDMI/DP,pcm=3
/dev/input/event16: HDA Intel PCH HDMI/DP,pcm=7
/dev/input/event17: HDA Intel PCH HDMI/DP,pcm=8
/dev/input/event18: ydotoold virtual device
/dev/input/event2: Power Button
/dev/input/event3: flirc.tv flirc Keyboard
/dev/input/event4: 2.4G Composite Devic
/dev/input/event5: 2.4G Composite Devic Mouse
/dev/input/event6: 2.4G Composite Devic Consumer Control
/dev/input/event7: 2.4G Composite Devic System Control
/dev/input/event8: Pulse-Eight USB-CEC Adapter
/dev/input/event9: PC Speaker
Select the device event number [0-18]:

The "four" devices which represent my remote are:

/dev/input/event4: 2.4G Composite Devic
/dev/input/event5: 2.4G Composite Devic Mouse
/dev/input/event6: 2.4G Composite Devic Consumer Control
/dev/input/event7: 2.4G Composite Devic System Control

The devices at 4, 6, and 7 represent the keyboards. The remote also can toggle into a gyroscopic mouse. This actually works out great for me since it is connected directly to my kodi box, so even though we will continue with the complicated setup that prevents the keyboards from directly interacting with the system, the mouse will still work if desired.

Anyway, if you try to access one of these devices, you will get:

Testing ... (interrupt to exit)
***********************************************
  This device is grabbed by another process.
  No events are available to evtest while the
  other grab is active.
  In most cases, this is caused by an X driver,
  try VT-switching and re-run evtest again.
  Run the following command to see processes with
  an open fd on this device
 "fuser -v /dev/input/event6"
***********************************************

Ok, so there are two things we have to accomplish.

we need to deactivate the devices
we need to give each device a stable name (eventX could change)

Luckily, there is one solution to accomplish both of these things- udev! I have added an example udev rule to my above linked git.

SUBSYSTEMS=="input", ATTRS{name}=="keyboard name from evtest", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_abc"

Each device will need its own file. Maybe it could be done with one, but I am not an expert, I did it with three. It is a one time procedure and then you forget about it. If you have a nice way to accomplish the same with one file, however, please let me know.

Anyway, my subsequent three files will look like:

SUBSYSTEMS=="input", ATTRS{name}=="2.4G Composite Devic Consumer Control", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_consumer"

SUBSYSTEMS=="input", ATTRS{name}=="2.4G Composite Devic System Control", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_system"

SUBSYSTEMS=="input", ATTRS{name}=="2.4G Composite Devic", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_keyboard"

For the symlink portion, keep "remote_" the same or else it will need to be changed in the script, but you can make the string after the underscore whatever you want. I chose remote_consumer, remote_system, and remote_keyboard just so I could remember which was which.

Ok so those three files can be named whatever (99-ignore-my device-whatever.rules) and they should be placed in /etc/udev/rules.d.

You can restart your system or enter the following command as root to activate the new rules:

udevadm control --reload-rules && udevadm trigger

Now, when you try and use evtest, it will be able to capture any of the three keyboards.

There are now a few modifications you have to make to the script so that it works for your setup.

change the BASE_API to point to your instance's API endpoint
from home assistant, create a long-lasting token and paste it in the API_KEY

So now, you could run the script to control one keyboard at a time and have each click send to home assistant by typing:

python3  /path_to_your_script/bt_remote_event.py %i

The argument is whatever keywords you decided to use after "remote_". For example "bt_remote_event.py keyboard" and it will run the script for the "keyboard" device.

One caveat to what we have done so far, even though the three keyboards are not being used by the system, the power button on the remote will still trigger whatever action your computer typically triggers when its power button is pressed. Therefore, I used the option in Gnome to disable the power button. This does have the consequence of even disabling the button on the PC itself, but that is not an issue for me. I am not sure if there is a way to only disable the power button only from the remote.

So now with the keyboard script running, you should be able to go into your home assistant instance, the developer's tools, events, and listen for the "bt_remote" events.

Hopefully you see your button presses and notice each press gives you two events:

event_type: bt_remote
data:
  cmd: KEY_HOMEPAGE
  cmd_type: down
  cmd_num: "172"
origin: REMOTE
time_fired: "2024-02-11T19:53:40.964943+00:00"
context:
  id: sdfgsdfgsdfgsdfgsd
  parent_id: null
  user_id: sdfgsdfgsdfgsdfgsd

event_type: bt_remote
data:
  cmd: KEY_HOMEPAGE
  cmd_type: up
  cmd_num: "172"
origin: REMOTE
time_fired: "2024-02-11T19:53:41.026000+00:00"
context:
  id: sdfgsdfgsdfgsdfgsdfgsdgsdf
  parent_id: null
  user_id: ddsfgsdfgsdfgsdfgsdfgsdfg

Each button will give you its down and then up events separately, and each even will give the key name, the button direction, and the raw key code. I added the key code because, for my setup, knowing the keycode was helpful for some subsequent actions.

Great so now that we know it works, how can we exit the running bt_remote_event script. We have to set up three scripts and make sure they start automatically. In the git repository, you should see btmon@.service. This will allow us to capture all three devices.

First, copy it to /etc/systemd/system/

sudo cp btmon@.service /etc/systemd/system/

Now you simply enable it three times for the three devices we named earlier:

sudo systemctl enable --now btmon@consumer.service
sudo systemctl enable --now btmon@system.service
sudo systemctl enable --now btmon@keyboard.service

Perfect! Now all those presses are headed straight for Home Assistant. The PC where the USB is physically plugged can be restarted and the scripts will pick up where its left off.

Solution: Represent Keypresses in Home Assistant

Ok, so now Home Assistant is getting all those key press events, but so what? I wanted a way to have a single press and a long press. The long press shouldn't set off the single press, and the single press shouldn't be more delayed than absolutely necessary.

First, I made a trigger based binary sensor to more easily display the button presses and the necessary information:

  - trigger:
      - platform: event
        event_type: bt_remote
        event_data:
          cmd_type: "down"
        id: "on"
      - platform: event
        event_type: bt_remote
        event_data:
          cmd_type: "up"
        id: "off"
    binary_sensor:
      - name: remote_button_press
        state: "{{ trigger.id }}"
        attributes:
          button: "{{ trigger.event.data.cmd }}"
          code: "{{ trigger.event.data.cmd_num }}"
          timestamp_up: >
            {% if trigger.id == 'off' %}
              {{ as_timestamp( trigger.event.time_fired)|float(default=0) }}
            {% else %}
              {{ this.attributes.timestamp_up|float(default=0) }}
            {% endif %}
          timestamp_down: >
            {% if trigger.id == 'on' %}
              {{ as_timestamp( trigger.event.time_fired)|float(default=0) }}
            {% else %}
              {{ this.attributes.timestamp_down|float(default=0) }}
            {% endif %}
          pressed_duration: >
            {% if trigger.id == 'on' %}
              0
            {% else %}
              {{ as_timestamp( trigger.event.time_fired)|float(default=0) - (this.attributes.timestamp_down)|float(default=0) }}
            {% endif %}

This creates a single binary_sensor. It is on during the time the button is pressed down, and off when it is released, so off the vast majority of the time. However, in its attributes, it has the current or last key and code pressed, but also the amount of time the key was pressed down in seconds (0 if the keys is pressed down at that moment). This has created a very easy system to make automations.

Now, as you probably guessed, to set everything up correctly, to identify each button and decide what you want it to do, it takes time. There also is not a one size fits all solution.

However, I will show examples of what I have done in case it can help anyone else.

Respond to key presses

So I essentially have two automations. One for single presses and one for double presses.

The single press is actually triggered when our binary_sensor goes off, meaning that someone has just released a button. I have chosen a hold time of 0.5 seconds to represent a button hold. Anything shorter is a single press, and longer is a hold. Therefore, for this automation, I want to make sure we only act if the button was released before the hold automation is activated.

The following is my trigger and condition:

alias: kodi remote
description: ""
trigger:
  - platform: state
    entity_id:
      - binary_sensor.remote_button_press
    from: "on"
    to: "off"
condition:
  - condition: numeric_state
    entity_id: binary_sensor.remote_long_press
    attribute: pressed_duration
    below: 0.5

Then I just have a giant choose between 30 actions. Again, it will be different for everyone, but I will show it before to highlight a couple things:

action:
  - choose:
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_POWER' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: binary_sensor.vizio_tv_on
                    state: "on"
                sequence:
                  - service: script.all_off
                    data: {}
            default:
              - if:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: power_off
                then:
                  - service: select.select_option
                    target:
                      entity_id: select.harmony_hub_2_activities
                    data:
                      option: Watch Kodi
                else:
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: PowerOn
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEUP' }}"
        sequence:
          - service: media_player.volume_up
            data: {}
            target:
              entity_id: media_player.family_room_tv
            enabled: false
          - service: script.increase_tv_volume_x1
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEDOWN' }}"
        sequence:
          - service: media_player.volume_down
            data: {}
            target:
              entity_id: media_player.family_room_tv
            enabled: false
          - service: script.decrease_tv_volume_x1
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PLAYPAUSE' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: media_player.family_room_speaker
                    state: playing
                  - condition: not
                    conditions:
                      - condition: state
                        entity_id: media_player.family_room_tv
                        state:
                          - paused
                          - playing
                sequence:
                  - service: media_player.media_play_pause
                    data: {}
                    target:
                      entity_id: media_player.family_room_speaker
            default:
              - service: media_player.media_play_pause
                data: {}
                target:
                  entity_id: media_player.family_room_speaker
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PREVIOUSSONG' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: media_player.family_room_speaker
                    state: playing
                  - condition: not
                    conditions:
                      - condition: state
                        entity_id: media_player.family_room_tv
                        state:
                          - paused
                          - playing
                sequence:
                  - service: media_player.media_previous_track
                    target:
                      entity_id: media_player.family_room_speaker
                    data: {}
            default:
              - service: media_player.media_previous_track
                target:
                  entity_id:
                    - media_player.family_room_tv
                  device_id: []
                  area_id: []
                data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_NEXTSONG' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: media_player.family_room_speaker
                    state: playing
                  - condition: not
                    conditions:
                      - condition: state
                        entity_id: media_player.family_room_tv
                        state:
                          - paused
                          - playing
                sequence:
                  - service: media_player.media_next_track
                    target:
                      entity_id: media_player.family_room_speaker
                    data: {}
            default:
              - service: media_player.media_next_track
                target:
                  entity_id:
                    - media_player.family_room_tv
                  device_id: []
                  area_id: []
                data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_CONFIG' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: Watch Kodi
                sequence:
                  - if:
                      - condition: state
                        entity_id: binary_sensor.vizio_tv_on
                        state: "off"
                    then:
                      - service: remote.send_command
                        target:
                          entity_id: remote.family_room_broadlink
                        data:
                          num_repeats: 1
                          delay_secs: 0.4
                          hold_secs: 0
                          device: Vizio TV
                          command: PowerOn
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 0
                          milliseconds: 200
                      - service: remote.send_command
                        data:
                          device: Vizio Amp
                          command: InputOptical
                        target:
                          entity_id: remote.family_room_broadlink
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 15
                          milliseconds: 0
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: InputHdmi5
            default:
              - service: select.select_option
                target:
                  entity_id: select.harmony_hub_2_activities
                data:
                  option: Watch Kodi
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_MAIL' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: Watch Roku
                sequence:
                  - if:
                      - condition: state
                        entity_id: binary_sensor.vizio_tv_on
                        state: "off"
                    then:
                      - service: remote.send_command
                        target:
                          entity_id: remote.family_room_broadlink
                        data:
                          num_repeats: 1
                          delay_secs: 0.4
                          hold_secs: 0
                          device: Vizio TV
                          command: PowerOn
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 0
                          milliseconds: 200
                      - service: remote.send_command
                        data:
                          device: Vizio Amp
                          command: InputOptical
                        target:
                          entity_id: remote.family_room_broadlink
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 15
                          milliseconds: 0
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: InputHdmi3
            default:
              - service: select.select_option
                target:
                  entity_id: select.harmony_hub_2_activities
                data:
                  option: Watch Roku
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_WWW' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: Watch Apple TV
                sequence:
                  - if:
                      - condition: state
                        entity_id: binary_sensor.vizio_tv_on
                        state: "off"
                    then:
                      - service: remote.send_command
                        target:
                          entity_id: remote.family_room_broadlink
                        data:
                          num_repeats: 1
                          delay_secs: 0.4
                          hold_secs: 0
                          device: Vizio TV
                          command: PowerOn
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 0
                          milliseconds: 200
                      - service: remote.send_command
                        data:
                          device: Vizio Amp
                          command: InputOptical
                        target:
                          entity_id: remote.family_room_broadlink
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 15
                          milliseconds: 0
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: InputHdmi2
            default:
              - service: select.select_option
                target:
                  entity_id: select.harmony_hub_2_activities
                data:
                  option: Watch Apple TV
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_BACK' }}"
        sequence:
          - service: script.kodi_remote_back
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_UP' }}"
        sequence:
          - service: script.kodi_remote_up
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_DOWN' }}"
        sequence:
          - service: script.kodi_remote_down
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_LEFT' }}"
        sequence:
          - service: script.kodi_remote_left
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_RIGHT' }}"
        sequence:
          - service: script.kodi_remote_right
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ENTER' }}"
        sequence:
          - service: script.kodi_remote_enter
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_HOMEPAGE' }}"
        sequence:
          - service: script.kodi_remote_home
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_FASTFORWARD' }}"
        sequence:
          - service: script.kodi_remote_fastforward
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_REWIND' }}"
        sequence:
          - service: script.kodi_remote_rewind
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_COMPOSE' }}"
        sequence:
          - service: script.kodi_remote_info
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_SEARCH' }}"
        sequence:
          - service: script.kodi_remote_search
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMIN' }}"
        sequence:
          - service: script.kodi_remote_toggle_subtitles
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMOUT' }}"
        sequence:
          - service: script.kodi_remote_audio_track
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEDOWN' }}"
        sequence:
          - service: script.kodi_remote_page_down
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEUP' }}"
        sequence:
          - service: script.kodi_remote_page_up
            data: {}
      - conditions:
          - condition: template
            value_template: "    {{ trigger.to_state.attributes.button.split(\"_\")[1]|length == 1 }}"
        sequence:
          - service: script.kodi_remote_letter
            data:
              key_pressed: |
                {{ trigger.to_state.attributes.button.split("_")[1]|lower }}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_BACKSPACE' }}"
        sequence:
          - service: script.kodi_remote_backspace
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_SPACE' }}"
        sequence:
          - service: script.kodi_remote_space
            data:
              key_pressed: 57
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ESC' }}"
        sequence:
          - service: script.kodi_remote_gen_button
            data:
              key_pressed: 1
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_DOT' }}"
        sequence:
          - service: script.kodi_remote_gen_button
            data:
              key_pressed: 52
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_COMMA' }}"
        sequence:
          - service: script.kodi_remote_gen_button
            data:
              key_pressed: 51

The majority of my actions involve identifying the button by name by matching to one of the binary sensor's attributes. Then I usually call for a script just to make things a little cleaner. There is some logic in this spiderweb too. It really just depends and I'm sorry I can't make this cleaner.

I want to point out one of the options though. The back of the remote has a keyboard! Wouldn't it be great to use it? I really don't want to make an option for every letter in the alphabet. Instead, I made a single option to capture anytime the key name is a single letter:

      - conditions:
          - condition: template
            value_template: "    {{ trigger.to_state.attributes.button.split(\"_\")[1]|length == 1 }}"
        sequence:
          - service: script.kodi_remote_letter
            data:
              key_pressed: |
                {{ trigger.to_state.attributes.button.split("_")[1]|lower }}

This will send the letter by itself to a special script I have which sends the letter wherever its needed depending on what device I am using. For single presses it is always a lower case letter, and as you will see, for long presses it is an upper case letter.

The long press automation is a little different. There are two situations to consider. One is a long press for something like volume. You want it to continue raising or lowering the volume as long as you are pressing it. Second is a long press which is simply a way to use the same button for a second action. In that case, we want it to perform a single action and then stop.

The automation is somewhat similarly triggered:

platform: state
entity_id:
  - binary_sensor.remote_button_press
for:
  hours: 0
  minutes: 0
  seconds: 0.5
from: "off"
to: "on"

And my action block is in a repeat action and will continue to repeat, unless stopped, as long as the button remains "on" (pressed), and will stop when it is "off" released.

repeat:
  sequence:
    - choose:
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEUP' }}"
          sequence:
            - service: script.increase_tv_volume_x1
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEDOWN' }}"
          sequence:
            - service: script.decrease_tv_volume_x1
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_PLAYPAUSE' }}"
          sequence:
            - choose:
                - conditions:
                    - condition: state
                      entity_id: media_player.family_room_speaker
                      state: playing
                    - condition: not
                      conditions:
                        - condition: state
                          entity_id: media_player.family_room_tv
                          state:
                            - paused
                            - playing
                  sequence:
                    - service: media_player.media_stop
                      target:
                        entity_id:
                          - media_player.family_room_speaker
                      data: {}
              default:
                - service: script.kodi_remote_stop
                  data: {}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_UP' }}"
          sequence:
            - service: script.kodi_remote_up
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_DOWN' }}"
          sequence:
            - service: script.kodi_remote_down
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_LEFT' }}"
          sequence:
            - service: script.kodi_remote_left
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_RIGHT' }}"
          sequence:
            - service: script.kodi_remote_right
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_FASTFORWARD' }}"
          sequence:
            - service: script.kodi_remote_fastforward
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_REWIND' }}"
          sequence:
            - service: script.kodi_remote_rewind
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMIN' }}"
          sequence:
            - service: script.kodi_remote_switch_subtitles
              data: {}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMOUT' }}"
          sequence:
            - service: media_player.volume_mute
              target:
                entity_id: media_player.family_room_tv
              data:
                is_volume_muted: false
            - stop: ""
        - conditions:
            - condition: template
              value_template: "    {{ trigger.to_state.attributes.button.split(\"_\")[1]|length == 1 }}"
          sequence:
            - service: script.kodi_remote_letter
              data:
                key_pressed: |
                  {{ trigger.to_state.attributes.button.split("_")[1] }}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_COMPOSE' }}"
          sequence:
            - service: script.kodi_remote_menu
              data: {}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_BACKSPACE' }}"
          sequence:
            - choose:
                - conditions:
                    - condition: state
                      entity_id: sensor.active_player
                      state: media_player.apple_tv
                  sequence:
                    - service: shell_command.pyatv_root
                      data:
                        command: text_clear
                    - stop: ""
              default:
                - service: script.kodi_remote_gen_button
                  data:
                    key_pressed: 14
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEDOWN' }}"
          sequence:
            - service: script.kodi_remote_page_down
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEUP' }}"
          sequence:
            - service: script.kodi_remote_page_up
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_WWW' }}"
          sequence:
            - choose:
                - conditions:
                    - condition: state
                      entity_id: select.harmony_hub_2_activities
                      state: Play Nintendo
                  sequence:
                    - if:
                        - condition: state
                          entity_id: binary_sensor.vizio_tv_on
                          state: "off"
                      then:
                        - service: remote.send_command
                          target:
                            entity_id: remote.family_room_broadlink
                          data:
                            num_repeats: 1
                            delay_secs: 0.4
                            hold_secs: 0
                            device: Vizio TV
                            command: PowerOn
                        - delay:
                            hours: 0
                            minutes: 0
                            seconds: 15
                            milliseconds: 0
                    - service: remote.send_command
                      target:
                        entity_id: remote.family_room_broadlink
                      data:
                        num_repeats: 1
                        delay_secs: 0.4
                        hold_secs: 0
                        device: Vizio TV
                        command: InputHdmi4
              default:
                - service: select.select_option
                  target:
                    entity_id: select.harmony_hub_2_activities
                  data:
                    option: Play Nintendo
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_CONFIG' }}"
          sequence:
            - service: automation.trigger
              metadata: {}
              data:
                skip_condition: true
              target:
                entity_id: automation.turn_on_family_room_speaker
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_ENTER' }}"
          sequence:
            - service: script.kodi_remote_return
              data: {}
            - stop: ""
    - delay:
        hours: 0
        minutes: 0
        seconds: 0
        milliseconds: 100
  while:
    - condition: state
      entity_id: binary_sensor.remote_long_press
      state: "on"

I added a short delay in the loop to make sure the button presses weren't being sent too quickly. You can see also that some of the actions are followed by a "stop" action when they are intended to run as a one-and-done action. Otherwise, the action will continue to repeat until the button is released.

Summary

So, at this point, we have a 2.4ghz remote. The button presses are ignored by the host system and sent to home assistant as events. Home assistant then distinguished between long and short presses and enacts whatever action is necessary. Obviously, these buttons don't have to only act on my media center. The buttons can control anything in the house.

My setup basically allows the remote to work as intended based on what is playing on TV. That way, nobody ever has to switch inputs or profiles or whatever. It just works. I have noticed that my family gravitates towards it even instead of the harmony remotes. I have been very pleased with it so far which is why I wanted to share.

In the future, I can give examples of how I send the commands (including keyboard presses) seemlessly to kodi, roku, or apple tv. Those (especially apple tv) were tricky to figure out.

On a different kodi box, I have a separate remote set up and just have it send the keypresses back. That way, only the keys I want to modify are modified by home assistant while the remote buttons otherwise work for the kodi box just like they would without all this complicated work around.

I'm sure there are many more possibilities and would be excited to see what other people come up with.

Links

Air Mouse Remote MX3 Pro with Backlit

Associate Link - https://amzn.to/487vXMU

Store Link - https://www.amazon.com/gp/product/B08YRL56BT

Rii MX3 Multifunction

Associate Link - https://amzn.to/3OGErnr

Store Link - https://www.amazon.com/gp/product/B01CL3ZXGO

Create a FREE public endpoint for self-hosted services using Oracle Cloud and OPNsense

thehelpfulidiot — Thu, 15 Sep 2022 00:00:00 +0000

Introduction

As many of you (the people who would be interested enough to read this post), I have a number of self hosted services which have become integral to my everyday life. For the most part, I use a wireguard (split) tunnel to access these services remotely from my phone or laptop. However, there are some services, like Nextcloud, for which I prefer public access so that I can access them from any computer. Up till now, I have opened port 443 on my router and forwarded incoming traffic to an instance of Nginx Proxy Manager. This has been working fine for the most part, but I have a residential ISP with a dynamic IP address. Even though I have my incoming connections go through my OPNsense router with crowdsec and geo block lists, I just never liked the idea of opening ports on my router.

Original Network Layout

To solve this, I became enamored with the idea of setting up a public IP address on a VPS which would send the connection requests to my NPM instance through a wireguard tunnel.

There are plenty of tutorials how to forward connections through Wireguard, and there is an amazing tool to do almost exactly what I described. However, the tool requires a linux endpoint on the local network as opposed to sending all connections to OPNsense. My ideal plan would have the VPS connections first passing through OPNsense where they could be acted upon as any other incoming connection.

New Network Layout

So ultimately, at the end of this tutorial, we will have a static IP address through a free oracle cloud virtual machine. It will forward whichever connections we want to OPNsense. I'm sure this could be configured as a VPN endpoint for all connections if desired, but that won't be covered here.

Oracle Cloud

Ok, so first thing we need to do is set up our virtual machine on the Oracle Cloud platform.

https://www.oracle.com/cloud/

Oracle has a free tier and "Always Free Resources". Obviously, they could always take these free resources as many cloud providers eventually do, but lets hope if it happens, it doesn't happen for a while.

Always free VM

So first step, create an account.

Instead of going straight to making a new compute instance, I had to make a "compartment". I think that's how the compute instances are clustered. To be honest, I am not sure that I actually needed to make a new compartment but had trouble getting my machine set up without it.

From the top left hamburger menu, go to Compartments.

From there, create a new compartment. Name it whatever you want, and for parent compartment, select your root.

Next click on the big Oracle Cloud image top left to go back to the main page. Now we will set up our compute instance.

Click Instances

Click the "Create Instsance" button.

On the next page, we will set up our machine and make sure to select options which are part of the "Always Free" subscription.

Under "create in compartment," be sure to select the compartment we just set up.

Keep the defaults for Placement.

Now for Image & Shape, we will change some options. For the purposes of this tutorial, I will use the Ubuntu image. It's free and it's what I am familiar with.

Under Shape Series, select "Ampere" and let's max out our machine specs under the Free Tier.

Free tier specs

Click select shape and proceed. Keep networking options as default.

Be sure to save your SSH key! It will be the only way to log in to your machine. Then leave the boot volume as default and we are done setting up our machine!

Now back at the instances screen, you should see your new instance along with its public IP address and its internal IP address. We will use the public IP address to SSH in, but first lets adjust the firewall settings.

Click on your instance name. Under "Primary VNIC", then subnet, there should be a hyperlink to the subnet your machine is using. Click on it. Scroll down and click on the default security list. I erased all of the default firewall rules and replaced them with a rule to allow all traffic as we can just rely on ufw to protect our machine.

Open Ingress Rules

Ok perfect, now lets get into our machine. We are going to use the Public IP address to SSH in to our server and get to work. Find your Private SSH key and add it to your ~/.ssh directory. Set the permissions for the key to 600. Now you can access your server with the following:

ssh -i /ssh/directory/private.key ubuntu@PUBLIC-IP

Install Wireguard with the following:

sudo apt update
sudo apt install wireguard

Now create a public and private wireguard key.

wg genkey | sudo tee /etc/wireguard/private.key
sudo chmod go= /etc/wireguard/private.key
sudo cat /etc/wireguard/private.key | wg pubkey | sudo tee /etc/wireguard/public.key

Make note of those keys as we will need them. Now we can start setting up our Wireguard server configurtion.

sudo nano /etc/wireguard/wg0.conf

And use the following:

[Interface]
PrivateKey = {{private key you just generated}}
ListenPort = 51820
#(any port above here)
Address = 10.1.0.1/24
#(an address from a different subnet from your private LAN)
MTU = 1420

PostUp = iptables -A FORWARD -i enp0s3 -j ACCEPT; iptables -t nat -A POSTROUTING -o wg0
PostDown = iptables -D FORWARD -i enp0s3 -j ACCEPT; iptables -t nat -A POSTROUTING -o wg0
#enp0s3 is whatever your virtual machines main interface is called

[Peer]
PublicKey = {{ we do not have this yet }}
AllowedIPs = 10.1.0.0/24, {{address of npm/32}}
PersistentKeepalive = 25

Now let's make sure we have forwarding set up in our Oracle machine. We are going to edit sysctl.conf. First open it.

sudo nano /etc/sysctl.conf

Add the following two lines:

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Then enter the following command or reboot:

sudo sysctl -p

Almost ready to move over to the OPNsense side of things. Let's just set up our firewall before we forget.

We will use ufw which is included in the ubuntu image.

Add ssh port:

sudo ufw allow 22/tcp

Https port:

sudo ufw allow 443/tcp

Wireguard port (change as necessary):

sudo ufw allow 51820/udp

Then enable the firewall:

sudo ufw enable

And check the status to make sure we are secured:

sudo ufw status

and we should see something like:

Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
51820/udp                  ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere

Ok perfect. Let's change our attention to OPNsense.

OPNsense

This part only requires a few things but took me DAYS of neglecting my family and responsibilities to figure it out. I was pointed in the right direction here. Like I said, I already have a Wireguard server set up on my OPNsense box, and was trying to incorporate this new client in that network. That proved to be impossible- at least for me. Then it struck me- I would just make a NEW wireguard network on the OPNsense machine, and this one would act as a client instead of a server as I had initially thought it would.

So if you haven't already, install either the wireguard-go or wireguard-kmod package on your OPNsense box.

Go to the Wireguard menu and then the Endpoints tab. Create a new endpoint and fill it out like so:

Wireguard OPNsense Endpoint

Now switch to the Local tab and create a new entry like so:

OPNsense local configuration

Make sure advanced options are checked. Call it whatever you want, set an MTU (I think?). Enter what this client's IP address will be. Since our VPS is 10.1.0.1, I chose 10.1.0.2 for this client, and make sure to use the /24 subnet. Select your previously made endpoint under peers. MAKE SURE TO CLICK DISABLE ROUTES. Otherwise when you initiate the connection, the routing tables will be edited to try and send all traffic through the Wireguard network. Then make sure to enter the VPS Wireguard IP address for the Gateway.

There is no private or public key but after you click save and reopen the entry, you should see that populate. Take the Public Key, go back to our wg0.conf from the VPS and add the public key as indicated earlier under the "[Peer]" section.

Now, on our VPS, you can bring up the wg0 interface by entering the following:

sudo wg-quick up wg0

You can also set it to start at boot and enable with:

sudo systemctl enable --now wg-quick@wg0

Back to OPNsense...

Turn Wireguard on and off from OPNsense. Now lets give it an interface so we can more easily manipulate the connection and create firewall rules. Go to Interface --> Assignments. There should be an option at the bottom to add a new interface and one of the interfaces from the dropdown menu should be the new Wireguard connection (wg0) and call it whatever you want.

After refreshing the page, the new VPS/Wireguard interface should be listed in the Interfaces section.

Click on it, make sure it is enabled, and click Promiscuous mode, save.

Ok almost done. Now we'll just go down the line and add the settings necessary for our VPS to act as the external gateway we want.

New gateway

Change the settings to match what I have above. The IP address should be the IP address of the Wireguard interface on the VPS.

Next, the connection we receive on port 443, will be forwarded to the OPNsense router on that same interface. We will go to the Port Forwarding tab and create a rule to forward that connection to wherever our NPM instance is. However, aside from the usual port forwarding rule, we are going to make two changes.

Add something to "Set local tag". I just put "oracle". It doesn't matter what you pick.
For "Filter Rule Association", change it to "none".

Now we will move on to the firewall rules for our new VPS/Wireguard interface. Currently, without any rules, all traffic will be blocked. I only have two rules:

Allow 443 traffic to the NPM instance.
Allow communication between clients on our interface subnet
Geo-block IPs if desired

VPS/Wireguard interface firewall rules

I also included the same "set local tag" directive with "oracle" to both of those above Pass rules. Not sure if that is necessary, but I figured it wouldn't hurt. Basically, I want anything that comes in through this interface to leave through it.

Finally, we will tell the router how to send the packets back out through the VPS as opposed to leaving through our regular gateway.

Go to the interface where your NPM instance is. Make a new firewall rule. This time, however, instead of choosing "in" for the direction, choose "out". Then we can leave everything alone and just make a few changes under the "Advanced" section.

Firewall "out" rule

There are three things to do here.

select allow options
change the reply-to gateway to the gateway we just created
match local tag to whatever you used in the previous rules. Mine is "oracle".

You can be more specific if you want, but the traffic we are interested in is already tagged so I think the firewall rule is likely fine how it is. Now the firewall will know to reply back to our VPS and not through Verizon or Comcast or whatever.

At this point, I would probably restart the VPS and then take the Wireguard service down and then up to make sure that all our new rules are working and activated.

Conclusion

If everything worked the way it should, you should now be able to set the DNS records for your web services to point to the public IP address of your VPS. Supposedly, this will remain free forever but of course that remains to be seen. Not sure if this really makes things safer or more secure than just opening port 443 of your firewall really, but I feel like it can't hurt? If nothing else, I have noticed a significant improvement with the initial web connection when accessing my home services. I suppose that these residential connections probably aren't designed to accommodate incoming web requests but this also may be placebo.

If you run into trouble with any of this, the linked OPNsense and Oracle Cloud docs are the best next stop.

Making an Automatic Email Backup - Part 3

thehelpfulidiot — Wed, 22 Jun 2022 00:00:00 +0000

Part 1

Part 2

Initially, I described a solution which required the use of LXC or some full Linux box to install and operate the necessary tools. Recently, however, I repackaged those tools into a couple docker containers, more easily deployable through a docker compose file with configuration done with environment variables.

https://github.com/jon6fingrs/mbsync-dovecot

https://hub.docker.com/repository/docker/thehelpfulidiot/dovecot

https://hub.docker.com/repository/docker/thehelpfulidiot/mbsync

Instructions on the github page. Let me know if you have any questions!

Integrating XMCOSY+ Outdoor String Lights with Home Assistant Locally

thehelpfulidiot — Fri, 17 Jun 2022 00:00:00 +0000

XMCOSY+ Outdoor String Lights

Home automation is incredible and obviously in my opinion, a lot of fun. However, I'm sure many of you would agree that buying a product for use in your home shouldn't have to connect to an outside computer, let alone one in another country.

Fortunately with the magic of Home Assistant and the Local Tuya plugin, I have managed to run this light locally without any loss of functionality. As you may know, Tuya is a Chinese manufacturer who creates a lot of white label IoT devices. The devices, in my experience, tend to work well, but require cloud access to work through official channels. Local Tuya, on the other hand, is a custom integration which can grant local access to these Tuya devices and can work very well for our purposes.

Unfortunately, Local Tuya, while set up to handle many different types of devices out of the box, is not able to handle the unique modes and functions of the XMCOSY+ Outdoor String Lights. However, by at least allowing access to the device API, templates and automations can be used to simplify and restore the device's normal function locally. In this post, I hope to explain my method for solving this problem and that it will work for others.

Steps

Set the device up in Local Tuya as a light
Create a template light entity to control the light seamlessly
Use Node Red to convert between hex color code and HSV
Create entities to change the custom colors related to one of the built-in scenes
(optional) Block light access to internet

Local Tuya

https://github.com/rospogrigio/localtuya

The documentation is pretty good at explaining how to get this custom integration installed, and then how to add your devices. Once you get your string lights added, you will have to instruct the plugin as to which data point (dp) controls which function of the light.

Here is a printout of the data points for the light from the command line tool from which local tuya is based:

{
  dps: {
    '20': false,
    '21': 'white',
    '22': 255,
    '23': 165,
    '24': '23eb230159ff38',
    '101': 'ffff0502808080ffff00'
  }
}

The explanation for each of these is as follows:

20: on/off

21: scene (white, colour, scene1, scene2, ..., scene8*, scene9)

*scene 8 is the only customizable scene allowing you to select two colors

22: brightness 0-255

23: white-temp on a bizarre scale of 0 - 255

24: the light color and when in color mode, its brightness (very complicated will explain below)

101: a string storing the customizable two colors for scene8 (explained below)

Explanation of DP24

Now DP 24. This took a really really long time to figure out and I found some helpful hints on decoding Tuya colors deep in the internet and I'm sorry I can't seem to find these sites again or else I would like to provide links and give them credit.

This is a combination of the lights hex code AND HSV (encoded in hex).

So the first 6 characters (in my example above) - 23eb23 is a simple hex code for the current color.

The 7th character is a 0. That is just a divider.

The next 7 characters are related to HSV, but each item is hex-encoded.

The first three charaters are Hue from 0-360 (ex above 159)

The following 2 characters are saturation on a scale of 0-255 (the standard is 0-100)

And the last 2 characters are Value (brightness) which is also on a scale 0-255 (and in this case the standard is 0-100)

Explanation of DP101

It seems to consist of a constant string in the beginning (example above ffff0502). It may mean something but I was not able to change it through any of the app functions and I was too scared to change it manually. If anyone knows what this might represent, please tell me.

But after that, are two back to back hex color codes. In the example above, color 1 is 808080, and color two is ffff00. I made two input selectors with a bunch of standard common color names. There is an automation and each time one of those two entities changes, the updated hex color code is inserted in the appropriate place.

The color combination is updated and selecting scene8 shows you the custom scene you just made.

Back to Local Tuya

So based on that, we will make a few entities using Local Tuya, all based around this device.

First, of course, is the light itself. You can match up the DPs as follows:

LocalTuya Light Configuration

This sets up a light entity, but like I said, it is very limited. Cannot change between scenes, adjust scene colors, etc.

Next, we need to set up some sensors from LocalTuya so that we can easily access the data for use in a template light.

sensor.string_lights_scene

sensor.string_light_color

sensor.string_light_color_value

sensor.custom_colors

So after all this, you should have five newly create LocalTuya entities:

light.colored_string_lights - the main light entity
sensor.custom_colors - a sensor displaying the hex code for the two custom colors used for scene8
sensor.string_light_color - brightness (don't ask me why I called it this)
sensor.string_light_color_value - the complicated hex and hsv color/brightness string
sensor.string_lights_scene - the current scene

Now it's time to move on to templates.

Templates

OK so now you can control the lights, but wouldn't it be nice to package everything in a single light entity? Fortunately, we can make template entities to combine the sensors and the light entity we created with LocalTuya.

  - platform: template
    lights:
      string_lights:
        level_template: >
          {% if state_attr('light.colored_string_lights','color_mode') == 'color_temp' %}
            {{ states('sensor.string_light_color')|int }}
          {% elif state_attr('light.colored_string_lights','color_mode') == 'hs' %}
            {{ states('sensor.string_light_color_value')[12:]|int(base=16) }}
          {% else %}
            255
          {% endif %}
        value_template: "{{ states('light.colored_string_lights') }}"
        temperature_template: >
          {% if state_attr('light.colored_string_lights','color_mode') == 'color_temp' %}
            {{ state_attr('light.colored_string_lights','color_temp') }}
          {% elif state_attr('light.colored_string_lights','color_mode') == 'hs' %}
            {{ color_temp|int(default=0) }}
          {% endif %}
        color_template: >
          {% if state_attr('light.colored_string_lights','color_mode') == 'hs' %}
            {{ state_attr('light.colored_string_lights','hs_color') }}
          {% endif %}
        effect_list_template: "{{['White','Color','Strobe','Fade','Pulse','Blink','Chase','Christmas','America','Pick 2','Multi']}}"
        effect_template: >
          {% if states('sensor.string_lights_scene') == 'scene1' %}
            {{ "Strobe" }}
          {% elif states('sensor.string_lights_scene') == 'scene2' %}
            {{ "Fade" }}
          {% elif states('sensor.string_lights_scene') == 'scene3' %}
            {{ "Pulse" }}
          {% elif states('sensor.string_lights_scene') == 'scene4' %}
            {{ "Blink" }}
          {% elif states('sensor.string_lights_scene') == 'scene5' %}
            {{ "Chase" }}
          {% elif states('sensor.string_lights_scene') == 'scene6' %}
            {{ "Christmas" }}
          {% elif states('sensor.string_lights_scene') == 'scene7' %}
            {{ "America" }}
          {% elif states('sensor.string_lights_scene') == 'scene8' %}
            {{ "Pick 2" }}
          {% elif states('sensor.string_lights_scene') == 'scene9' %}
            {{ "Multi" }}
          {% elif states('sensor.string_lights_scene') == 'white' %}
            {{ "White" }}
          {% elif states('sensor.string_lights_scene') == 'colour' %}
            {{ "Color" }}
          {% endif %}
        min_mireds_template: "{{ state_attr('light.colored_string_lights','min_mireds') }}"
        max_mireds_template: "{{ state_attr('light.colored_string_lights','max_mireds') }}"
        turn_on:
          service: light.turn_on
          target:
            entity_id: light.colored_string_lights
        turn_off:
          service: light.turn_off
          target:
            entity_id: light.colored_string_lights
        set_level:
          - service: script.brightness_with_white_or_color
            data:
              brightness: '{{ brightness|int }}'
        set_color:
          - service: script.check_color_mode
            data:
              h: '{{ hs[0] }}'
              s: '{{ hs[1] }}'
        set_temperature:
          - service: script.check_white_mode
            data:
              color_temp: '{{color_temp|int }}'
        set_effect:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 21
              value: >
                {% if effect == 'Strobe' %}
                  {{ "scene1" }}
                {% elif effect == 'Fade' %}
                  {{ "scene2" }}
                {% elif effect == 'Pulse' %}
                  {{ "scene3" }}
                {% elif effect == 'Blink' %}
                  {{ "scene4" }}
                {% elif effect == 'Chase' %}
                  {{ "scene5" }}
                {% elif effect == 'Christmas' %}
                  {{ "scene6" }}
                {% elif effect == 'America' %}
                  {{ "scene7" }}
                {% elif effect == 'Pick 2' %}
                  {{ "scene8" }}
                {% elif effect == 'Multi' %}
                  {{ "scene9" }}
                {% elif effect == 'White' %}
                  {{ "white" }}
                {% elif effect == 'Color' %}
                  {{ "colour" }}
                {% endif %}

alias: check white mode
sequence:
  - if:
      - condition: not
        conditions:
          - condition: state
            entity_id: sensor.string_lights_scene
            state: white
    then:
      - service: localtuya.set_dp
        data:
          device_id: eb3f15638dccc2fe59wdgj
          dp: 21
          value: white
  - service: light.turn_on
    data:
      color_temp: '{{ color_temp }}'
    entity_id: light.colored_string_lights
mode: single

alias: check color mode
sequence:
  - if:
      - condition: not
        conditions:
          - condition: state
            entity_id: sensor.string_lights_scene
            state: colour
    then:
      - service: localtuya.set_dp
        data:
          device_id: eb3f15638dccc2fe59wdgj
          dp: 21
          value: colour
      - wait_for_trigger:
          - platform: state
            entity_id:
              - sensor.string_lights_scene
            to: colour
        timeout: '1'
  - service: light.turn_on
    data:
      hs_color:
        - '{{ h }}'
        - '{{ s }}'
    entity_id: light.colored_string_lights

One issue I noticed is that the brightness value is only relevant when the light is in white mode. When the light is in a color mode, the brightness is part of the hex/hsv string. This template selects the correct brightness based on the mode. When the light is showing a scene other than color or white, brightness is always at 100%. I believe that is a limitation of the light.

Anyway, to make sure the correct brightness value is modified when you modify the brightness from the light entity in home assistant, I offloaded that to a script seen below:

alias: brightness with white or color
sequence:
  - choose:
      - conditions:
          - condition: template
            value_template: >-
              {{ state_attr('light.colored_string_lights','color_mode') ==
              'color_temp' }}
        sequence:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 22
              value: '{{brightness|int}}'
    default:
      - service: rest_command.color_converter
        data:
          brightness: '{{brightness|int}}'
      - service: localtuya.set_dp
        data:
          device_id: eb3f15638dccc2fe59wdgj
          dp: 24
          value: >
            {{ states('sensor.color_output') }}{{
            states('sensor.string_light_color_value')[6:12] }}{{ "%0x" | format(
            brightness | int ) }}
mode: single

Another issue was converting between HSV and hex colors. I could not find an easy formula or function to accomplish this in Home Assistant. After much searching, I found a module to use in Node Red.

Color and Brightness

This is important for the DP24 which controls color and brightness when in single color mode. Basically, a hex value and hex encoded HSV is given each time. Just changing the hex value would change the color but reset the lights back to 100%. Apparently you need to change both and they need to match? Converting with template got complicated and I could not figure out a way to solve it. After much searching, I came across a module in Node Red.

I am not a Node Red power user. I do very few things with it, and it feels like whenever I use it, I have to relearn it.

So to start, I needed a way to get information from home assistant to Node Red. I used a rest command in Home Assistant to send information to an HTTP receiver in Node Red:

  change_color_not_brightness:
    url: http://nodered:1880/color-converter
    method: POST
    payload: '{"hsv":{"hue":{{ h }},"saturation":{{ s }},"brightness":{{ (state_attr(''light.string_lights'',''brightness'')/2.55)|int}}}}'
    content_type:  'application/json; charset=utf-8'
    verify_ssl: false

This sends over the current color and brightness in HSV.

Node Red

Now to switch gears to Node Red. For those who don't know, it is a very commonly used add-on for Home Assistant, for creating automations by visually programming in Java. Many people use it exclusively for automations and don't touch the built-in YAML-based automations. Aside from the ease of creating automations, it also has access to a number of add-on modules which can add additional functions to your palette.

The one we will need is called node-red-contrib-colorspace. It allowed me to create a "Flow" which received the HSV value through HTTP request, convert it to rgb, and send it back to Home Assistant. I send it back by using the Home Assistant nodes to update a sensor called "sensor.color_output".

sensor.color_output

My flow looks like this:

Node-red Flow

Final Product

Ok, so now we have a fully functional entity called "light.string_lights". It can control (almost) every light function as one would expect from a regular Home Assistant light entity.

white light

Color Select

scene select

It can control brightness, color, and the scenes. It can change brightness for color or white seamlessly. But there's one more thing- scene8.

scene8

Scene 8 is one of 9 scenes, and is the only customizable one. It alternates two colors which can be changed independently. I made two select entities for them. I just chose a handful of colors and created a dropdown menu although it would be possible to choose any color as well.

Color Select

Whenever these entities are changed, they trigger an automation to change the color of the light:

alias: Change string light colors
description: ''
trigger:
  - platform: state
    entity_id: input_select.color_1
    id: color1
  - platform: state
    entity_id: input_select.color_2
    id: color2
condition: []
action:
  - choose:
      - conditions:
          - condition: trigger
            id: color1
        sequence:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 101
              value: >
                {% if states('input_select.color_1') == 'Black' %}
                  {% set new1 = '000000' %}
                {% elif states('input_select.color_1') == 'Silver' %}
                  {% set new1 = 'c0c0c0' %}
                {% elif states('input_select.color_1') == 'Gray' %}
                  {% set new1 = '808080' %}
                {% elif states('input_select.color_1') == 'White' %}
                  {% set new1 = 'ffffff' %}
                {% elif states('input_select.color_1') == 'Maroon' %}
                  {% set new1 = '800000' %}
                {% elif states('input_select.color_1') == 'Red' %}
                  {% set new1 = 'ff0000' %}
                {% elif states('input_select.color_1') == 'Purple' %}
                  {% set new1 = '800080' %}
                {% elif states('input_select.color_1') == 'Fuchsia' %}
                  {% set new1 = 'ff00ff' %}
                {% elif states('input_select.color_1') == 'Green' %}
                  {% set new1 = '008000' %}
                {% elif states('input_select.color_1') == 'Lime' %}
                  {% set new1 = '00ff00' %}
                {% elif states('input_select.color_1') == 'Olive' %}
                  {% set new1 = '808000' %}
                {% elif states('input_select.color_1') == 'Yellow' %}
                  {% set new1 = 'ffff00' %}
                {% elif states('input_select.color_1') == 'Navy' %}
                  {% set new1 = '000080' %}
                {% elif states('input_select.color_1') == 'Blue' %}
                  {% set new1 = '0000ff' %}
                {% elif states('input_select.color_1') == 'Teal' %}
                  {% set new1 = '008080' %}
                {% elif states('input_select.color_1') == 'Aqua' %}
                  {% set new1 = '00ffff' %}
                {% endif %}

                {{ 'ffff0502' }}{{new1}}{{
                states('sensor.custom_colors').split('ffff0502')[1][6:] }}
      - conditions:
          - condition: trigger
            id: color2
        sequence:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 101
              value: >
                {% if states('input_select.color_2') == 'Black' %}
                  {% set new2 = '000000' %}
                {% elif states('input_select.color_2') == 'Silver' %}
                  {% set new2 = 'c0c0c0' %}
                {% elif states('input_select.color_2') == 'Gray' %}
                  {% set new2 = '808080' %}
                {% elif states('input_select.color_2') == 'White' %}
                  {% set new2 = 'ffffff' %}
                {% elif states('input_select.color_2') == 'Maroon' %}
                  {% set new2 = '800000' %}
                {% elif states('input_select.color_2') == 'Red' %}
                  {% set new2 = 'ff0000' %}
                {% elif states('input_select.color_2') == 'Purple' %}
                  {% set new2 = '800080' %}
                {% elif states('input_select.color_2') == 'Fuchsia' %}
                  {% set new2 = 'ff00ff' %}
                {% elif states('input_select.color_2') == 'Green' %}
                  {% set new2 = '008000' %}
                {% elif states('input_select.color_2') == 'Lime' %}
                  {% set new2 = '00ff00' %}
                {% elif states('input_select.color_2') == 'Olive' %}
                  {% set new2 = '808000' %}
                {% elif states('input_select.color_2') == 'Yellow' %}
                  {% set new2 = 'ffff00' %}
                {% elif states('input_select.color_2') == 'Navy' %}
                  {% set new2 = '000080' %}
                {% elif states('input_select.color_2') == 'Blue' %}
                  {% set new2 = '0000ff' %}
                {% elif states('input_select.color_2') == 'Teal' %}
                  {% set new2 = '008080' %}
                {% elif states('input_select.color_2') == 'Aqua' %}
                  {% set new2 = '00ffff' %}
                {% endif %}

                {{ 'ffff0502' }}{{
                states('sensor.custom_colors').split('ffff0502')[1][:6] }}{{
                new2 }}
    default: []
mode: single

To confirm the light color was changed to the desired color, I also made two sensors to read the scene's custom colors from the light itself.

- sensor:
      - name: "Custom Color 2"
        state: >
          {% if states('sensor.custom_colors').split('ffff0502')[1][6:] == '000000' %}
            {{ 'Black' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'c0c0c0' %}
            {{ 'Silver' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '808080' %}
            {{ 'Gray' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ffffff' %}
            {{ 'White' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '800000' %}
            {{ 'Maroon' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ff0000' %}
            {{ 'Red' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '800080' %}
            {{ 'Purple' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ff00ff' %}
            {{ 'Fuchsia' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '008000' %}
            {{ 'Green' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '00ff00' %}
            {{ 'Lime' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '808000' %}
            {{ 'Olive' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ffff00' %}
            {{ 'Yellow' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '000080' %}
            {{ 'Navy' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '0000ff' %}
            {{ 'Blue' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '008080' %}
            {{ 'Teal' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '00ffff' %}
            {{ 'Aqua' }}
          {% else %}
            {{ states('sensor.custom_colors').split('ffff0502')[1][6:] }}
          {% endif %}
- sensor:
      - name: "Custom Color 1"
        state: >
          {% if states('sensor.custom_colors').split('ffff0502')[1][:6] == '000000' %}
            {{ 'Black' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'c0c0c0' %}
            {{ 'Silver' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '808080' %}
            {{ 'Gray' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ffffff' %}
            {{ 'White' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '800000' %}
            {{ 'Maroon' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ff0000' %}
            {{ 'Red' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '800080' %}
            {{ 'Purple' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ff00ff' %}
            {{ 'Fuchsia' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '008000' %}
            {{ 'Green' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '00ff00' %}
            {{ 'Lime' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '808000' %}
            {{ 'Olive' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ffff00' %}
            {{ 'Yellow' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '000080' %}
            {{ 'Navy' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '0000ff' %}
            {{ 'Blue' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '008080' %}
            {{ 'Teal' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '00ffff' %}
            {{ 'Aqua' }}
          {% else %}
            {{ states('sensor.custom_colors').split('ffff0502')[1][:6] }}
          {% endif %}

That produces these two entities:

Color Select Entities

Light Picker

Finally

Our light is finished and fully controllable locally. The next step is to block the light's internet access from your router.

Completed Light

Hope this helps some people! Let me know if you have any questions!

Link to Purchase

Affiliate Link:

XMCOSY+ Outdoor String Lights, 49Ft

Non-Affiliate Link:

XMCOSY+ Outdoor String Lights, 49Ft

Custom 3D Printed Mount for Fire Tablet (2015)

thehelpfulidiot — Fri, 26 Nov 2021 00:00:00 +0000

I have been enamored by the slew of Home Assistant Dashboard mounts lately and have wanted to make one for a while. I saw that the newer Amazon tablets have commercially available options, but my old 2015 Fire Tablet was on its own.

I decided to make one using my Creality Ender 3 Pro. I designed one with screw holes to fit a standard 2-gang box, an opening for the cord to exit posteriorly, and cut outs for the tablet's buttons.

Printing

Unfortunately, each one took about a day to print and I went through several drafts, slowly tweaking the design to fit perfectly. This slow process probably took weeks, but I finally came up with something that seemed to work.

Finished frame

Cut outs for buttons and cord

Tablet in the frame- tight fit

Attaching a tablet to a wall is not hard, but doing it in a way to hide the cords is definitely more challenging. I know that the standard way is to run power to a new gang box which is why I designed my tablet frame to fit that scenario. However, I realized that I could probably do something a little simpler.

I just ran the USB cable behind the wall and made a seperate opening towards the floor so that at least, the cable would be hidden and not dangling from the tablet.

Future tablet location

Attached frame

Cord exit

And here is the final result:

Final product

In case anyone is interested, here is a copy of my STL file:

fire-tablet-mount

And finally, here is the cord I used and the brush wall plate:

Affiliate Link:

GELRHONR Micro USB Cable U Shape,180 Degree Angle USB 5 Pin Nylon Braided Fast Data Sync Charging Cord USB A to USB B Fast Charger Wire for Most Micro USB Devices - Black 4.9Ft/1.5M

Single Brush Wall Plate 2-Pack,Single Gang faceplate Cable Pass Through Insert Wall Socket for HDTV, HDMI, Network, Coaxial Cables, Speaker Wires White

Non-affiliate Link:

GELRHONR Micro USB Cable U Shape,180 Degree Angle USB 5 Pin Nylon Braided Fast Data Sync Charging Cord USB A to USB B Fast Charger Wire for Most Micro USB Devices - Black 4.9Ft/1.5M

Single Brush Wall Plate 2-Pack,Single Gang faceplate Cable Pass Through Insert Wall Socket for HDTV, HDMI, Network, Coaxial Cables, Speaker Wires White

Let me know if you have any questions. Thanks!

UPDATE 2: A Wired Sonoff Zigbee Alternative

thehelpfulidiot — Mon, 18 Oct 2021 00:00:00 +0000

ORIGINAL POST

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

The updates for this little device keep coming. Thanks to the folks leaving comments, we keep getting more information. I apologize for not noticing this myself, but I did not spend any time trying Zigbee2Tasmota on this device having played with it enough on the Sonoff hub.

A commenter, Ronald, noted that using Zigbee2Tasmota, attempting to pair Zigbee devices caused the hub to reboot. I confirmed this myself, and tried with both Zigbee firmware packages from the EmberZNet NCP UART EZSP firmware page.

I attempted to build a firmware using the development branch of Tasmota instead of the Release branch, but it totally bricked my coordinator. After re-soldering and flashing again, I tried the unofficial Tasmota32-zigbeebridge.bin mentioned on the Blakadder Template Repo. At the time of this writing, the Tasmota version is 9.5.0.9. Miraculously, I was able to pair devices as expected.

Device allowed to join

After flashing this firmware, you would set it up following the instructions from the initial post by changing the template and then disabling WiFi if desired.

To use ZHA, I would still recommend you follow the instructions from the first update and building a custom firmware so that the necessary rules to self-correct following a cold boot will work.

Hopefully, whatever fixes exist in the nightly build will make their way to the stable branch and a single firmware will be able to accomplish ZHA and Zigbee2Tasmota.

Please keep me updated with any issues or questions. Thanks!

ORIGINAL POST

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

**UPDATE 10/30: **The above issue was acknowledged on Github and apparently had something to do with the serial communication in the Tasmota firmware. As noted above, that is why the unofficial zigbeebridge firmware works. It containts a fix for the serial communication.

UPDATE: A Wired Sonoff Zigbee Alternative

thehelpfulidiot — Fri, 15 Oct 2021 00:00:00 +0000

ORIGINAL POST

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

So while the documentation in the initial post on how to compile and flash Tasmota to the ZB-GW03 works, there is a bug when using the Tasmotized device with Home Assistant.

After changing the device to a ZHA Bridge, a power disruption will prevent a reconnection to Home Assistant once power is restored. The only way to reconnect appears to be by switching the device template back to Zigbee2Tasmota, letting it start, and then back to the ZHA Bridge template. Afterwards it will reconnect as usual. A software initiated reboot DOES NOT cause the same issue.

I do not know enough to say exactly why this is the case, but my theory is that the Zigbee2Tasmota template initializes the Zigbee Radio when it boots up and maybe that is not happening with Home Assistant when the device is shut down in an unsafe state.

Fortunately, I have developed a work around which appears to work well and essentially fixes the problem without further intervention.

It requires two rules:

Turning on after an unsafe shutdown changes the device back to Zigbee2Tasmota
Once the device boots into Zigbee2Tasmota and the Zigbee radio is fully initialized, the device switches back to ZHA bridge.

In my somewhat brief testing, this has worked really well and the device reconnects as expected after an unsafe shutdown without any user intervention.

Unfortunately, to accomplish this, requires a bit more work than one would think. Tasmota only saves one user defined Module at a time, and I was not able to save a rule with template information, probably due to the fact that it is in JSON format. Therefore, I had to compile a new Tasmota firmware, with the two file modifications on the initial instruction page, plus two new modules, one for each template.

tasmota_template

To compile it yourself, you just need to replace tasmota_template.h with the following. Towards the end are the modules for ESP32 devices and you can see two additional modules for the ZB-GW03, one for ZHA and one for Zigbee2Tasmota.

Module 2 - ZHA_Bridge
Module 3 - Zigbee2Tasmota

If you prefer, you can just download the updated firmware below:

tasmota32-EWELINK-updated

Once you have the new firmware installed, you can use the following rules:

backlog rule1 on system#boot do backlog wifi 0; TCPStart 8888 endon; rule1 on
backlog rule2 on Info3#RestartReason=Vbat power on reset do module 3 endon; rule2 on
backlog rule3 on ZbState#Message=Started do module 2 endon; rule3 on

The first rule combines rules 1 and 2 from the initial post and sets up the Zigbee Web Server and turns off wifi. If you intend to use wifi (which I probably wouldn't recommend), just use the following instead for the first rule:

backlog rule1 on system#boot do TCPStart 8888 endon; rule1 on

Rule2 changes the module to Zigbee2Tasmota ONLY when the power is cut and the device undergoes an unsafe shutdown.

Rule3 as soon as the device boots to Zigbee2Tasmota and the Zigbee radio is fully initialized, it switched back to ZHA Radio at which point the device should be able to reconnect to Home Assistant as expected.

An issue has been opened on Github so please reply if you have experienced the same problem:

https://github.com/arendst/Tasmota/issues/13350

Please let me know about any additional issues or if you have any problems with the above solution.

ORIGINAL POST

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

Making an Automatic Email Backup - Part 2

thehelpfulidiot — Sat, 09 Oct 2021 00:00:00 +0000

UPDATE: PART 3 has instructions for email backup using pre-made docker images based on this post and Part 1.

<--- Part 1

Summary so far

Ok, so if you have gone through Part 1 already, you have Mbsync set up to sync all emails from your remote IMAP server and then Dovecot set up to host those emails on a local SMTP server, but without SSL.

That may be enough for many, but we can take things a step further. In this part, we will add self-signed SSL certificates for Dovecot, and we will install Roundcube. Roundcube is an open sourced webmail client. We will configure it to access our local email server and then to send emails through our remote server.

Now one thing I would highly highly recommend to get the most out of this setup is to install Wireguard on your network and to set up a split-tunnel on your devices so that wherever you are, you always have access to all your local services. There are likely many of these tutorials on the internet and the instructions may vary a bit with the exact Wireguard instance you decide to use. Once you have it set up though, you will have access to all home hosted services - Pihole is always blocking ads for me on my cell phone, even on mobile, I can access router settings, Proxmox, whatever.

In this case, I can also access my "local" IMAP server from anywhere in the world without exposing the IMAP server to the wide scary internet. Of course, if you wanted, I'm sure you could also expose whatever services you want using a reverse proxy, but to me that just seems like a little less secure.

Installing SSL

So in my home, I have created a root Certificate Authority and have many server certificates for the various services on my network, mostly used with Nginx Reverse Proxy. There are several ways to do this, but if you have Pfsense or OPNsense, it comes with a way to create and manage these certificates using a graphical format. Below is a link to the OPNsense website detailing the creation of these certificates and a great video by Allen Sampsell.

OPNsense documentation

Youtube video by Allen Sampsell

If you don't have OPNsense or PFsense, this is the website I previously followed. Everything is done by the command line so a little more work, but with good instructions:

Self-signed SSL certificate command line instructions

Now, pick a local DNS name for your IMAP server. For example, imap.thehelpfulidiot.lan. Use whatever router software you have to create a local DNS entry pointing to your email backup server. Once you have the server name, you can create the Server Certificate for the server.

By now you should have three things:

Certificate Authority Certificate
IMAP Server Certificate
IMAP Server Key

SSH into your email backup server.

Copy over your server certificate and keys. Go to the Dovecot directory and make a directory to store SSL certificates.

cd /etc/dovecot
sudo mkdir ssl
cd ssl

sudo nano email.pem

If your certificate has a "*.crt" extension, you will need to convert it to "*.pem" with the following command.

openssl x509 -in email.crt -out email.pem -outform PEM

In the new file that opens, copy and paste the contents of your server certificate file.

Ctrl+X, Y, enter.

Next, do the same with your key file.

sudo nano email.key

Paste the contents of your key file.

Ctrl+X, Y, enter.

Now your certificate and key files are at the following locations, respectively:

/etc/dovecot/ssl/email.pem
/etc/dovecot/ssl/email.key

Ok, so now we have to tell Dovecot where the files are and what its server name is.

cd /etc/dovecot/conf.d

sudo nano 10-auth.conf

Remove the line "disable_plaintext_auth = no" we placed in part 1 so that the default to not accept plaintext authentication will be reinstated.

Now let's edit the SSL file.

sudo nano 10-ssl.conf

Change "ssl = yes" to "ssl = required" towards the top.

Then you should see "ssl_cert" and "ssl_key" entries near the top as well. Change those to the appropriate file locations, but do not remove the "<" before each file path as this tells Dovecot to read the contents of the files and not just the file names.

ssl_cert = </etc/dovecot/ssl/email.pem
ssl_key = </etc/dovecot/ssl/email.key

Finally, we have to tell Dovecot what its sever name is so it knows to trust connections to it:

sudo nano /usr/share/dovecot/dovecot-openssl.cnf

Change the following line from:

commonName = @commonName@

to:

commonName = imap.thehelpfulidiot.lan

Now, assuming you have installed your root certificate on your devices, connecting to this IMAP server should be like connecting to any server.

Installing Roundcube

Next, wouldn't it be cool to be able to access this email from a nice Web GUI? Roundcube is a great way to do that and is something else which can be hosted internally. Even better, it comes as a Docker image.

I use Portainer to easily spin up docker compose files and recommend it to anyone for simple Docker management.

Create a new stack called "Roundcube" and use the following docker compose file, edited as necessary for your setup:

version: '2'

services:
  roundcubemail:
    image: roundcube/roundcubemail:latest
    container_name: roundcubemail
    restart: unless-stopped
    volumes:
      - /path/to/roundcube/www:/var/www/html
      - /path/to/roundcube/ssl:/var/ssl
    ports:
      - 9002:80
    depends_on:
      - "roundcubedb"
    environment:
      - ROUNDCUBEMAIL_DB_TYPE=pgsql
      - ROUNDCUBEMAIL_DB_HOST=roundcubedb # same as pgsql container name
      - ROUNDCUBEMAIL_DB_NAME=roundcube # same as pgsql POSTGRES_DB env name
      - ROUNDCUBEMAIL_DB_USER=roundcube # same as pgsql POSTGRES_USER env name
      - ROUNDCUBEMAIL_DB_PASSWORD=roundcube # same as pgsql POSTGRES_PASSWORD env name
      - ROUNDCUBEMAIL_DB_PORT=5432
      - ROUNDCUBEMAIL_SKIN=elastic
      - ROUNDCUBEMAIL_DEFAULT_HOST=ssl://<localserver>
      - ROUNDCUBEMAIL_DEFAULT_PORT=993
      - ROUNDCUBEMAIL_SMTP_SERVER=tls://<remoteserver>
      - ROUNDCUBEMAIL_SMTP_PORT=587
     
  roundcubedb:
    image: postgres:latest
    container_name: roundcubedb
    restart: unless-stopped
    volumes:
      - /path/to/db/postgres:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=roundcube
      - POSTGRES_USER=roundcube
      - POSTGRES_PASSWORD=roundcube

Save your root Certificate Authority to the SSL folder you have mapped for Roundcube.

Spin that up, but don't worry about logging in just yet. Go to the folder you used for '/path/to/www'.

There should now be a bunch of files and folders generated by Roundcube's first run.

cd config

Edit config.inc.php

sudo nano config.inc.php

you should see the following:

<?php
    $config['plugins'] = [];
    $config['log_driver'] = 'stdout';
    $config['zipdownload_selection'] = true;
    $config['des_key'] = 'some_key';
    include(__DIR__ . '/config.docker.inc.php');

Just below that, add the following:

    $config['enable_caching'] = false;
    $config['imap_conn_options'] = array(
      'ssl'         => array(
         'verify_peer'  => true,
         'verify_depth' => 3,
         'cafile'       => '/var/ssl/rootCA.pem',
         'allow_self_signed' => true,
         'verify_peer_name' => true,
       ),
     );
     $config['smtp_user'] = 'remote_user';
     $config['smtp_pass'] = 'remote_pass';

Ctrl+x, y, enter.

Almost done.

Now log into Roundcube using the credentials for your Linux username and password from your email backup container.

Roundcube Login Screen

Go to Settings --> Identities --> and select the User which is probably something like "thehelpfulidiot@imap.thehelpfulidiot.lan". If we send an email from roundcube, we don't want that to be the identity the recipients see.

Change the Display Name and Email to your actual name and email addresses.

Closing

That's it! Now you can log in to Roundcube as if you were using a commercial email service. You can view your emails neatly organized and send emails directly from your remote email address!

If you were so inclined, you could set up a reverse proxy and put Roundcube behind it and maybe something like Authelia and you could access your email inbox from anywhere. Personally, I'll leave something as sensitive as email safe behind my Wireguard server though.

Please let me know if you have any questions!

Making an Automatic Email Backup - Part 1

thehelpfulidiot — Mon, 04 Oct 2021 00:00:00 +0000

UPDATE: PART 3 has instructions for email backup using pre-made docker images based on this post and Part 2.

Intro

In the world of self-hosting, creating a personal email server always seemed like the Holy Grail of personal privacy. Unfortunately, with all the work that would go into maintaining it, and the high probability of failure when operated by someone as inexperienced as myself, it has never seemed like a viable option for me. However, I have wanted a system for some time whereby I can automatically backup my email in a standard format which could potentially be served to me through standard mail apps by hosting my own internal IMAP server.

After doing a fair amount of research, unfortunately, I could not find any suitable ready-made Docker solutions to accomplish this. However, I was able to create something fairly simple and elegant.

I am going to share my solution which, while it took me a lot of work, should be fairly easy to set up.

It requires three parts:

mbsync - a command line utility to synchronize IMAP servers
dovecot - IMAP server software
an email client

A major benefit of this set up could be for people with limited space in their email mailboxes. Instead of deleting emails to remain under 15gigs, you could essentially backup all emails to your home server, delete emails on the remote server, and still have easy, ready access to everything.

Initial Set Up

I have done this on Proxmox by creating a Ubuntu 20.04 LXC. I am using iSCSI for storage so that while the LXC is being run on my Proxmox machine, the data is being saved on my NAS where it has RAID protection, snapshots, and backups.

I will go through the process assuming Ubuntu LXC, but I imagine this should work with any Ubuntu installation.

I made my LXC with 2 cores and 1GB of RAM. Something I think that's important, is to make a non-root user with sudo privileges. This username and password will be the username and password for your IMAP server so you may want to make it relatively secure. To do so, enter the following commands:

sudo adduser <username>
sudo usermod -aG sudo <username>

Then you can sign out of the root username and log in to the LXC using your new credentials.

Afterwards, update the LXC to the latest with

sudo apt update && sudo apt upgrade -y

Mbsync

Mbsync is a very powerful IMAP syncing software. It downloads your IMAP emails into a Maildir format in which each email is its own file and the file structure of your mailbox disk directory mimics the mailbox itself. It is a command line tool, but was fortunately fairly easy to work with. It can do a lot and I'm sure I barely scratched the surface, but for my purposes, I set it up to download all new emails from the server, but to not propagate any deletions, and to not propagate any changes from my local instances back. This is a one way trip, but allowing deletions of this archive would defeat a major purpose of creating it.

Confusingly, to install mbsync, you have to use its legacy name, isync.

sudo apt install isync

Then it's important to make the configuration file or else the program will not run.

sudo nano ~/.mbsyncrc

and populate accordingly. Below is the template I use with a short explanation. There are many other configuration options if you are interested in further customizing it.

# Set up according to your email provider. It will probably involve creating an application password.
IMAPAccount <email account>
Host <host name>
Port <port>
User <username>
Pass <password>
# select one
SSLType < None | STARTTLS | IMAPS >
# select one
SSLVersions < SSLv3 | TLSv1 | TLSv1.1 | TLSv1.2 | TLSv1.3 > 

# I use fastmail and so my settings will reflect that but change as needed.
# I believe IMAPStore is relatively arbitrary and has to do with how settings are stored. <something>-remote appears to be convention.
IMAPStore fastmail-remote
# pick a name
Account fastmail

# This section describes the local storage
# pick a name similar to what you picked for IMAPStore
MaildirStore fastmail-local

#The root of your mailbox. By convention the Maildir directory is in the user's home folder.
Path /home/thehelpfulidiot/Maildir/
#The inbox seems to be a special folder in a mailbox so you need to specify it explicitly and then the other folders will be auto-created.
Inbox /home/thehelpfulidiot/Maildir/Inbox/
# Adds additional subfolders to match IMAP folder names
SubFolders Verbatim

# A channel defines how the sync will occur between the server and local storage.
# pick a name
Channel fastmail
# use what you have for IMAPStore above between two :'s
Master :fastmail-remote:
# use what you have for MaildirStore above between two :'s
Slave :fastmail-local:
# You can control what gets synced. '*' syncs everything.
Patterns *
# Do not remove messages marked for deletion
Expunge None
# Do not remove deleted mailboxes
Remove None
# use the arrival date and time from the IMAP server for the original message
CopyArrivalDate yes
# You can see all the different ways to sync the mailboxes by going to the link above, but this will sync from remote to local without propagating deletions.
Sync PullNew
# automatically create necessary mailboxes locally
Create Slave
#keeps the mailbox sync state files in the local mailbox
SyncState *

Press "ctrl+x", Y to save, enter.

Now make the mail directory you specified a moment ago. If using the same directory as I specified in the example config file, from your home directory, enter:

mkdir Maildir

Then run mbsync for the first time to download the emails.

mbsync -a

This will take a while if your email folder is like mine. Take a break and come back when it's done. If you set up your config file like mine, mbsync will recreate all your mail folders locally.

I got a socket error during the first sync, but luckily, mbsync can pick up from wherever it leaves off, so just run 'mbsync -a' as many times as you need to get the whole mailbox synced initially.

So now that your current mailbox is fully synced with your local archive, we need to set up a mechanism to pull new messages. As in all things, the archwiki page on mbsync has an excellent example of how to do this.

First, create the systemd unit. I think that placing it in the system directory and adding your specific User to the unit file will ensure that the process runs on boot without login, but as your user.

sudo nano /etc/systemd/system/mbsync.service

And paste this text, making sure to change the user:

[Unit]
Description=Mailbox synchronization service

[Service]
Type=oneshot
User=<user>
ExecStart=/usr/bin/mbsync -Va

Then we need to make a timer file so that this Unit is called periodically.

sudo nano /etc/systemd/system/mbsync.timer

And enter the following:

[Unit]
Description=Mailbox synchronization timer

[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=mbsync.service

[Install]
WantedBy=timers.target

Now activate the new files.

sudo systemctl daemon-reload
sudo systemctl enable mbsync.timer
sudo systemctl start mbsync.timer

Great, so now we have successfully backed up our IMAP account into a Maildir directory and we have set up automatic fetching. For some, this might be adequate, but the next thing I wanted to do was to make sure I had an easy way to consume the mail. Ideally, I wanted it to be viewable in any mail app. Next, we have to install Dovecot which is an IMAP server.

Dovecot

Dovecot is another really powerful piece of software used at super high levels, and I don't presume to be an expert with it, but I will show how it can be used in this situation as a local IMAP server.

So first, let's get it installed:

sudo apt install dovecot-imapd

Now we need to make some changes to the configuration.

sudo nano /etc/dovecot/conf.d/10-mail.conf

Locate the following line:

mail_location = mbox:~/mail:INBOX=/var/mail/%u

Comment it out and replace it with:

mail_location=maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX

Save and close the file.

Next, open 10-auth.conf.

Replace

 #disable_plaintext_auth = yes

with

disable_plaintext_auth = no

This will ensure you can connect to your server without SSL, but if you don't want that option. You can skip it and keep plaintext authentication disabled.

Now open 15-mailboxes.conf.

Under "namespace inbox {", add "inbox = yes" like so:

namespace inbox {
  inbox = yes

Go down farther and uncomment the blocks about the virtual "all" and "flagged" mailboxes so that it looks like this:

  # If you have a virtual "All messages" mailbox:
  mailbox virtual/All {
    special_use = \All
  #  comment = All my messages
  }

  # If you have a virtual "Flagged" mailbox:
  mailbox virtual/Flagged {
    special_use = \Flagged
  #  comment = All my flagged messages
  }

Start Dovecot.

sudo systemctl enable dovecot
sudo systemctl start dovecot

Dovecot is ready and your IMAP server should be functioning on your internal network, but without SSL. Since this is on your local network, that may or may not be adequate for you.

Viewing

Now if you use a mail app, you should be able to enter the IP address of your Mail Backup LXC in the server name and your Linux account credentials and your email will populate like any other server. For example, on my iPhone, I entered the following settings:

Home SMTP settings

You can basically make the name, email, and descriptions what you would normally use for your remote email server. It will not be able to authenticate of course and you will then have an opportunity to create separate username, passwords, and server identities for IMAP and SMTP. The IMAP host name is your LXC's IP address, the user name is your Linux username, and the password is your Linux password.

The SMTP configuration would be according to your remote server's SMTP configuration to ensure you can send emails as ususal.

You'll notice that attempting to connect from an iPhone (and I suspect from other systems) gives this warning:

SSL error

I will continue in a second part on how to take things a bit further. I will show you how to install self-signed certificates, and how to set up a local webmail application called Roundcube to view your backed-up mailbox and send emails through your remote server.

Conclusion

I hope that helps some folks. It took me quite a while figuring out these settings, but now that I have, I think its pretty straightforward. Please feel free to let me know if you have any questions or if I got something wrong.

Thanks for reading!

Continue to Part 2 --->

Controlling Home Assistant with an Old Remote

thehelpfulidiot — Sun, 03 Oct 2021 00:00:00 +0000

I, like many other I'm sure, have plenty of old infrared remotes laying around from TVs past. They were just collecting dust, but what if they could be useful again? Well thanks to the Flirc USB, a small USB IR receiver, those old remotes can be used to control anything your Home Assistant Box can control.

The Flirc USB

Banana for scale

The amazing thing about this device is its simplicity. First, you plug it into a PC, Mac, or Linux machine. Then download the software. The software lets you assign buttons to a virtual keyboard by selecting individual keys and firing your remote at it. Those codes are saved on the device itself, so it can be moved to your Home Assistant Operating System box and will be seen as just an ordinary keyboard. Except, instead of interacting by physically pressing keys, the key is "pressed" when it detects the corresponding infrared frequency.

Set Up

So let's walk through how this would be set up.

First, download the software from the website and plug in the USB device.

Flirc Software

There are a number of virtual keyboard or remotes you can select. I think selecting a standard keyboard makes the most sense to maximize the number of programmable keys.

Next, click on a key, and it will ask you to press the corresponding button on the remote. You do not need to remember which buttons are mapped to which keys because they will later be identified in Home Assistant by watching the event log.

Once you have recorded all the keys you want, you unplug the device from your computer and plug it into your Home Assistant box.

Home Assistant

Now the Flirc can be set up in Home Assistant using the Keyboard Remote integration. First, confirm it is recognized by Home Assistant by looking in the Supervisor's Hardware tab.

Flirc recognized in Hardware tab

Once you see it's recognized, add the following to your configuration.yaml and restart:

keyboard_remote:
  - device_name: 'flirc.tv flirc Keyboard'
    type: 'key_down'

Creating Automations:

Now you can make automations capable of anything your Home Assistant can do from the press of a button. To do so, you need to obtain the "key_code" for each programmed button. Go to the Developer Tools and then the Events tab. Listen for events from "keyboard_remote_command_received" and press the individual buttons. For example:

Example of keyboard_remote_command_received response

So now I know that my remote's "Netflix" button is mapped to keycode 50.

A corresponding automation trigger would look like this:

platform: event
event_type: keyboard_remote_command_received
event_data:
  device_name: flirc.tv flirc Keyboard
  key_code: 50

And from there it can do whatever you want- turn off lights, lock doors, turn on music, or even turn on your TV using your old remote!

Logitech Harmony

Now, I happen to be a big fan of the Harmony Remote and its Home Assistant integration. As far as I know, the integration is one way, meaning that you can fire commands through the Harmony but Home Assistant cannot receive commands. If instead of an old remote you wanted to use the Harmony, you could simply set up some dummy device and make sure the the Harmony Hub or an IR mini blaster will have line of sight to the Flirc Receiver. Set up the device in the same way and then you can integrate Home Assistant actions into your Harmony Hub Activity.

BONUS - MagiQuest Wands

Some of you may be familiar with MagiQuest. It is an interactive live role playing game at many tourist attractions. My family and I, specifically, encountered it at Great Wolf Lodge. To participate, you purchase (expensive) customized wands and can run around the hotel or tourist destination waving it at different objects.

MagiQuest Wand

Well the wand is actually just an infrared transmitter which is fired when the wand is flicked. Instead of bringing them home and having life-less souvenirs, you can integrate them into Home Assistant using the Flirc!

Conclusion

I hope this write-up was helpful. For only $19.95 on Amazon, I definitely think it is a good device at a good price with a lot of possibility. Please let me know if you have any questions.

Affiliate Link

FLIRC USB Universal Remote Control Receiver

Non-Affiliate Link

FLIRC USB Universal Remote Control Receiver

How to Use Zigbee2Tasmota with Home Assistant

thehelpfulidiot — Sat, 02 Oct 2021 00:00:00 +0000

To follow up my last post, integrating a newly flashed Tasmota Zigbee Coordinator with Home Assistant has a third option besides ZHA and Zigbee2MQTT; it can be used directly as a Zigbee Coordinator using its own amazing Tasmota Software.

The device itself uses a zigbee implementation called Zigbee2Tasmota, which is a full-fledged zigbee routing platform with the usual MQTT integration. Unfortunately, it is probably overlooked because its use is limited to the Tasmota console, but it may be an attractive option for some, especially for a potentially more reliable connection when relying on the WiFi radio of the Sonoff Zigbee Coordinator.

In this case, the Pros and Cons may have some subjectivity, but I will share why I was interested in using it, and the drawbacks associated with it.

First, as I stated in my first post, when I began my self-hosting journey seriously by purchasing a machine with the intention of running ProxMox, I did what I'm sure everyone did. I put EVERYTHING I possibly could onto this single machine. That included Pihole (my dns server), Untangled (my firewall), Home Assistant, Mosquitto, EVERYTHING. I thought it was so cool to have a single machine running so many discrete tasks so elegantly through ProxMox, but I found that a host reboot was disastrous and caused a very long downtime in my house. Everything went down and the entire boot process could take up to 10 minutes. This had an extremely negative impact on my WAF.

To solve this, I began splitting up my various services and really only have my plex server, my backup pihole, home assistant plus many other non-critical services on my ProxMox box now. I got a number of cheap Zotac mini-pcs from Amazon Warehouse and have begun using them to run baremetal services like OPNsense, zigbee/zwave radios, NVR, MQTT server, etc.

Back to my point, I did not initially enjoy turning my Tasmotized Zigbee Coordinator into a Home Assistant Zombie. When you integrate the Zigbee coordinator into ZHA, you lose all native Zigbee commands of the coordinator.

In an effort to further decentralize my smart home infrastructure, I would prefer for the zigbee coordinator to maintain functioning even if my Home Assistant instance goes down. Having said that, the device now becomes dependent on a functioning MQTT server so that is certainly a trade-off. I personally keep my MQTT server running on my NAS so that it is always up. I also have a second instance of Home Assistant which meant that I could only integrate the coordinator natively with one instance, whereas when the connection is over MQTT, it can connect to an unlimited number of Home Assistant Servers. I also think an MQTT server is more stable than a WiFi serial over TCP connection. I subjectively and anecdotally noticed a quicker reaction time with Zigbee2Tasmota vs ZHA.

Second, based on my testing and internet research, it is really not possible to change the Zigbee channel from 11 when integrating the Zigbee Coordinator in ZHA. I have a Hue Hub and WiFi channels 1, 6, and 11 throughout my home and so for me, the ideal channel was 20. With Zigbee2Tasmota, this is a trivial command.

zbconfig {"Channel":<enter channel>}

Third, I purchased a number of cheap Sonoff and Aqara sensors without realizing the issues I would have when trying to pair them with standard Zigbee coordinators. I had many drop offs requiring a physical reset on the device when on ZHA, but the devices all just seemed to work with Zigbee2Tasmota. Now, this could be related to my second point and while on ZHA, there just may have been too much interference, but changing the channels of my other devices was not a viable option for me.

But it isn't all positive. There are certainly plenty of downsides with Zigbee2Tasmota.

First, all commands are done through the console except Zigbee Join and viewing the Zigbee Map, as you can see below. I suppose that negative depends on the individual, but it is certainly a nice option to have an easy to use GUI nicely integrated into Home Assistant. However, I should note once devices are added, they are listed on the main Tasmota page along with their current state, connection strength, battery level, etc.

It is definitely nice for a quick glance or to troubleshoot.

Second, along with all Zigbee network controls requiring commands through the Tasmota terminal, adding devices into Home Assistant also required manually configured MQTT entities and Template entities. Again, may or may not be a big deal. I will show some examples of my devices below. Honestly, once one device was written, it was trivial to copy and paste an additional device.

Third, while sensors worked quickly and flawlessly, lights were a bit of a different story. You can only send ONE command at a time. So changing the hue/color and brightness requires two separate commands. In Home Assistant, this required me to get a little creative with a template light ON TOP of the manually configured MQTT light. Also, the light updates in Home Assistant were a little slow since it required possibly sending multiple commands to the hub and then receiving a state message back to home assistant to update the entity. This probably wasn't a huge deal, but as you can imagine, my testing was artificial and involved a lot of stress testing which is probably beyond real world usage.

Summary Comparison

Zigbee2Tasmota	ZHA
Functions independently from Home Assistant	Nice and user-friendly GUI
Easily change Zigbee channel	Automatic device/entity creation after pairing
Good compatibility with nontraditional Zigbee protocols	Better light bulb compatibility and control

Helpful Zigbee2Tasmota Options

Ok, so you've read about Zigbee2Tasmota so far and you want to give it a try. There were a few global Tasmota options I found very helpful in simplifying the administration of the network.

By default, every Zigbee device is communicated over MQTT using the same topic, tele/%topic%/SENSOR.

By setting SetOption89 to 1, each device is given their own topic, tele/%topic%/<device>/SENSOR. I found this much easier to keep the devices separated.

Each device is given a unique identifier when it joins the network, and that is used to initially populate the MQTT topic if you decided to make the change I suggested above, but there is fortunately a way to give friendly names to each device.

ZBName <device>,<new_friendly_name>

Once you do that, the device friendly name will display on the Tasmota home screen, and you can set SetOption112 to 1 so that the friendly name will be used in the MQTT topic instead of the unique identifier.

Ok, so now that we've done that, our Zigbee coordinator should be good to go. We have it connected to our MQTT broker and have tweaked the settings enough to hopefully add some devices.

Example Home Assistant Entities

Here are some examples I made to control various devices. Like I said above, the MQTT topic for each device will now be tele/Zigbee/<friendly_device_name>/SENSOR. I set my topic to zbbridge in Tasmota, so change yours accordingly along with the device names. I did my best to bold the device-specific names and options below.

Motion detector:

  - platform: mqtt
    name: master bath motion
    state_topic: "tele/zbbridge/master_bath_motion/SENSOR"
    value_template: "{{ value_json['ZbReceived']['master_bath_motion']['Occupancy'] }}"
    payload_on: 1
    payload_off: 0
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "motion"

Contact sensor:

  - platform: mqtt
    name: half bath door sensor
    state_topic: "tele/zbbridge/half_bath_door/SENSOR"
    value_template: "{{ value_json['ZbReceived']['half_bath_door']['ZoneStatusChange'] }}"
    payload_on: 1
    payload_off: 0
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "door"

Battery sensor:

This was a little trickier. A battery percentage is not sent with each device update message. I added a template to the MQTT entity to say, "if message has updated value, use that, otherwise, keep old value." Without that addition, each time an updated message came without the specific value, the entity in Home Assistant became undefined.

  - platform: mqtt
    name: half bath door battery
    state_topic: "tele/zbbridge/half_bath_door/SENSOR"
    value_template: >
      {% if value_json['ZbReceived']['half_bath_door']['BatteryPercentage'] > 0 %}
        {{ value_json['ZbReceived']['half_bath_door']['BatteryPercentage'] }}
      {% else %}
        {{ states(entity_id) }}
      {% endif %}
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "battery"
    expire_after: 86400
    unit_of_measurement: '%'

Humidity and Temperature Sensors:

The same is true for the Aqara Humidity and Temperature Sensor I use. Each message did not have ever variable and so it was important to only update the MQTT entity when the message had the necessary information. Also, to note, the temperature was reported in Celsius to Tasmota and so I had to add a freedom conversion to my MQTT entity for Fahrenheit.

  - platform: mqtt
    name: refrigerator temperature sensor
    state_topic: "tele/zbbridge/refrigerator_temperature_sensor/SENSOR"
    value_template: >
      {% if value_json['ZbReceived']['refrigerator_temperature_sensor']['Temperature'] > 0 %}
        {{ (value_json['ZbReceived']['refrigerator_temperature_sensor']['Temperature']|float + 1.8) + 32 }}
      {% else %}
        {{ states(entity_id) }}
      {% endif %}
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "temperature"
    expire_after: 86400
    unit_of_measurement: 'F'

  - platform: mqtt
    name: refrigerator humidity sensor
    state_topic: "tele/zbbridge/refrigerator_temperature_sensor/SENSOR"
    value_template: >
      {% if value_json['ZbReceived']['refrigerator_temperature_sensor']['Humidity'] > 0 %}
        {{ value_json['ZbReceived']['refrigerator_temperature_sensor']['Humidity'] }}
      {% else %}
        {{ states(entity_id) }}
      {% endif %}
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "humidity"
    expire_after: 86400
    unit_of_measurement: '%'

Lights:

Ok, so far things have been relatively straightforward. Here is where they get tricky. I am in no way saying that this is the easiest or best way to accomplish light control. This is what I did which seemed to work. I did not attach any colored lights (thank god as I can only imagine that would add to the complication), but I did attach a number of white tunable lights.

Like I said above, the lights can only receive a single command at a time. Luckily, it is possible to group lights and send 1 command for multiple lights. To do so, use this following command:

ZbSend {"device":"IKEA_Light","Send":{"AddGroup":100}}

Replace the device with either the friendly name or unique identifier. The group number is a unique, arbitrary integer for each group. As far as I know, it is not possible to name the groups from within Tasmota and their status is not displayed on the front page.

First I will show how I handled a single light. It required both an mqtt entity and a template entity. The MQTT entity basically brought the device into home assistant, and the template entity made its control simpler by allowing me to change color temp and brightness at the same time by breaking down the single home assistant command into two separate MQTT commands. The MQTT device, for me, basically just sat there, hidden, and the Template light is how I interacted with the light.

Single Light:

   - platform: template
     lights:
       guest_bedroom_lamp_template:
         level_template: >
           {% if state_attr('light.guest_bedroom_lamp','brightness')|int >0 %}
             {{ state_attr('light.guest_bedroom_lamp','brightness') }}
           {% else %}
             0
           {% endif %}
         value_template: "{{ states('light.guest_bedroom_lamp') }}"
         temperature_template: >
           {% if state_attr('light.guest_bedroom_lamp','color_temp')|int >0 %}
             {{ state_attr('light.guest_bedroom_lamp','color_temp') }}
           {% else %}
             0
           {% endif %}
         turn_on:
           - service: light.turn_on
             data_template:
               entity_id: light.guest_bedroom_lamp
               brightness: "{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}"
           - service: light.turn_on
             data_template:
               entity_id: light.guest_bedroom_lamp
               color_temp: "{{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }}"
         turn_off:
           - service: light.turn_off
             data_template:
               entity_id: light.guest_bedroom_lamp
         set_level:
         - service: light.turn_on
           data_template:
             brightness: "{{ brightness|int }}"
             entity_id: light.guest_bedroom_lamp
         set_temperature:
         - service: script.change_brightness_and_color_temp
           data_template:
             light_ct: light.guest_bedroom_lamp
             brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
         - service: light.turn_on
           data_template:
             color_temp: "{{ color_temp|int }}"
             entity_id: light.guest_bedroom_lamp
         max_mireds_template: 370
         min_mireds_template: 153

   - platform: mqtt
     schema: template
     name: "Guest Bedroom Lamp"
     command_topic: "cmnd/zbbridge/ZbSend"
     state_topic: "tele/zbbridge/guest_room_lamp/SENSOR"
     command_on_template: >
       {"device": "guest_room_lamp","send":{
       {%- if brightness is defined -%}
       "Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness }}{%- endif -%}
       {%- elif color_temp is defined -%}
       "CT": {{ color_temp }}
       {%- else -%}
       "Power": 1
       {%- endif -%}
       }}
     command_off_template: '{"device": "guest_room_lamp","send":{"Power":0}}'
     state_template: >
       {% if value_json['ZbReceived']['guest_room_lamp']['Power'] == 1 %}
         on
       {% elif value_json['ZbReceived']['guest_room_lamp']['Power'] == 0 %}
         off
       {% else  %}
         {{ states('light.guest_bedroom_lamp') }}
       {% endif %}
     brightness_template: >
       {% if value_json['ZbReceived']['guest_room_lamp']['Dimmer'] is defined %}
         {% if value_json['ZbReceived']['guest_room_lamp']['Dimmer'] == 255 %}
           254
         {% else %}
           {{ value_json['ZbReceived']['guest_room_lamp']['Dimmer'] }}
         {% endif %}
       {% else %}
         {{ state_attr('light.guest_bedroom_lamp','brightness') }}
       {% endif %} 
     color_temp_template: >
       {% if value_json['ZbReceived']['guest_room_lamp']['CT'] is defined %}
         {% if value_json['ZbReceived']['guest_room_lamp']['CT'] > 370 %}
           370
         {% elif value_json['ZbReceived']['guest_room_lamp']['CT'] < 153 %}
           153
         {% else %}
           {{ value_json['ZbReceived']['guest_room_lamp']['CT'] }}
         {% endif %}
       {% else %}
         {{ state_attr('light.guest_bedroom_lamp','color_temp') }}
       {% endif %} 
     availability_topic: "tele/zbbridge/LWT"
     payload_available: "Online"
     payload_not_available: "Offline"
     optimistic: true
     qos: 0
     max_mireds: 370
     min_mireds: 153

So basically, Guest Bedroom Lamp is the mqtt light, and Guest bedroom lamp template is (obviously), the template light. The only reason I made a template light was so I could change the temperature and brightness at the same time. For a template light, if the brightness variable is defined, the set_level script is called. However, if brightness and color_temp are both defined, the color_temp script is called. To deal with this, I made a separate script which basically turns the light on at the desired brightness first if brightness is defined and then sends the color_temp command. The script which you can see is called from my template script is:

alias: change brightness and color temp
sequence:
  - choose:
      - conditions:
          - condition: template
            value_template: '{{ brightness != -1 }}'
        sequence:
          - service: light.turn_on
            data_template:
              entity_id: '{{ light_ct }}'
              brightness: '{{ brightness }}'
    default: []
mode: parallel
max: 100

As you can see, my lights are also customized with the min and max color_temp and to use circadian lighting (which is an awesome HACS integration).

Group Lights

Controlling a group of lights was also a little tricky. I set up the same MQTT and template entities for each light, but the group light required its own template entity.

  - platform: template
    lights:
      kids_bath_1:
        friendly_name: "kids_bath_1"
        level_template: >
          {% if state_attr('light.kids_bath_1_mqtt','brightness')|int >0 %}
            {{ state_attr('light.kids_bath_1_mqtt','brightness') }}
          {% else %}
            0
          {% endif %}
        value_template: "{{ states('light.kids_bath_1_mqtt') }}"
        temperature_template: >
          {% if state_attr('light.kids_bath_1_mqtt','color_temp')|int >0 %}
            {{ state_attr('light.kids_bath_1_mqtt','color_temp') }}
          {% else %}
            0
          {% endif %}
        turn_on:
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_1_mqtt
              brightness: "{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}"
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_1_mqtt
              color_temp: "{{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }}"
        turn_off:
          - service: light.turn_off
            data_template:
              entity_id: light.kids_bath_1_mqtt
        set_level:
        - service: light.turn_on
          data_template:
            brightness: "{{ brightness|int }}"
            entity_id: light.kids_bath_1_mqtt
        set_temperature:
        - service: script.change_brightness_and_color_temp
          data_template:
            light_ct: light.kids_bath_1_mqtt
            brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
        - service: light.turn_on
          data_template:
            color_temp: "{{ color_temp|int }}"
            entity_id: light.kids_bath_1_mqtt
        max_mireds_template: 370
        min_mireds_template: 153

  - platform: mqtt
    schema: template
    name: "kids bath 1 mqtt"
    command_topic: "cmnd/zbbridge/ZbSend"
    state_topic: "tele/zbbridge/kids_bath_1/SENSOR"
    command_on_template: >
      {"device": "kids_bath_1","send":{
      {%- if brightness is defined -%}
      "Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness }}{%- endif -%}
      {%- elif color_temp is defined -%}
      "CT": {{ color_temp }}
      {%- else -%}
      "Power": 1
      {%- endif -%}
      }}
    command_off_template: '{"device": "kids_bath_1","send":{"Power":0}}'
    state_template: >
      {% if value_json['ZbReceived']['kids_bath_1']['Power'] == 0 %}
        off
      {% elif value_json['ZbReceived']['kids_bath_1']['Power'] == 1 %}
        on
      {% else  %}
        {{ states('light.kids_bath_1_mqtt') }}
      {% endif %}
    brightness_template: >
      {% if value_json['ZbReceived']['kids_bath_1']['Dimmer'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_1']['Dimmer'] == 255 %}
          254
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_1']['Dimmer'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_1_mqtt','brightness') }}
      {% endif %} 
    color_temp_template: >
      {% if value_json['ZbReceived']['kids_bath_1']['CT'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_1']['CT'] > 370 %}
          370
        {% elif value_json['ZbReceived']['kids_bath_1']['CT'] < 153 %}
          153
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_1']['CT'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_1_mqtt','color_temp') }}
      {% endif %} 
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    optimistic: false
    qos: 0
    max_mireds: 370
    min_mireds: 153
    
  - platform: template
    lights:
      kids_bath_2:
        friendly_name: "kids_bath_2"
        level_template: >
          {% if state_attr('light.kids_bath_2_mqtt','brightness')|int >0 %}
            {{ state_attr('light.kids_bath_2_mqtt','brightness') }}
          {% else %}
            0
          {% endif %}
        value_template: "{{ is_state('light.kids_bath_2_mqtt','on') }}"
        temperature_template: >
          {% if state_attr('light.kids_bath_2_mqtt','color_temp')|int >0 %}
            {{ state_attr('light.kids_bath_2_mqtt','color_temp') }}
          {% else %}
            0
          {% endif %}
        turn_on:
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_2_mqtt
              brightness: "{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}"
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_2_mqtt
              color_temp: "{{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }}"
        turn_off:
          - service: light.turn_off
            data_template:
              entity_id: light.kids_bath_2_mqtt
        set_level:
        - service: light.turn_on
          data_template:
            brightness: "{{ brightness|int }}"
            entity_id: light.kids_bath_2_mqtt
        set_temperature:
        - service: script.change_brightness_and_color_temp
          data_template:
            light_ct: light.kids_bath_2_mqtt
            brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
        - service: light.turn_on
          data_template:
            color_temp: "{{ color_temp|int }}"
            entity_id: light.kids_bath_2_mqtt
        max_mireds_template: 370
        min_mireds_template: 153

  - platform: mqtt
    schema: template
    name: "kids bath 2 mqtt"
    command_topic: "cmnd/zbbridge/ZbSend"
    state_topic: "tele/zbbridge/kids_bath_2/SENSOR"
    command_on_template: >
      {"device": "kids_bath_2","send":{
      {%- if brightness is defined -%}
      "Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness }}{%- endif -%}
      {%- elif color_temp is defined -%}
      "CT": {{ color_temp }}
      {%- else -%}
      "Power": 1
      {%- endif -%}
      },"endpoint":1}
    command_off_template: '{"device": "kids_bath_2","send":{"Power":0}}'
    state_template: >
      {% if value_json['ZbReceived']['kids_bath_2']['Power'] == 0 %}
        off
      {% elif value_json['ZbReceived']['kids_bath_2']['Power'] == 1 %}
        on
      {% else  %}
        {{ states('light.kids_bath_2_mqtt') }}
      {% endif %}
    brightness_template: >
      {% if value_json['ZbReceived']['kids_bath_2']['Dimmer'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_2']['Dimmer'] == 255 %}
          254
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_2']['Dimmer'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_2_mqtt','brightness') }}
      {% endif %} 
    color_temp_template: >
      {% if value_json['ZbReceived']['kids_bath_2']['CT'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_2']['CT'] > 370 %}
          370
        {% elif value_json['ZbReceived']['kids_bath_2']['CT'] < 153 %}
          153
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_2']['CT'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_2_mqtt','color_temp') }}
      {% endif %} 
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    optimistic: false
    qos: 0
    max_mireds: 370
    min_mireds: 153
    
  - platform: template
    lights:
      kids_bath:
        friendly_name: "kids_bath"
        level_template: "{{ (state_attr('light.kids_bath_1_mqtt','brightness')|int + state_attr('light.kids_bath_2_mqtt','brightness')|int )/2 }}"
        value_template: "{{ is_state('light.kids_bath_2_mqtt','on') and is_state('light.kids_bath_1_mqtt','on') }}"
        temperature_template: "{{ (state_attr('light.kids_bath_1_mqtt','color_temp')|int + state_attr('light.kids_bath_2_mqtt','color_temp')|int )/2 }}"
        turn_on:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}{%- endif -%}}}
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"CT": {{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }} }}
        turn_off:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: '{"group": "101","send":{"Power":0}}'
        set_level:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness|int }}{%- endif -%}}}
        set_temperature:
          - service: script.change_brightness_and_color_temp_group
            data_template:
              group: 101
              brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"CT": {{color_temp}} }}
        max_mireds_template: 370
        min_mireds_template: 153

This assumes an arbitrarily defined group number of 101 and uses a similar script to control both brightness and color_temp at the same time.

alias: change brightness and color temp group
sequence:
  - choose:
      - conditions:
          - condition: template
            value_template: '{{ brightness != -1 }}'
        sequence:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "{{ group }}","send":{"Dimmer": {%- if brightness ==
                255 -%}254{%- else -%}{{ brightness|int }}{%- endif -%}}}
    default: []
mode: parallel
max: 100

Conclusion:

As you can see, Zigbee2Tasmota is a fairly powerful platform, incredibly operating on a system with very limited resources. It is not for everyone, but could be enticing primarily for people with the Tasmotized Sonoff Zigbee Bridge. Connecting a WiFi device to Home Assistant through serial over TCP may not be the best or most stable setup. Instead allowing the coordinator to function independently and sharing the device information over MQTT might be more desirable.

On the other hand, if you have Tube's wired zigbee coordinator or the ZB-GW03 wired coordinator, you may be better off using ZHA for simplicity sake. Regarding Zigbee2MQTT, it should be noted that the EZSP radio found in the Sonoff Zigbee Adapter and the ZB-GW03 is not finalized and may be buggy.

Links to adapters:

Sonoff Zigbee

Affiliate link:

SONOFF ZBBridge Smart Zigbee Bridge Hub, WI-FI & Zigbee Dual-protocol Supporting, APP Control and Multi-device Management

Non-affiliate link:

SONOFF ZBBridge Smart Zigbee Bridge Hub, WI-FI & Zigbee Dual-protocol Supporting, APP Control and Multi-device Management

Eachen ZB-GW03

Tube's ZB Coordinator

Let me know if you have any questions. Thanks for reading!

Edit: changed instances of zigbee router to the correct terminology zigbee coordinator thanks to /u/chick_repellent.

Playing a sound directly from Home Assistant Operating System

thehelpfulidiot — Sat, 02 Oct 2021 00:00:00 +0000

This may sound like it should've been a trivial task to accomplish but took me way longer to solve than I anticipated. As I said it my first post, I started my Home Automation journey with things like Nest. This included the (now killedbygoogle) Nest Secure with Nest Detects. I did enjoy a very pleasing *DING *each time a door opened. As time went on, I tried to distance myself as much as possible from the cloud.

Integrating them into Home Assistant was a major PITA though. I had to purchase the amazing Starling Home Hub, and use Homekit to create an automation where each door open and door close event would trigger an automation to control an MQTT switch through Homebridge which would then pass the information into Home Assistant. This worked pretty well for the most part, but once the batteries started to die in the Nest Detects, it seemed like the perfect opportunity to replace the door sensors.

I just purchased cheap Aqara Door Sensors and mounted them on my door. Obviously, the open/close information was passed far more easily into Home Assistant and I was able to simplify my work flow, but one thing I did miss was the DING.

DING

I wanted something as simple as possible, and to me, the obvious solution was to connect a speaker to my Home Assistant box's 3.5mm jack and just play the sound. This was wayyyy more complicated than I expected.

Home Assistant's addon store has a very bizarre version of VLC. It seems like it is feature incomplete and has almost no options. I couldn't figure out how to use it.

VLC from Official Add-on Store

Fortunately, there exists another version of VLC from a repository belonging to rodripf. Simply add the following address to the add-on repository list to install it:

https://github.com/rodripf/hassio-local-vlc

Once installed, it can be configured for telnet access. This is what I found to be the easiest and most reliable way to play a sound. Set the following options as desired.

Once that is set up, it is necessary to add a vlc_telnet entity which is done through yaml.

media_player:
  - platform: vlc_telnet
    name: vlc
    host: 6872d8d8-local-vlc
    password: <secret password>

Edit the entry as you see fit, but the important thing is the host, which identifies the Home Assistant container, should be written as above.

After a reboot, "media_player.vlc" should exist and through this, you can play whatever sound you want, locally. Although, I had trouble playing the music unless I uploaded it to the "/share" directory.

Once you've uploaded your sound file to the /share directory, the following will play it:

service: media_player.play_media
target:
  entity_id: media_player.vlc
data:
  media_content_id: /share/chime.wav
  media_content_type: music

It is important to make the media_content_type "music" as that is the only media type currently playable by the vlc_telnet integration.

You can go to a web gui of your VLC instance by going to your homeassistanturl:9892 and using the password you created in the addon settings page.

It's important to make sure your volume here is set to something reasonable. At first, I could not get any sound to play since the volume defaulted to 0.

You can also set the volume through Home Assistant as you would for any media player.

service: media_player.volume_set
target:
  entity_id: media_player.vlc
data:
  volume_level: 0.35

Hope that helps. Seems like a fairly complex solution for a fairly simply problem, but like I said, this was the only way I could figure to do something like this from within Home Assistant Operating System (obviously other installations in standard operating systems can probably do this trivially).

The nice thing about this as opposed to the Nest solution, is that now I have control over when the door dings and when it doesn't. For example, my ding automation does not fire when my kids are asleep. Now I don't have to worry about waking them when I run out to the car.

Thanks for reading.

A Wired Sonoff Zigbee Alternative

thehelpfulidiot — Wed, 29 Sep 2021 00:00:00 +0000

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

UPDATE 12/5: We've come full circle. The amazing developers of Tasmota have fixed the issue causing serial port disconnections to the zigbee radio with Tasmota 10. Now, the below instructions can be followed, without relying on the previous update posts from 10/15 and 10/18. The uploaded precompiled Tasmota firmware below has been updated to the newest version as of today.

Many of us have seen the Sonoff Zigbee Hub and how easy it is to flash to Tasmota thanks to many awesome videos and my personal favorite by DigiblurDIY. The device is amazing and works well, but some things like your Zigbee bridge shouldn't rely on WiFi.

Recently, another amazing individual going by Tube has created a homemade Franken Zigbee router using an ESP32 board, ethernet, zigbee, and a 3D printed case. It comes preflashed with ESPHome and is instantly compatible with Home Assistant ZHA or Zigbee2MQTT. However, each one of these devices is being made by hand and supply cannot meet demand.

I came across another wired zigbee hub using an ESP32 - the EACHEN Zigbee Bridge Pro. For only $30, I thought I would give it a try. It shipped from China, but arrived fairly quickly after only a couple of weeks.

The hub

An ethernet port, a USB-C power port, and a reset pin hole

The device is probably the same size as the Sonoff Zigbee Router. It is easy enough to disassemble though since there are no screws and the bottom can simply be pried off.

The board

The board pops right out. It is now ripe for flashing and you can see some solder still on mine. My experience with Tasmota so far had been with mainly Tuya-convert and flashing the Sonoff Zigbee router which did not require soldering. This was my first time soldering and was way more fun than I expected.

Now before we get to soldering and flashing, we need to compile our own Tasmota firmware. We need to make sure Tasmota has access to the Ethernet port and the Zigbee Radio. As in all things Tasmota, Blakadder has a page devoted to this device and instructions. The instructions were excellent, but assumed a certain level of knowledge to compile the firmware. Having not done this before, this part took me the longest. The tasmota32-zigbeebridge precompiled firmware mentioned on the site did not provide ethernet access or else this process would have been a lot easier.

After much trial and error, the easiest way I found to compile this firmware was by going to GitPod. After you log in, you are taken to a VS code window prepopulated with the latest Tasmota build.

The instructions from Blakadder's website say to edit two files- user_config_override.h and platform_override.ini.

'Platform_override.ini' is located in the root directory - /workspace/Tasmota. Open the file, scroll to the bottom, and add:

[env:tasmota32-EWELINK]
extends                 = env:tasmota32
board_build.f_cpu       = 240000000L
board_build.f_flash     = 40000000L
build_flags             = ${env:tasmota32_base.build_flags} -DFIRMWARE_EWELINK

Now find the 'user_config_override.h' file which is located in the tasmota subdirectory (/workspace/Tasmota/tasmota). Open that file, erase all its contents and replace with:

#ifdef FIRMWARE_EWELINK
  #warning **** Build: FIRMWARE FOR EWELINK ZB-GW03-V1.2 ****

  #undef  SERIAL_LOG_LEVEL
  #define SERIAL_LOG_LEVEL LOG_LEVEL_NONE

  #define USE_ZIGBEE
  #undef  USE_ZIGBEE_ZNP
  #define USE_ZIGBEE_EZSP
  #define USE_UFILESYS
  #define USE_ZIGBEE_EEPROM // T24C512A
  #define USE_TCP_BRIDGE
  #undef  USE_ZIGBEE_CHANNEL
  #define USE_ZIGBEE_CHANNEL 11 // (11-26)
  
  #define USE_ETHERNET
  #undef  ETH_TYPE
  #define ETH_TYPE 0 // ETH_PHY_LAN8720
  #undef  ETH_CLKMODE
  #define ETH_CLKMODE 3 // ETH_CLOCK_GPIO17_OUT
  #undef  ETH_ADDRESS
  #define ETH_ADDRESS 1 // PHY1
#endif

Ok, so those two parts were relatively easy, but the next part on how to compile took me probably longer than it should have to figure it out. In the terminal at the bottom of the screen, while in the /workspace/Tasmota directory, type the following:

platformio run -e tasmota32-EWELINK

Now wait patiently for the new firmware to compile. Once it does, it can be found in /workspace/Tasmota/build_output/firmware. Download 'tasmota32-EWELINK.bin.

tasmota32-EWELINK

I compiled this on 10/7/2021, Tasmota32 9.5.0. I promise that I did not tamper with it (and would not know how to tamper with it if I wanted), but you should probably not download firmware from some random guy on the internet.

This is Tasmota 10 firmware compiled 12/5/21 using the above method. It should work as intended without the previously described work-arounds.

NOTE: As reported by erkoc, my new tasmota firmware based on v10 was still giving him the same issue with reboots losing connection to ZHA without manual intervention. He forked Tasmota32 and made sure the serial communication patch was included. He has instructions on how to compile your own or you can simply download his precompiled version.

https://github.com/vahempio/Tasmota-for-eWeLink

Ok, now that we have the firmware, it's time to start flashing. Since we have an ESP32. I do not believe we can use Tasmotizer. I have Windows and used ESP-Flasher-Windows-x64. Open it up, choose your serial port, the firmware we just compiled.

Now the fun part with soldering. Below I have a photo of the ESP32 board and the corresponding connection on my Serial Adapter (I use this one).

ESP32 board - labels point to corresponding pin on Serial Converter

Serial Converter - labels point to corresponding pin on ESP32

Now Flash.

Once you finish, it's a matter of setting up Tasmota and a few more finishing touches.

Set up Tasmota through the access point first and add this Template by issuing the following command:

backlog Template {"NAME":"ZB-GW03-V1.2","GPIO":[0,0,3552,0,3584,0,0,0,5793,5792,320,544,5536,0,5600,0,0,0,0,5568,0,0,0,0,0,0,0,0,608,640,32,0,0,0,0,0],"FLAG":0,"BASE":1} ; module 0

From this website, select ncp-uart-sw-6.7.8_115200.ota. Go to the upgrade firmware screen to update the zigbee radio. The device will automatically recognize this as firmware for the zigbee radio and will apply it accordingly.

Now try plugging the device in to Ethernet. I noticed it was sort of strange and took a couple reboots and unplugging and plugging the cable back in, but I saw messages about setting up the Eth connection in the console. There was no real way to change Ethernet settings like you can with WiFi, but I noticed two DHCP listings in my router. Once Ethernet is set up, you can disconnect WiFi with the following, but do not do the following until Ethernet is set up! Make sure the first command turning WiFi off works first before applying the subsequent rule which will disable it on each reboot. The first command by itself will not persist across reboots.

Wifi 0
backlog rule2 on system#boot do Wifi 0 endon ; rule2 1

If something went wrong and now you are locked out of the device because Ethernet isn't working and WiFi is down, you can try Tasmota device recovery.

The device is now set up to use with Zigbee2Tasmota. For Home Assistant, there are a few more steps.

To integrate with Home Assistant, we must remap the Zigbee Tx and Zigbee Rx pins directly to TCP Tx and TCP Rx. This will disable Zigbee2Tasmota, but will allow the device to integrate with Home Assistant. Use the following command to update the template:

backlog template {"NAME":"ZHA-bridge","GPIO":[0,0,5472,0,5504,0,0,0,5793,5792,320,544,5536,0,5600,0,0,0,0,5568,0,0,0,0,0,0,0,0,608,640,32,0,0,0,0,0],"FLAG":0,"BASE":1} ; module 0

Notice the change in the third and fifth positions. Afterwards, it will be possible to start the TCP Server which is what ZHA needs to connect to the coordinator.

Enter the following from the console to turn on the webserver for the Zigbee radio:

backlog rule1 on system#boot do TCPStart 8888 endon ; rule1 1 ; tcpstart 8888

Finally, you can jump into Home Assistant and set up the device for ZHA exactly how it's (masterfully) described by DigiBlur.

https://www.digiblur.com/2020/07/how-to-use-sonoff-zigbee-bridge-with.html

Thanks for reading. I hope this helps fill in the blanks for some folks like myself who are not as familiar with the inner workings of Tasmota. Please let me know if you have any questions.

Thanks to DigiBlur and Blakadder and all the other folks who did the hard work in getting this device to work.

Alternative: a single template, no toggling

A reader working through this guide found a cleaner approach that avoids the ZHA/Z2T toggling described above entirely. Rather than the ZHA-bridge template, use:

backlog template {"NAME":"Zig_Bridge32","GPIO":[1,1,5472,1,5504,1,1,1,5793,5792,320,544,5536,1,5600,1,0,1,1,5568,0,1,1,1,0,0,0,0,608,640,32,1,1,3552,3584,1],"FLAG":0,"BASE":1}; module 0;

The reasoning: the ZHA-bridge template has no Zigbee Tx/Rx GPIOs at all, so Tasmota never resets the EZSP chip at boot. Mapping Zigbee Rx and Tx to otherwise unused GPIOs means the reset does happen, and the TCP connection works across a cold start. The Zigbee2Tasmota UI is still visible but unused.

For Zigbee2MQTT, adding this to your config makes it reconnect within a minute of a bridge reboot:

availability:
  active:
    timeout: 1

Link to Purchase

Affiliate Link:

AliExpress.com Product - EACHEN EWeLink Zigbee Bridge Pro (Zbbridge Pro)

eWelink ZigBee 3.0 Smart Wired Wireless Gateway Hub

Non-Affiliate Link:

AliExpress.com Product - EACHEN EWeLink Zigbee Bridge Pro (Zbbridge Pro)

eWelink ZigBee 3.0 Smart Wired Wireless Gateway Hub

EACHEN Website:

EACHEN eWeLink Zigbee Bridge Pro (zbbridge pro)

CORRECTION 10/2: added steps to remap zigbee Tx and Rx to TCP.

CORRECTION 10/7: made changes based on feedback from /u/Vertigo722. Edited platformIO text. Changed some order of setup.

CORRECTION 10/14: hard reset of the coordinator breaks connection with ZHA until toggled back to Z2T and then back again. Using this firmware and the rules below automatically toggle back and forth only on cut power. Hopefully this workaround fixes things for now.

tasmota32-EWELINK-1

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

About

thehelpfulidiot — Wed, 29 Sep 2021 00:00:00 +0000

As someone without a tech background, my introduction to the world of self hosting and home automation began as a slow and bumpy ride with many false starts. Since beginning seriously in 2019, I am proud of what I have accomplished with OPNsense, Home Assistant, Proxmox, Docker, to name a few. Here is a tech blog, not from a computer scientist or an engineer, but from an ophthalmologist.

The Birth of a Smart Home

thehelpfulidiot — Wed, 29 Sep 2021 00:00:00 +0000

When the first Google Home came out in 2016, I briefly thought about buying it, but decided I had no use for it. Shortly after, my brother-in-law bought me two Chinese Tuya White-label smart plugs from Amazon. I plugged a couple of lamps into them and was hooked. Google home followed shortly after. At the time, we were renting and so my ability to "smartify" our home was limited. I replaced many light bulbs with Sengled bulbs, changed out smoke detectors for Nest, but that was pretty much it.

In 2019 we bought our first home. I knew from the beginning that building a smart home would be an obsession. During the year leading up to the move, I kept my eye on deals and snatched up some of the essentials- Hue light bulbs, Nest smoke detectors, Nest lock, Nest thermostats, and maybe a few others. Anything that could be replaced by an electronic wireless device was on the chopping block.

At this point I knew nothing about Linux or Home Assistant. The best smart home options were the well-known cloud versions. I set up a few automations with SmartThings, but that was it.

After a few months, I came across a Reddit Post about Mozilla Webthings. It was an incredibly simple smart home solution which could be installed on a raspberry pi. I drove straight to Micro Center to pick up the necessary equipment and got started. This solution was game changing but I quickly reached it's limitations and the never-ending quest for smart home supremacy began.

Below is a diagram of my current setup:

Network Diagram

Initially, I migrated everything to the fewest number of boxes as possible. This scared me since there was only a single point of failure. More recently, I have purchased very cheap celeron computers from Amazon Warehouse and have used a few to host various services.

I hope to share some lessons learned as someone who entered this knowing nothing and after a lot of hard work, I can honestly say I know more than nothing, but still very little.

Thanks for reading.