The Helpful Idiot

Making an Automatic Email Backup (Updated 9/15/2024)

thehelpfulidiot — Sun, 15 Sep 2024 00:00:00 +0000

This is basically an update to the project I discussed in these three previous posts:

Part 1 - https://thehelpfulidiot.com/making-an-automatic-email-backup

Part 2 -https://thehelpfulidiot.com/making-an-automatic-email-backup-part-2

Part 3 - https://thehelpfulidiot.com/making-an-automatic-email-backup-part-3

I have been using this method of backing up emails for the past several years and have been very please with the reliability. It has not required any modification or tinkering and has just seemed to work.

Having said that, even though it works reliably and doesn't need to be fixed, that doesn't mean I shouldn't try.

https://github.com/jon6fingrs/mbsync-dovecot

Enter this newest interation of my docker images for Mbsync and Dovecot. It uses updated versions of the software of course, but I have also made a few big changes which may be of use to some of you.

Solr

Now includes Solr integration!

Solr is an indexing and search platform. Previously, if performing a remote IMAP search through a generic IMAP client (some search their offline index), it would search each email sequentially and so could take a LONG time. Now, by deploying Solr at the same time, the search is MARKEDLY improved when I search using the Apple Mail app.

The solr docker container I have made includes the necessary setup for Dovecot, and the Dovecot container will look for the Solr container and so neither will now work without the other.

The solr container also includes cron and will write commits every 5 minutes and will optimize the database each night.

Mbsync Timing

Previously, the mbsync container would run once and then quit. Now, it will run to synchronize the mailbox, and then afterwards, it will open cron as the primary process, which will check for mail each 5 minutes. This avoids having to set a manual cronjob and keeps the container running.

If you want your own sync schedule, you can deactivate the built in cron with an environment variable and use whatever interval you would like.

Better Environment Variables

Some of the environment variables are now optional, and more intuitive. The run scripts are more robust.

What's the point?

But why did I do this? What's the point? Well, I would love to have my own mail server, but I am too worried about the administrative burden and dropped emails. This way, I feel like I am able to offload some of my dependence on the cloud by downloading and archiving every email, and just periodically purging my cloud provide inbox.

I have a complete email archive with all emails since my accounts opened. This is offline and does not require increasing my available cloud storage more and more.

Also, because I could.

So check it out if that sounds interesting and let me know how it works:

https://github.com/jon6fingrs/mbsync-dovecot

Nextcloud Docker with Nginx Proxy Manager

thehelpfulidiot — Sun, 31 Mar 2024 00:00:00 +0000

Nextcloud has become an integral part of my work flow these days. I love having a "place" to go that's totally mine. I have been running it from a docker container since before the AIO images were available and so have not had a reason to change.

The container comes in two flavors- Apache vs php-fpm. The I had been running the Apache version since it is generally easier, but always hated the idea of using Nginx Proxy Manager in front of Apache. It just seemed like a redundancy. Finally, I started to get some weird behaviors in Nextcloud which I tracked down as some webserver problem.

So now I have the perfect opportunity to fix something that really isn't breaking by doing wayyyy too much work.

I decided to try the php-fpm version and really wanted to take advantage of Nginx Proxy Manager and not have to use a second Nginx container!

Looking far and wide on the internet, most people said it was impossible, but I found really this single post that laid is out pretty nicely.

https://help.nextcloud.com/t/docker-nextcloud-fpm-behind-nginx-proxy-manager-do-i-need-another-nginx-container/133431/6

It turned out to be much easier than expected and I was essentially able to seamlessly swap out the Apache image for the php-fpm image by updating the docker compose file.

I did, however, update the nginx proxy configuration a bit from what was in that link to resolve various errors, to add the built-in collabora server, and to add push notifications.

	# set max upload size
        client_max_body_size 512M;
        fastcgi_buffers 64 4K;



	include mime.types;
	types 
	{
		application/javascript mjs;
	}

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Pagespeed is not supported by Nextcloud, so if your server is built
        # with the `ngx_pagespeed` module, uncomment this line to disable it.
        #pagespeed off;

        # HTTP response headers borrowed from Nextcloud `.htaccess`
        add_header Referrer-Policy                      "no-referrer"       always;
        add_header X-Content-Type-Options               "nosniff"           always;
        add_header X-Download-Options                   "noopen"            always;
        add_header X-Frame-Options                      "SAMEORIGIN"        always;
        add_header X-Permitted-Cross-Domain-Policies    "none"              always;
        add_header X-Robots-Tag                         "noindex, nofollow" always;
        add_header X-XSS-Protection                     "1; mode=block"     always;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        # Path to the root of your installation
        root /var/www/html;

        # Specify how to handle directories -- specifying `/index.php$request_uri`
        # here as the fallback means that Nginx always exhibits the desired behaviour
        # when a client requests a path that corresponds to a directory that exists
        # on the server. In particular, if that directory contains an index.php file,
        # that file is correctly served; if it doesn't, then the request is passed to
        # the front-end controller. This consistent behaviour means that we don't need
        # to specify custom rules for certain paths (e.g. images and other assets,
        # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
        # `try_files $uri $uri/ /index.php$request_uri`
        # always provides the desired behaviour.
        index index.php index.html /index.php$request_uri;

        # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
        location = / {
            if ( $http_user_agent ~ ^DavClnt ) {
                return 302 /remote.php/webdav/$is_args$args;
            }
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # Make a regex exception for `/.well-known` so that clients can still
        # access it despite the existence of the regex rule
        # `location ~ /(\.|autotest|...)` which would otherwise handle requests
        # for `/.well-known`.
        location ^~ /.well-known {
            # The rules in this block are an adaptation of the rules
            # in `.htaccess` that concern `/.well-known`.

            location = /.well-known/carddav { return 301 /remote.php/dav/; }
            location = /.well-known/caldav  { return 301 /remote.php/dav/; }

            location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
            location /.well-known/pki-validation    { try_files $uri $uri/ =404; }

            # Let Nextcloud's API for `/.well-known` URIs handle all other
            # requests by passing them to the front-end controller.
            return 301 /index.php$request_uri;
        }

        # Rules borrowed from `.htaccess` to hide certain paths from clients
        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }



        # Ensure this block, which passes PHP files to the PHP process, is above the blocks
        # which handle static assets (as seen below). If this block is not declared first,
        # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
        # to the URI, resulting in a HTTP 500 error response.
        location ~ \.php(?:$|/) {
            # Required for legacy support
            rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            set $path_info $fastcgi_path_info;

            try_files $fastcgi_script_name =404;

            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $path_info;
            #fastcgi_param HTTPS on;

            fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
            fastcgi_param front_controller_active true;     # Enable pretty urls
            fastcgi_pass nextcloud-nextcloud-1:9000;

            fastcgi_intercept_errors on;
            fastcgi_request_buffering off;
        }

    	location ~ \.(?:css|js|mjs|svg|gif|png|jpg|ico|wasm|tflite|map|ogg|flac)$ {
            try_files $uri /index.php$request_uri;
            expires 6M;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
    	}

        location ~ \.woff2?$ {
            try_files $uri /index.php$request_uri;
            expires 7d;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        # Rule borrowed from `.htaccess`
        location /remote {
            return 301 /remote.php$request_uri;
        }

	location ^~ /push/ {
        	proxy_pass http://nextcloud_notify:7867/;
        	proxy_http_version 1.1;
        	proxy_set_header Upgrade $http_upgrade;
        	proxy_set_header Connection "Upgrade";
        	proxy_set_header Host $host;
        	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        	}




        location / {
            try_files $uri $uri/ /index.php$request_uri;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Forwarded-Host $server_name;
        }

This has worked for Nextcloud 28.

Otherwise, the post pretty much explains everything else!

Fully Local Universal Remote Control with Home Assistant

thehelpfulidiot — Sun, 11 Feb 2024 00:00:00 +0000

Background

I have had a Logitech Harmony remote solution for almost 10 years. In my opinion, it is a very nice product. It integrates nicely into Home Assistant, and so, while it didn't behave as intuitively as I wanted out of the box, Home Assistant automations were able to modify its behavior so that it largely performed seamlessly in the way that I wanted.

Unfortunately, as with many other quality products, Logitech decided to discontinue the Harmony line of products.

Luckily, Logitech has learned from its previous mistake when it discontinued the Harmony Link years earlier. At the time, they announced that the devices would lose support and become bricks. This rustled a few jimmies and so they ultimately shipped a free Harmony Hub (the upgraded, but now current generation of IR hub) to the unhappy customers.

So even though Logitech has announced an end to the Harmony brand, they have stated that they will continue to support the devices currently in the wild. Well this is obviously a relief, but who can say how long we will get this reprieve before Logitech decides they've given us enough of their "free" computing power?

Recently, my dog ate one of my Harmony remotes. I had to purchase a used replacement from ebay. When it came, it was still paired to its previous hub, and trying to repair it set off a very difficult mission which required my downloading the official Harmony software onto a Windows Box so that I could restore my old profile.

It worked, but took several hours and was extremely frustrating to have to rely on a black box to fix something which had become integral to many automations and to my WAF. This got me thinking, how much longer do I have with Harmony? When will this next remote be destroyed? When will Harmony's recovery tools become out-dated and unusable?

This set me off on searching for a more stable, long-term solution.

Alternatives

So what are the alternatives? Well surprisingly few.

There appears to be a pretty popular Harmony replacement, called the Sofabaton. It seems to have received pretty positive reviews all around, and if I'm being honest, I didn't really look into it too much. Regardless of whether or not it behaves locally, I just saw it as trading one walled garden for another. It seemed like you would need an internet connection and the company's servers in order to add new devices. I wanted something that would be totally local.

Immediately, I was drawn to the Broadlink RM4 mini. It is a cheap IR blaser which can learn from IR remotes, or can use codes found online. It integrates very nicely into Home Assistant, and works over a local connection. I attempted to set it up normally, but there was some issue. The device was added to my wifi but not to my Broadlink account. However, this turned out to be the perfect outcome since I was able to integrate it into Home Assistant and now do not need to worry about a device calling home constantly.

The IR blaser is for my TV and soundbar which are not controllable by wifi. Otherwise, I have a kodi box, an Apple TV, and a Roku. These are all controllable over wifi.

So now, we have a way to interact with my media devices. I can use it through various automations, from my phone, a computer, etc. But I really just want a remote. There is nothing quite like controlling your media with an old-fashioned remote. It is always there and the batteries last like 10 years.

Perhaps there are bluetooth remotes available? My Home Assistant instance is in the basement while the media center is on the first floor. So if I am able to find a bluetooth remote, there needs to be a way to forward all commands to my Home Assistant instance.

Once that's done, it's just a matter of connecting the correct input with the correct output which would be fairly easy.

Solution: Remote Hardware

After much searching, I came across the Air Mouse Remote MX3 Pro with Backlit. It is not Bluetooth. I actually could not find a decent Bluetooth Remote. It does still use a 2.4ghz signal. It comes with a pre-paired USB dongle. Just plug it into a computer, and BOOM- the computer has a hardware keyboard device added.

Another nice feature is it has 5 programmable IR buttons at the bottom for backup in the event something isn't working.

There are many iterations of this remote. It seems to be a white labelled device and is just branded and repackaged by various companies. I did try several of them to try and find the best, but to be honest, they are all the same. I simply prefer the remote with the backlight versus without (although you do lose a button).

Problem: Send keypresses to Home Assistant

Ok, so the next problem. The remote is plugged into my Kodi box (Manjaro) in the family room and my Home Assistant box is in the basement. How do I get keypresses from one to the other?

I have played with USBIP in the past and, I don't like it very much. In my experience (and I am far from an expert), it works great until it doesn't. It doesn't really have a way to catch and recover from errors. It just wasn't the sort of stable, low maintenance solution I wanted (but that may have been an error on my part).

After searching far and low, I came across the git:

https://github.com/tjntomas/MI-Bluetooth-Remote-for-Home-Assistant.

It is a script for capturing a keyboard presses, and sending them to Home Assistsant so that each key_down and key_up event is converted to a Home Assistant event. From there, the hardware and "wiring" will be complete and it is a matter of using Home Assistant to create whatever series of automations for each key that your particular setup requires.

To use it, follow the instructions on the original repo. You need to make sure python is installed and that the python evtest module is installed.

The script wasn't perfect for me and so I made just a few changes. I wanted to specify the keyboard using a commandline argument since the remote I linked above creates three keyboard devices in linux. This is just easier that creating three seperate scripts. I also made sure that it would send the key code, along with the key name and type of press in the event data.

The following is a fork of the original git repo with the above mentioned and some other changes.

https://github.com/jon6fingrs/Bluetooth-Remote-for-Home-Assistant/tree/main

Here is my version of bt_remote_event.py:

"""
Script to listen to events from a bluetooth remote control and send
the events to Home Assistant.

Author: Tomas Jansson, https://github.com/tjntomas

"""

import json
import evdev # https://pypi.org/project/evdev/
import asyncio
import aiohttp
import logging
import os
import sys

# Set up logging and log levels.
logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
logging.getLogger('asyncio').setLevel(logging.CRITICAL)

BASE_API      = "http://IP_ADDRESS:8123/api/"  # URL to your HA instance.
DEV_INPUT     = "/dev/remote_"             
API_KEY       = "API_KEY_STRING"
HA_EVENT_NAME = "bt_remote" # Arbitrary name of the event that will get fired.
GRAB_DEVICE   = True # If set to True, the devices will be locked to this script and the system will not receive any events.

EVENT_PATH    = "events/" + HA_EVENT_NAME
BASE_API_URL  = BASE_API + EVENT_PATH
HEADERS       = {'content-type': 'application/json','Authorization': 'Bearer {}'.format(API_KEY)}
CMD           = "cmd"
CMD_TYPE      = "cmd_type"
CMD_NUM       = "cmd_num"

EVENT_LOG_TEMPLATE = "Fired event {} with event data{}"

async def run():
  device = evdev.InputDevice(DEV_INPUT + sys.argv[1])
  print("Using Bluetooth", str(device))
  
  if GRAB_DEVICE:
    device.grab()

  # Listen for events from the remote through the async-based evdev library.
  for event in device.read_loop():
    if event.type == evdev.ecodes.EV_KEY:   
      # Get the name of the remote key pressed, one of:
      # KEY_HOME    
      # KEY_F5      
      # KEY_COMPOSE   
      # KEY_BACK       
      # KEY_UP          
      # KEY_DOWN   
      # KEY_LEFT      
      # KEY_RIGHT    
      # KEY_ENTER      
      # KEY_VOLUMEUP   
      # KEY_VOLUMEDOWN 

      # The event string returned from the evdev library looks like this:
      # key event at 1609103448.769025, 28 (KEY_ENTER), down
      # We are only interested in the key name "KEY_ENTER" and the keypress type "down".
      cmd = str(evdev.categorize(event)).split(",")[1].split("(")[1].replace(")","")

      cmd_num = str(evdev.categorize(event)).split(",")[1].split("(")[0].replace(" ","")

      # Get the type of keypress, one of:
      # up
      # down
      # hold
      cmd_type = str(evdev.categorize(event)).split(",")[2].replace(" ","")

      # Compose the payload to send to HA when firing the event.
      payload = {CMD: cmd, CMD_TYPE: cmd_type, CMD_NUM: cmd_num}

      # Since the evdev library is async-based, we use async to send the event to HA.
      async with aiohttp.ClientSession() as session:
        await session.post(BASE_API_URL, data=json.dumps(payload), headers=HEADERS)
        logging.info(EVENT_LOG_TEMPLATE.format(HA_EVENT_NAME,payload))
        await session.close()

if __name__ == '__main__':
    # Create, start and gracefully shut down the asyncio event loop.
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(run())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

So, first we need to figure out what our keyboards are with a tool called evtest. On Ubuntu, it can be installed with:

sudo apt install evtest

This will list the name and location of all connected input devices, like the following:

No device specified, trying to scan all of /dev/input/event*
Not running as root, no devices may be available.
Available devices:
/dev/input/event0: Sleep Button
/dev/input/event1: Power Button
/dev/input/event10: Dell WMI hotkeys
/dev/input/event11: Video Bus
/dev/input/event12: DP-1
/dev/input/event13: HDA Intel PCH Headphone Mic
/dev/input/event14: HDA Intel PCH Front Line Out
/dev/input/event15: HDA Intel PCH HDMI/DP,pcm=3
/dev/input/event16: HDA Intel PCH HDMI/DP,pcm=7
/dev/input/event17: HDA Intel PCH HDMI/DP,pcm=8
/dev/input/event18: ydotoold virtual device
/dev/input/event2: Power Button
/dev/input/event3: flirc.tv flirc Keyboard
/dev/input/event4: 2.4G Composite Devic
/dev/input/event5: 2.4G Composite Devic Mouse
/dev/input/event6: 2.4G Composite Devic Consumer Control
/dev/input/event7: 2.4G Composite Devic System Control
/dev/input/event8: Pulse-Eight USB-CEC Adapter
/dev/input/event9: PC Speaker
Select the device event number [0-18]:

The "four" devices which represent my remote are:

/dev/input/event4: 2.4G Composite Devic
/dev/input/event5: 2.4G Composite Devic Mouse
/dev/input/event6: 2.4G Composite Devic Consumer Control
/dev/input/event7: 2.4G Composite Devic System Control

The devices at 4, 6, and 7 represent the keyboards. The remote also can toggle into a gyroscopic mouse. This actually works out great for me since it is connected directly to my kodi box, so even though we will continue with the complicated setup that prevents the keyboards from directly interacting with the system, the mouse will still work if desired.

Anyway, if you try to access one of these devices, you will get:

Testing ... (interrupt to exit)
***********************************************
  This device is grabbed by another process.
  No events are available to evtest while the
  other grab is active.
  In most cases, this is caused by an X driver,
  try VT-switching and re-run evtest again.
  Run the following command to see processes with
  an open fd on this device
 "fuser -v /dev/input/event6"
***********************************************

Ok, so there are two things we have to accomplish.

we need to deactivate the devices
we need to give each device a stable name (eventX could change)

Luckily, there is one solution to accomplish both of these things- udev! I have added an example udev rule to my above linked git.

SUBSYSTEMS=="input", ATTRS{name}=="keyboard name from evtest", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_abc"

Each device will need its own file. Maybe it could be done with one, but I am not an expert, I did it with three. It is a one time procedure and then you forget about it. If you have a nice way to accomplish the same with one file, however, please let me know.

Anyway, my subsequent three files will look like:

SUBSYSTEMS=="input", ATTRS{name}=="2.4G Composite Devic Consumer Control", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_consumer"

SUBSYSTEMS=="input", ATTRS{name}=="2.4G Composite Devic System Control", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_system"

SUBSYSTEMS=="input", ATTRS{name}=="2.4G Composite Devic", ENV{LIBINPUT_IGNORE_DEVICE}="1", SYMLINK+="remote_keyboard"

For the symlink portion, keep "remote_" the same or else it will need to be changed in the script, but you can make the string after the underscore whatever you want. I chose remote_consumer, remote_system, and remote_keyboard just so I could remember which was which.

Ok so those three files can be named whatever (99-ignore-my device-whatever.rules) and they should be placed in /etc/udev/rules.d.

You can restart your system or enter the following command as root to activate the new rules:

udevadm control --reload-rules && udevadm trigger

Now, when you try and use evtest, it will be able to capture any of the three keyboards.

There are now a few modifications you have to make to the script so that it works for your setup.

change the BASE_API to point to your instance's API endpoint
from home assistant, create a long-lasting token and paste it in the API_KEY

So now, you could run the script to control one keyboard at a time and have each click send to home assistant by typing:

python3  /path_to_your_script/bt_remote_event.py %i

The argument is whatever keywords you decided to use after "remote_". For example "bt_remote_event.py keyboard" and it will run the script for the "keyboard" device.

One caveat to what we have done so far, even though the three keyboards are not being used by the system, the power button on the remote will still trigger whatever action your computer typically triggers when its power button is pressed. Therefore, I used the option in Gnome to disable the power button. This does have the consequence of even disabling the button on the PC itself, but that is not an issue for me. I am not sure if there is a way to only disable the power button only from the remote.

So now with the keyboard script running, you should be able to go into your home assistant instance, the developer's tools, events, and listen for the "bt_remote" events.

Hopefully you see your button presses and notice each press gives you two events:

event_type: bt_remote
data:
  cmd: KEY_HOMEPAGE
  cmd_type: down
  cmd_num: "172"
origin: REMOTE
time_fired: "2024-02-11T19:53:40.964943+00:00"
context:
  id: sdfgsdfgsdfgsdfgsd
  parent_id: null
  user_id: sdfgsdfgsdfgsdfgsd

event_type: bt_remote
data:
  cmd: KEY_HOMEPAGE
  cmd_type: up
  cmd_num: "172"
origin: REMOTE
time_fired: "2024-02-11T19:53:41.026000+00:00"
context:
  id: sdfgsdfgsdfgsdfgsdfgsdgsdf
  parent_id: null
  user_id: ddsfgsdfgsdfgsdfgsdfgsdfg

Each button will give you its down and then up events separately, and each even will give the key name, the button direction, and the raw key code. I added the key code because, for my setup, knowing the keycode was helpful for some subsequent actions.

Great so now that we know it works, how can we exit the running bt_remote_event script. We have to set up three scripts and make sure they start automatically. In the git repository, you should see btmon@.service. This will allow us to capture all three devices.

First, copy it to /etc/systemd/system/

sudo cp btmon@.service /etc/systemd/system/

Now you simply enable it three times for the three devices we named earlier:

sudo systemctl enable --now btmon@consumer.service
sudo systemctl enable --now btmon@system.service
sudo systemctl enable --now btmon@keyboard.service

Perfect! Now all those presses are headed straight for Home Assistant. The PC where the USB is physically plugged can be restarted and the scripts will pick up where its left off.

Solution: Represent Keypresses in Home Assistant

Ok, so now Home Assistant is getting all those key press events, but so what? I wanted a way to have a single press and a long press. The long press shouldn't set off the single press, and the single press shouldn't be more delayed than absolutely necessary.

First, I made a trigger based binary sensor to more easily display the button presses and the necessary information:

  - trigger:
      - platform: event
        event_type: bt_remote
        event_data:
          cmd_type: "down"
        id: "on"
      - platform: event
        event_type: bt_remote
        event_data:
          cmd_type: "up"
        id: "off"
    binary_sensor:
      - name: remote_button_press
        state: "{{ trigger.id }}"
        attributes:
          button: "{{ trigger.event.data.cmd }}"
          code: "{{ trigger.event.data.cmd_num }}"
          timestamp_up: >
            {% if trigger.id == 'off' %}
              {{ as_timestamp( trigger.event.time_fired)|float(default=0) }}
            {% else %}
              {{ this.attributes.timestamp_up|float(default=0) }}
            {% endif %}
          timestamp_down: >
            {% if trigger.id == 'on' %}
              {{ as_timestamp( trigger.event.time_fired)|float(default=0) }}
            {% else %}
              {{ this.attributes.timestamp_down|float(default=0) }}
            {% endif %}
          pressed_duration: >
            {% if trigger.id == 'on' %}
              0
            {% else %}
              {{ as_timestamp( trigger.event.time_fired)|float(default=0) - (this.attributes.timestamp_down)|float(default=0) }}
            {% endif %}

This creates a single binary_sensor. It is on during the time the button is pressed down, and off when it is released, so off the vast majority of the time. However, in its attributes, it has the current or last key and code pressed, but also the amount of time the key was pressed down in seconds (0 if the keys is pressed down at that moment). This has created a very easy system to make automations.

Now, as you probably guessed, to set everything up correctly, to identify each button and decide what you want it to do, it takes time. There also is not a one size fits all solution.

However, I will show examples of what I have done in case it can help anyone else.

Respond to key presses

So I essentially have two automations. One for single presses and one for double presses.

The single press is actually triggered when our binary_sensor goes off, meaning that someone has just released a button. I have chosen a hold time of 0.5 seconds to represent a button hold. Anything shorter is a single press, and longer is a hold. Therefore, for this automation, I want to make sure we only act if the button was released before the hold automation is activated.

The following is my trigger and condition:

alias: kodi remote
description: ""
trigger:
  - platform: state
    entity_id:
      - binary_sensor.remote_button_press
    from: "on"
    to: "off"
condition:
  - condition: numeric_state
    entity_id: binary_sensor.remote_long_press
    attribute: pressed_duration
    below: 0.5

Then I just have a giant choose between 30 actions. Again, it will be different for everyone, but I will show it before to highlight a couple things:

action:
  - choose:
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_POWER' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: binary_sensor.vizio_tv_on
                    state: "on"
                sequence:
                  - service: script.all_off
                    data: {}
            default:
              - if:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: power_off
                then:
                  - service: select.select_option
                    target:
                      entity_id: select.harmony_hub_2_activities
                    data:
                      option: Watch Kodi
                else:
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: PowerOn
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEUP' }}"
        sequence:
          - service: media_player.volume_up
            data: {}
            target:
              entity_id: media_player.family_room_tv
            enabled: false
          - service: script.increase_tv_volume_x1
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEDOWN' }}"
        sequence:
          - service: media_player.volume_down
            data: {}
            target:
              entity_id: media_player.family_room_tv
            enabled: false
          - service: script.decrease_tv_volume_x1
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PLAYPAUSE' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: media_player.family_room_speaker
                    state: playing
                  - condition: not
                    conditions:
                      - condition: state
                        entity_id: media_player.family_room_tv
                        state:
                          - paused
                          - playing
                sequence:
                  - service: media_player.media_play_pause
                    data: {}
                    target:
                      entity_id: media_player.family_room_speaker
            default:
              - service: media_player.media_play_pause
                data: {}
                target:
                  entity_id: media_player.family_room_speaker
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PREVIOUSSONG' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: media_player.family_room_speaker
                    state: playing
                  - condition: not
                    conditions:
                      - condition: state
                        entity_id: media_player.family_room_tv
                        state:
                          - paused
                          - playing
                sequence:
                  - service: media_player.media_previous_track
                    target:
                      entity_id: media_player.family_room_speaker
                    data: {}
            default:
              - service: media_player.media_previous_track
                target:
                  entity_id:
                    - media_player.family_room_tv
                  device_id: []
                  area_id: []
                data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_NEXTSONG' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: media_player.family_room_speaker
                    state: playing
                  - condition: not
                    conditions:
                      - condition: state
                        entity_id: media_player.family_room_tv
                        state:
                          - paused
                          - playing
                sequence:
                  - service: media_player.media_next_track
                    target:
                      entity_id: media_player.family_room_speaker
                    data: {}
            default:
              - service: media_player.media_next_track
                target:
                  entity_id:
                    - media_player.family_room_tv
                  device_id: []
                  area_id: []
                data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_CONFIG' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: Watch Kodi
                sequence:
                  - if:
                      - condition: state
                        entity_id: binary_sensor.vizio_tv_on
                        state: "off"
                    then:
                      - service: remote.send_command
                        target:
                          entity_id: remote.family_room_broadlink
                        data:
                          num_repeats: 1
                          delay_secs: 0.4
                          hold_secs: 0
                          device: Vizio TV
                          command: PowerOn
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 0
                          milliseconds: 200
                      - service: remote.send_command
                        data:
                          device: Vizio Amp
                          command: InputOptical
                        target:
                          entity_id: remote.family_room_broadlink
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 15
                          milliseconds: 0
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: InputHdmi5
            default:
              - service: select.select_option
                target:
                  entity_id: select.harmony_hub_2_activities
                data:
                  option: Watch Kodi
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_MAIL' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: Watch Roku
                sequence:
                  - if:
                      - condition: state
                        entity_id: binary_sensor.vizio_tv_on
                        state: "off"
                    then:
                      - service: remote.send_command
                        target:
                          entity_id: remote.family_room_broadlink
                        data:
                          num_repeats: 1
                          delay_secs: 0.4
                          hold_secs: 0
                          device: Vizio TV
                          command: PowerOn
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 0
                          milliseconds: 200
                      - service: remote.send_command
                        data:
                          device: Vizio Amp
                          command: InputOptical
                        target:
                          entity_id: remote.family_room_broadlink
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 15
                          milliseconds: 0
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: InputHdmi3
            default:
              - service: select.select_option
                target:
                  entity_id: select.harmony_hub_2_activities
                data:
                  option: Watch Roku
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_WWW' }}"
        sequence:
          - choose:
              - conditions:
                  - condition: state
                    entity_id: select.harmony_hub_2_activities
                    state: Watch Apple TV
                sequence:
                  - if:
                      - condition: state
                        entity_id: binary_sensor.vizio_tv_on
                        state: "off"
                    then:
                      - service: remote.send_command
                        target:
                          entity_id: remote.family_room_broadlink
                        data:
                          num_repeats: 1
                          delay_secs: 0.4
                          hold_secs: 0
                          device: Vizio TV
                          command: PowerOn
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 0
                          milliseconds: 200
                      - service: remote.send_command
                        data:
                          device: Vizio Amp
                          command: InputOptical
                        target:
                          entity_id: remote.family_room_broadlink
                      - delay:
                          hours: 0
                          minutes: 0
                          seconds: 15
                          milliseconds: 0
                  - service: remote.send_command
                    target:
                      entity_id: remote.family_room_broadlink
                    data:
                      num_repeats: 1
                      delay_secs: 0.4
                      hold_secs: 0
                      device: Vizio TV
                      command: InputHdmi2
            default:
              - service: select.select_option
                target:
                  entity_id: select.harmony_hub_2_activities
                data:
                  option: Watch Apple TV
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_BACK' }}"
        sequence:
          - service: script.kodi_remote_back
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_UP' }}"
        sequence:
          - service: script.kodi_remote_up
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_DOWN' }}"
        sequence:
          - service: script.kodi_remote_down
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_LEFT' }}"
        sequence:
          - service: script.kodi_remote_left
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_RIGHT' }}"
        sequence:
          - service: script.kodi_remote_right
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ENTER' }}"
        sequence:
          - service: script.kodi_remote_enter
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_HOMEPAGE' }}"
        sequence:
          - service: script.kodi_remote_home
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_FASTFORWARD' }}"
        sequence:
          - service: script.kodi_remote_fastforward
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_REWIND' }}"
        sequence:
          - service: script.kodi_remote_rewind
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_COMPOSE' }}"
        sequence:
          - service: script.kodi_remote_info
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_SEARCH' }}"
        sequence:
          - service: script.kodi_remote_search
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMIN' }}"
        sequence:
          - service: script.kodi_remote_toggle_subtitles
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMOUT' }}"
        sequence:
          - service: script.kodi_remote_audio_track
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEDOWN' }}"
        sequence:
          - service: script.kodi_remote_page_down
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEUP' }}"
        sequence:
          - service: script.kodi_remote_page_up
            data: {}
      - conditions:
          - condition: template
            value_template: "    {{ trigger.to_state.attributes.button.split(\"_\")[1]|length == 1 }}"
        sequence:
          - service: script.kodi_remote_letter
            data:
              key_pressed: |
                {{ trigger.to_state.attributes.button.split("_")[1]|lower }}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_BACKSPACE' }}"
        sequence:
          - service: script.kodi_remote_backspace
            data: {}
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_SPACE' }}"
        sequence:
          - service: script.kodi_remote_space
            data:
              key_pressed: 57
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_ESC' }}"
        sequence:
          - service: script.kodi_remote_gen_button
            data:
              key_pressed: 1
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_DOT' }}"
        sequence:
          - service: script.kodi_remote_gen_button
            data:
              key_pressed: 52
      - conditions:
          - condition: template
            value_template: "{{ trigger.to_state.attributes.button == 'KEY_COMMA' }}"
        sequence:
          - service: script.kodi_remote_gen_button
            data:
              key_pressed: 51

The majority of my actions involve identifying the button by name by matching to one of the binary sensor's attributes. Then I usually call for a script just to make things a little cleaner. There is some logic in this spiderweb too. It really just depends and I'm sorry I can't make this cleaner.

I want to point out one of the options though. The back of the remote has a keyboard! Wouldn't it be great to use it? I really don't want to make an option for every letter in the alphabet. Instead, I made a single option to capture anytime the key name is a single letter:

      - conditions:
          - condition: template
            value_template: "    {{ trigger.to_state.attributes.button.split(\"_\")[1]|length == 1 }}"
        sequence:
          - service: script.kodi_remote_letter
            data:
              key_pressed: |
                {{ trigger.to_state.attributes.button.split("_")[1]|lower }}

This will send the letter by itself to a special script I have which sends the letter wherever its needed depending on what device I am using. For single presses it is always a lower case letter, and as you will see, for long presses it is an upper case letter.

The long press automation is a little different. There are two situations to consider. One is a long press for something like volume. You want it to continue raising or lowering the volume as long as you are pressing it. Second is a long press which is simply a way to use the same button for a second action. In that case, we want it to perform a single action and then stop.

The automation is somewhat similarly triggered:

platform: state
entity_id:
  - binary_sensor.remote_button_press
for:
  hours: 0
  minutes: 0
  seconds: 0.5
from: "off"
to: "on"

And my action block is in a repeat action and will continue to repeat, unless stopped, as long as the button remains "on" (pressed), and will stop when it is "off" released.

repeat:
  sequence:
    - choose:
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEUP' }}"
          sequence:
            - service: script.increase_tv_volume_x1
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_VOLUMEDOWN' }}"
          sequence:
            - service: script.decrease_tv_volume_x1
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_PLAYPAUSE' }}"
          sequence:
            - choose:
                - conditions:
                    - condition: state
                      entity_id: media_player.family_room_speaker
                      state: playing
                    - condition: not
                      conditions:
                        - condition: state
                          entity_id: media_player.family_room_tv
                          state:
                            - paused
                            - playing
                  sequence:
                    - service: media_player.media_stop
                      target:
                        entity_id:
                          - media_player.family_room_speaker
                      data: {}
              default:
                - service: script.kodi_remote_stop
                  data: {}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_UP' }}"
          sequence:
            - service: script.kodi_remote_up
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_DOWN' }}"
          sequence:
            - service: script.kodi_remote_down
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_LEFT' }}"
          sequence:
            - service: script.kodi_remote_left
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_RIGHT' }}"
          sequence:
            - service: script.kodi_remote_right
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_FASTFORWARD' }}"
          sequence:
            - service: script.kodi_remote_fastforward
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_REWIND' }}"
          sequence:
            - service: script.kodi_remote_rewind
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMIN' }}"
          sequence:
            - service: script.kodi_remote_switch_subtitles
              data: {}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_ZOOMOUT' }}"
          sequence:
            - service: media_player.volume_mute
              target:
                entity_id: media_player.family_room_tv
              data:
                is_volume_muted: false
            - stop: ""
        - conditions:
            - condition: template
              value_template: "    {{ trigger.to_state.attributes.button.split(\"_\")[1]|length == 1 }}"
          sequence:
            - service: script.kodi_remote_letter
              data:
                key_pressed: |
                  {{ trigger.to_state.attributes.button.split("_")[1] }}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_COMPOSE' }}"
          sequence:
            - service: script.kodi_remote_menu
              data: {}
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_BACKSPACE' }}"
          sequence:
            - choose:
                - conditions:
                    - condition: state
                      entity_id: sensor.active_player
                      state: media_player.apple_tv
                  sequence:
                    - service: shell_command.pyatv_root
                      data:
                        command: text_clear
                    - stop: ""
              default:
                - service: script.kodi_remote_gen_button
                  data:
                    key_pressed: 14
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEDOWN' }}"
          sequence:
            - service: script.kodi_remote_page_down
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_PAGEUP' }}"
          sequence:
            - service: script.kodi_remote_page_up
              data: {}
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_WWW' }}"
          sequence:
            - choose:
                - conditions:
                    - condition: state
                      entity_id: select.harmony_hub_2_activities
                      state: Play Nintendo
                  sequence:
                    - if:
                        - condition: state
                          entity_id: binary_sensor.vizio_tv_on
                          state: "off"
                      then:
                        - service: remote.send_command
                          target:
                            entity_id: remote.family_room_broadlink
                          data:
                            num_repeats: 1
                            delay_secs: 0.4
                            hold_secs: 0
                            device: Vizio TV
                            command: PowerOn
                        - delay:
                            hours: 0
                            minutes: 0
                            seconds: 15
                            milliseconds: 0
                    - service: remote.send_command
                      target:
                        entity_id: remote.family_room_broadlink
                      data:
                        num_repeats: 1
                        delay_secs: 0.4
                        hold_secs: 0
                        device: Vizio TV
                        command: InputHdmi4
              default:
                - service: select.select_option
                  target:
                    entity_id: select.harmony_hub_2_activities
                  data:
                    option: Play Nintendo
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_CONFIG' }}"
          sequence:
            - service: automation.trigger
              metadata: {}
              data:
                skip_condition: true
              target:
                entity_id: automation.turn_on_family_room_speaker
            - stop: ""
        - conditions:
            - condition: template
              value_template: "{{ trigger.to_state.attributes.button == 'KEY_ENTER' }}"
          sequence:
            - service: script.kodi_remote_return
              data: {}
            - stop: ""
    - delay:
        hours: 0
        minutes: 0
        seconds: 0
        milliseconds: 100
  while:
    - condition: state
      entity_id: binary_sensor.remote_long_press
      state: "on"

I added a short delay in the loop to make sure the button presses weren't being sent too quickly. You can see also that some of the actions are followed by a "stop" action when they are intended to run as a one-and-done action. Otherwise, the action will continue to repeat until the button is released.

Summary

So, at this point, we have a 2.4ghz remote. The button presses are ignored by the host system and sent to home assistant as events. Home assistant then distinguished between long and short presses and enacts whatever action is necessary. Obviously, these buttons don't have to only act on my media center. The buttons can control anything in the house.

My setup basically allows the remote to work as intended based on what is playing on TV. That way, nobody ever has to switch inputs or profiles or whatever. It just works. I have noticed that my family gravitates towards it even instead of the harmony remotes. I have been very pleased with it so far which is why I wanted to share.

In the future, I can give examples of how I send the commands (including keyboard presses) seemlessly to kodi, roku, or apple tv. Those (especially apple tv) were tricky to figure out.

On a different kodi box, I have a separate remote set up and just have it send the keypresses back. That way, only the keys I want to modify are modified by home assistant while the remote buttons otherwise work for the kodi box just like they would without all this complicated work around.

I'm sure there are many more possibilities and would be excited to see what other people come up with.

Links

Air Mouse Remote MX3 Pro with Backlit

Associate Link - https://amzn.to/487vXMU

Store Link - https://www.amazon.com/gp/product/B08YRL56BT

Rii MX3 Multifunction

Associate Link - https://amzn.to/3OGErnr

Store Link - https://www.amazon.com/gp/product/B01CL3ZXGO

Create a FREE public endpoint for self-hosted services using Oracle Cloud and OPNsense

thehelpfulidiot — Thu, 15 Sep 2022 00:00:00 +0000

Introduction

As many of you (the people who would be interested enough to read this post), I have a number of self hosted services which have become integral to my everyday life. For the most part, I use a wireguard (split) tunnel to access these services remotely from my phone or laptop. However, there are some services, like Nextcloud, for which I prefer public access so that I can access them from any computer. Up till now, I have opened port 443 on my router and forwarded incoming traffic to an instance of Nginx Proxy Manager. This has been working fine for the most part, but I have a residential ISP with a dynamic IP address. Even though I have my incoming connections go through my OPNsense router with crowdsec and geo block lists, I just never liked the idea of opening ports on my router.

Original Network Layout

To solve this, I became enamored with the idea of setting up a public IP address on a VPS which would send the connection requests to my NPM instance through a wireguard tunnel.

There are plenty of tutorials how to forward connections through Wireguard, and there is an amazing tool to do almost exactly what I described. However, the tool requires a linux endpoint on the local network as opposed to sending all connections to OPNsense. My ideal plan would have the VPS connections first passing through OPNsense where they could be acted upon as any other incoming connection.

New Network Layout

So ultimately, at the end of this tutorial, we will have a static IP address through a free oracle cloud virtual machine. It will forward whichever connections we want to OPNsense. I'm sure this could be configured as a VPN endpoint for all connections if desired, but that won't be covered here.

Oracle Cloud

Ok, so first thing we need to do is set up our virtual machine on the Oracle Cloud platform.

https://www.oracle.com/cloud/

Oracle has a free tier and "Always Free Resources". Obviously, they could always take these free resources as many cloud providers eventually do, but lets hope if it happens, it doesn't happen for a while.

Always free VM

So first step, create an account.

Instead of going straight to making a new compute instance, I had to make a "compartment". I think that's how the compute instances are clustered. To be honest, I am not sure that I actually needed to make a new compartment but had trouble getting my machine set up without it.

From the top left hamburger menu, go to Compartments.

From there, create a new compartment. Name it whatever you want, and for parent compartment, select your root.

Next click on the big Oracle Cloud image top left to go back to the main page. Now we will set up our compute instance.

Click Instances

Click the "Create Instsance" button.

On the next page, we will set up our machine and make sure to select options which are part of the "Always Free" subscription.

Under "create in compartment," be sure to select the compartment we just set up.

Keep the defaults for Placement.

Now for Image & Shape, we will change some options. For the purposes of this tutorial, I will use the Ubuntu image. It's free and it's what I am familiar with.

Under Shape Series, select "Ampere" and let's max out our machine specs under the Free Tier.

Free tier specs

Click select shape and proceed. Keep networking options as default.

Be sure to save your SSH key! It will be the only way to log in to your machine. Then leave the boot volume as default and we are done setting up our machine!

Now back at the instances screen, you should see your new instance along with its public IP address and its internal IP address. We will use the public IP address to SSH in, but first lets adjust the firewall settings.

Click on your instance name. Under "Primary VNIC", then subnet, there should be a hyperlink to the subnet your machine is using. Click on it. Scroll down and click on the default security list. I erased all of the default firewall rules and replaced them with a rule to allow all traffic as we can just rely on ufw to protect our machine.

Open Ingress Rules

Ok perfect, now lets get into our machine. We are going to use the Public IP address to SSH in to our server and get to work. Find your Private SSH key and add it to your ~/.ssh directory. Set the permissions for the key to 600. Now you can access your server with the following:

ssh -i /ssh/directory/private.key ubuntu@PUBLIC-IP

Install Wireguard with the following:

sudo apt update
sudo apt install wireguard

Now create a public and private wireguard key.

wg genkey | sudo tee /etc/wireguard/private.key
sudo chmod go= /etc/wireguard/private.key
sudo cat /etc/wireguard/private.key | wg pubkey | sudo tee /etc/wireguard/public.key

Make note of those keys as we will need them. Now we can start setting up our Wireguard server configurtion.

sudo nano /etc/wireguard/wg0.conf

And use the following:

[Interface]
PrivateKey = <strong>{{private key you just generated}}</strong>
ListenPort = 51820
<strong>#(any port above here)</strong>
Address = 10.1.0.1/24
<strong>#(an address from a different subnet from your private LAN)</strong>
MTU = 1420

PostUp = iptables -A FORWARD -i enp0s3 -j ACCEPT; iptables -t nat -A POSTROUTING -o wg0
PostDown = iptables -D FORWARD -i enp0s3 -j ACCEPT; iptables -t nat -A POSTROUTING -o wg0
<strong>#enp0s3 is whatever your virtual machines main interface is called</strong>

[Peer]
PublicKey = <strong>{{ we do not have this yet }}</strong>
AllowedIPs = 10.1.0.0/24, <strong>{{address of npm/32}}</strong>
PersistentKeepalive = 25

Now let's make sure we have forwarding set up in our Oracle machine. We are going to edit sysctl.conf. First open it.

sudo nano /etc/sysctl.conf

Add the following two lines:

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Then enter the following command or reboot:

sudo sysctl -p

Almost ready to move over to the OPNsense side of things. Let's just set up our firewall before we forget.

We will use ufw which is included in the ubuntu image.

Add ssh port:

sudo ufw allow 22/tcp

Https port:

sudo ufw allow 443/tcp

Wireguard port (change as necessary):

sudo ufw allow 51820/udp

Then enable the firewall:

sudo ufw enable

And check the status to make sure we are secured:

sudo ufw status

and we should see something like:

Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
51820/udp                  ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere

Ok perfect. Let's change our attention to OPNsense.

OPNsense

This part only requires a few things but took me DAYS of neglecting my family and responsibilities to figure it out. I was pointed in the right direction here. Like I said, I already have a Wireguard server set up on my OPNsense box, and was trying to incorporate this new client in that network. That proved to be impossible- at least for me. Then it struck me- I would just make a NEW wireguard network on the OPNsense machine, and this one would act as a client instead of a server as I had initially thought it would.

So if you haven't already, install either the wireguard-go or wireguard-kmod package on your OPNsense box.

Go to the Wireguard menu and then the Endpoints tab. Create a new endpoint and fill it out like so:

Wireguard OPNsense Endpoint

Now switch to the Local tab and create a new entry like so:

OPNsense local configuration

Make sure advanced options are checked. Call it whatever you want, set an MTU (I think?). Enter what this client's IP address will be. Since our VPS is 10.1.0.1, I chose 10.1.0.2 for this client, and make sure to use the /24 subnet. Select your previously made endpoint under peers. MAKE SURE TO CLICK DISABLE ROUTES. Otherwise when you initiate the connection, the routing tables will be edited to try and send all traffic through the Wireguard network. Then make sure to enter the VPS Wireguard IP address for the Gateway.

There is no private or public key but after you click save and reopen the entry, you should see that populate. Take the Public Key, go back to our wg0.conf from the VPS and add the public key as indicated earlier under the "[Peer]" section.

Now, on our VPS, you can bring up the wg0 interface by entering the following:

sudo wg-quick up wg0

You can also set it to start at boot and enable with:

sudo systemctl enable --now wg-quick@wg0

Back to OPNsense...

Turn Wireguard on and off from OPNsense. Now lets give it an interface so we can more easily manipulate the connection and create firewall rules. Go to Interface --> Assignments. There should be an option at the bottom to add a new interface and one of the interfaces from the dropdown menu should be the new Wireguard connection (wg0) and call it whatever you want.

After refreshing the page, the new VPS/Wireguard interface should be listed in the Interfaces section.

Click on it, make sure it is enabled, and click Promiscuous mode, save.

Ok almost done. Now we'll just go down the line and add the settings necessary for our VPS to act as the external gateway we want.

New gateway

Change the settings to match what I have above. The IP address should be the IP address of the Wireguard interface on the VPS.

Next, the connection we receive on port 443, will be forwarded to the OPNsense router on that same interface. We will go to the Port Forwarding tab and create a rule to forward that connection to wherever our NPM instance is. However, aside from the usual port forwarding rule, we are going to make two changes.

Add something to "Set local tag". I just put "oracle". It doesn't matter what you pick.
For "Filter Rule Association", change it to "none".

Now we will move on to the firewall rules for our new VPS/Wireguard interface. Currently, without any rules, all traffic will be blocked. I only have two rules:

Allow 443 traffic to the NPM instance.
Allow communication between clients on our interface subnet
Geo-block IPs if desired

VPS/Wireguard interface firewall rules

I also included the same "set local tag" directive with "oracle" to both of those above Pass rules. Not sure if that is necessary, but I figured it wouldn't hurt. Basically, I want anything that comes in through this interface to leave through it.

Finally, we will tell the router how to send the packets back out through the VPS as opposed to leaving through our regular gateway.

Go to the interface where your NPM instance is. Make a new firewall rule. This time, however, instead of choosing "in" for the direction, choose "out". Then we can leave everything alone and just make a few changes under the "Advanced" section.

Firewall "out" rule

There are three things to do here.

select allow options
change the reply-to gateway to the gateway we just created
match local tag to whatever you used in the previous rules. Mine is "oracle".

You can be more specific if you want, but the traffic we are interested in is already tagged so I think the firewall rule is likely fine how it is. Now the firewall will know to reply back to our VPS and not through Verizon or Comcast or whatever.

At this point, I would probably restart the VPS and then take the Wireguard service down and then up to make sure that all our new rules are working and activated.

Conclusion

If everything worked the way it should, you should now be able to set the DNS records for your web services to point to the public IP address of your VPS. Supposedly, this will remain free forever but of course that remains to be seen. Not sure if this really makes things safer or more secure than just opening port 443 of your firewall really, but I feel like it can't hurt? If nothing else, I have noticed a significant improvement with the initial web connection when accessing my home services. I suppose that these residential connections probably aren't designed to accommodate incoming web requests but this also may be placebo.

If you run into trouble with any of this, the linked OPNsense and Oracle Cloud docs are the best next stop.

Making an Automatic Email Backup - Part 3

thehelpfulidiot — Wed, 22 Jun 2022 00:00:00 +0000

Part 1

Part 2

Initially, I described a solution which required the use of LXC or some full Linux box to install and operate the necessary tools. Recently, however, I repackaged those tools into a couple docker containers, more easily deployable through a docker compose file with configuration done with environment variables.

https://github.com/jon6fingrs/mbsync-dovecot

https://hub.docker.com/repository/docker/thehelpfulidiot/dovecot

https://hub.docker.com/repository/docker/thehelpfulidiot/mbsync

Instructions on the github page. Let me know if you have any questions!

Integrating XMCOSY+ Outdoor String Lights with Home Assistant Locally

thehelpfulidiot — Fri, 17 Jun 2022 00:00:00 +0000

XMCOSY+ Outdoor String Lights

Home automation is incredible and obviously in my opinion, a lot of fun. However, I'm sure many of you would agree that buying a product for use in your home shouldn't have to connect to an outside computer, let alone one in another country.

Fortunately with the magic of Home Assistant and the Local Tuya plugin, I have managed to run this light locally without any loss of functionality. As you may know, Tuya is a Chinese manufacturer who creates a lot of white label IoT devices. The devices, in my experience, tend to work well, but require cloud access to work through official channels. Local Tuya, on the other hand, is a custom integration which can grant local access to these Tuya devices and can work very well for our purposes.

Unfortunately, Local Tuya, while set up to handle many different types of devices out of the box, is not able to handle the unique modes and functions of the XMCOSY+ Outdoor String Lights. However, by at least allowing access to the device API, templates and automations can be used to simplify and restore the device's normal function locally. In this post, I hope to explain my method for solving this problem and that it will work for others.

Steps

Set the device up in Local Tuya as a light
Create a template light entity to control the light seamlessly
Use Node Red to convert between hex color code and HSV
Create entities to change the custom colors related to one of the built-in scenes
(optional) Block light access to internet

Local Tuya

https://github.com/rospogrigio/localtuya

The documentation is pretty good at explaining how to get this custom integration installed, and then how to add your devices. Once you get your string lights added, you will have to instruct the plugin as to which data point (dp) controls which function of the light.

Here is a printout of the data points for the light from the command line tool from which local tuya is based:

{
  dps: {
    '20': false,
    '21': 'white',
    '22': 255,
    '23': 165,
    '24': '23eb230159ff38',
    '101': 'ffff0502808080ffff00'
  }
}

The explanation for each of these is as follows:

20: on/off

21: scene (white, colour, scene1, scene2, ..., scene8*, scene9)

*scene 8 is the only customizable scene allowing you to select two colors

22: brightness 0-255

23: white-temp on a bizarre scale of 0 - 255

24: the light color and when in color mode, its brightness (very complicated will explain below)

101: a string storing the customizable two colors for scene8 (explained below)

Explanation of DP24

Now DP 24. This took a really really long time to figure out and I found some helpful hints on decoding Tuya colors deep in the internet and I'm sorry I can't seem to find these sites again or else I would like to provide links and give them credit.

This is a combination of the lights hex code AND HSV (encoded in hex).

So the first 6 characters (in my example above) - 23eb23 is a simple hex code for the current color.

The 7th character is a 0. That is just a divider.

The next 7 characters are related to HSV, but each item is hex-encoded.

The first three charaters are Hue from 0-360 (ex above 159)

The following 2 characters are saturation on a scale of 0-255 (the standard is 0-100)

And the last 2 characters are Value (brightness) which is also on a scale 0-255 (and in this case the standard is 0-100)

Explanation of DP101

It seems to consist of a constant string in the beginning (example above ffff0502). It may mean something but I was not able to change it through any of the app functions and I was too scared to change it manually. If anyone knows what this might represent, please tell me.

But after that, are two back to back hex color codes. In the example above, color 1 is 808080, and color two is ffff00. I made two input selectors with a bunch of standard common color names. There is an automation and each time one of those two entities changes, the updated hex color code is inserted in the appropriate place.

The color combination is updated and selecting scene8 shows you the custom scene you just made.

Back to Local Tuya

So based on that, we will make a few entities using Local Tuya, all based around this device.

First, of course, is the light itself. You can match up the DPs as follows:

LocalTuya Light Configuration

This sets up a light entity, but like I said, it is very limited. Cannot change between scenes, adjust scene colors, etc.

Next, we need to set up some sensors from LocalTuya so that we can easily access the data for use in a template light.

sensor.string_lights_scene

sensor.string_light_color

sensor.string_light_color_value

sensor.custom_colors

So after all this, you should have five newly create LocalTuya entities:

light.colored_string_lights - the main light entity
sensor.custom_colors - a sensor displaying the hex code for the two custom colors used for scene8
sensor.string_light_color - brightness (don't ask me why I called it this)
sensor.string_light_color_value - the complicated hex and hsv color/brightness string
sensor.string_lights_scene - the current scene

Now it's time to move on to templates.

Templates

OK so now you can control the lights, but wouldn't it be nice to package everything in a single light entity? Fortunately, we can make template entities to combine the sensors and the light entity we created with LocalTuya.

  - platform: template
    lights:
      string_lights:
        level_template: >
          {% if state_attr('light.colored_string_lights','color_mode') == 'color_temp' %}
            {{ states('sensor.string_light_color')|int }}
          {% elif state_attr('light.colored_string_lights','color_mode') == 'hs' %}
            {{ states('sensor.string_light_color_value')[12:]|int(base=16) }}
          {% else %}
            255
          {% endif %}
        value_template: "{{ states('light.colored_string_lights') }}"
        temperature_template: >
          {% if state_attr('light.colored_string_lights','color_mode') == 'color_temp' %}
            {{ state_attr('light.colored_string_lights','color_temp') }}
          {% elif state_attr('light.colored_string_lights','color_mode') == 'hs' %}
            {{ color_temp|int(default=0) }}
          {% endif %}
        color_template: >
          {% if state_attr('light.colored_string_lights','color_mode') == 'hs' %}
            {{ state_attr('light.colored_string_lights','hs_color') }}
          {% endif %}
        effect_list_template: "{{['White','Color','Strobe','Fade','Pulse','Blink','Chase','Christmas','America','Pick 2','Multi']}}"
        effect_template: >
          {% if states('sensor.string_lights_scene') == 'scene1' %}
            {{ "Strobe" }}
          {% elif states('sensor.string_lights_scene') == 'scene2' %}
            {{ "Fade" }}
          {% elif states('sensor.string_lights_scene') == 'scene3' %}
            {{ "Pulse" }}
          {% elif states('sensor.string_lights_scene') == 'scene4' %}
            {{ "Blink" }}
          {% elif states('sensor.string_lights_scene') == 'scene5' %}
            {{ "Chase" }}
          {% elif states('sensor.string_lights_scene') == 'scene6' %}
            {{ "Christmas" }}
          {% elif states('sensor.string_lights_scene') == 'scene7' %}
            {{ "America" }}
          {% elif states('sensor.string_lights_scene') == 'scene8' %}
            {{ "Pick 2" }}
          {% elif states('sensor.string_lights_scene') == 'scene9' %}
            {{ "Multi" }}
          {% elif states('sensor.string_lights_scene') == 'white' %}
            {{ "White" }}
          {% elif states('sensor.string_lights_scene') == 'colour' %}
            {{ "Color" }}
          {% endif %}
        min_mireds_template: "{{ state_attr('light.colored_string_lights','min_mireds') }}"
        max_mireds_template: "{{ state_attr('light.colored_string_lights','max_mireds') }}"
        turn_on:
          service: light.turn_on
          target:
            entity_id: light.colored_string_lights
        turn_off:
          service: light.turn_off
          target:
            entity_id: light.colored_string_lights
        set_level:
          - service: script.brightness_with_white_or_color
            data:
              brightness: '{{ brightness|int }}'
        set_color:
          - service: script.check_color_mode
            data:
              h: '{{ hs[0] }}'
              s: '{{ hs[1] }}'
        set_temperature:
          - service: script.check_white_mode
            data:
              color_temp: '{{color_temp|int }}'
        set_effect:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 21
              value: >
                {% if effect == 'Strobe' %}
                  {{ "scene1" }}
                {% elif effect == 'Fade' %}
                  {{ "scene2" }}
                {% elif effect == 'Pulse' %}
                  {{ "scene3" }}
                {% elif effect == 'Blink' %}
                  {{ "scene4" }}
                {% elif effect == 'Chase' %}
                  {{ "scene5" }}
                {% elif effect == 'Christmas' %}
                  {{ "scene6" }}
                {% elif effect == 'America' %}
                  {{ "scene7" }}
                {% elif effect == 'Pick 2' %}
                  {{ "scene8" }}
                {% elif effect == 'Multi' %}
                  {{ "scene9" }}
                {% elif effect == 'White' %}
                  {{ "white" }}
                {% elif effect == 'Color' %}
                  {{ "colour" }}
                {% endif %}

alias: check white mode
sequence:
  - if:
      - condition: not
        conditions:
          - condition: state
            entity_id: sensor.string_lights_scene
            state: white
    then:
      - service: localtuya.set_dp
        data:
          device_id: eb3f15638dccc2fe59wdgj
          dp: 21
          value: white
  - service: light.turn_on
    data:
      color_temp: '{{ color_temp }}'
    entity_id: light.colored_string_lights
mode: single

alias: check color mode
sequence:
  - if:
      - condition: not
        conditions:
          - condition: state
            entity_id: sensor.string_lights_scene
            state: colour
    then:
      - service: localtuya.set_dp
        data:
          device_id: eb3f15638dccc2fe59wdgj
          dp: 21
          value: colour
      - wait_for_trigger:
          - platform: state
            entity_id:
              - sensor.string_lights_scene
            to: colour
        timeout: '1'
  - service: light.turn_on
    data:
      hs_color:
        - '{{ h }}'
        - '{{ s }}'
    entity_id: light.colored_string_lights

One issue I noticed is that the brightness value is only relevant when the light is in white mode. When the light is in a color mode, the brightness is part of the hex/hsv string. This template selects the correct brightness based on the mode. When the light is showing a scene other than color or white, brightness is always at 100%. I believe that is a limitation of the light.

Anyway, to make sure the correct brightness value is modified when you modify the brightness from the light entity in home assistant, I offloaded that to a script seen below:

alias: brightness with white or color
sequence:
  - choose:
      - conditions:
          - condition: template
            value_template: >-
              {{ state_attr('light.colored_string_lights','color_mode') ==
              'color_temp' }}
        sequence:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 22
              value: '{{brightness|int}}'
    default:
      - service: rest_command.color_converter
        data:
          brightness: '{{brightness|int}}'
      - service: localtuya.set_dp
        data:
          device_id: eb3f15638dccc2fe59wdgj
          dp: 24
          value: >
            {{ states('sensor.color_output') }}{{
            states('sensor.string_light_color_value')[6:12] }}{{ "%0x" | format(
            brightness | int ) }}
mode: single

Another issue was converting between HSV and hex colors. I could not find an easy formula or function to accomplish this in Home Assistant. After much searching, I found a module to use in Node Red.

Color and Brightness

This is important for the DP24 which controls color and brightness when in single color mode. Basically, a hex value and hex encoded HSV is given each time. Just changing the hex value would change the color but reset the lights back to 100%. Apparently you need to change both and they need to match? Converting with template got complicated and I could not figure out a way to solve it. After much searching, I came across a module in Node Red.

I am not a Node Red power user. I do very few things with it, and it feels like whenever I use it, I have to relearn it.

So to start, I needed a way to get information from home assistant to Node Red. I used a rest command in Home Assistant to send information to an HTTP receiver in Node Red:

  change_color_not_brightness:
    url: http://nodered:1880/color-converter
    method: POST
    payload: '{"hsv":{"hue":{{ h }},"saturation":{{ s }},"brightness":{{ (state_attr(''light.string_lights'',''brightness'')/2.55)|int}}}}'
    content_type:  'application/json; charset=utf-8'
    verify_ssl: false

This sends over the current color and brightness in HSV.

Node Red

Now to switch gears to Node Red. For those who don't know, it is a very commonly used add-on for Home Assistant, for creating automations by visually programming in Java. Many people use it exclusively for automations and don't touch the built-in YAML-based automations. Aside from the ease of creating automations, it also has access to a number of add-on modules which can add additional functions to your palette.

The one we will need is called node-red-contrib-colorspace. It allowed me to create a "Flow" which received the HSV value through HTTP request, convert it to rgb, and send it back to Home Assistant. I send it back by using the Home Assistant nodes to update a sensor called "sensor.color_output".

sensor.color_output

My flow looks like this:

Node-red Flow

Final Product

Ok, so now we have a fully functional entity called "light.string_lights". It can control (almost) every light function as one would expect from a regular Home Assistant light entity.

white light

Color Select

scene select

It can control brightness, color, and the scenes. It can change brightness for color or white seamlessly. But there's one more thing- scene8.

scene8

Scene 8 is one of 9 scenes, and is the only customizable one. It alternates two colors which can be changed independently. I made two select entities for them. I just chose a handful of colors and created a dropdown menu although it would be possible to choose any color as well.

Color Select

Whenever these entities are changed, they trigger an automation to change the color of the light:

alias: Change string light colors
description: ''
trigger:
  - platform: state
    entity_id: input_select.color_1
    id: color1
  - platform: state
    entity_id: input_select.color_2
    id: color2
condition: []
action:
  - choose:
      - conditions:
          - condition: trigger
            id: color1
        sequence:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 101
              value: >
                {% if states('input_select.color_1') == 'Black' %}
                  {% set new1 = '000000' %}
                {% elif states('input_select.color_1') == 'Silver' %}
                  {% set new1 = 'c0c0c0' %}
                {% elif states('input_select.color_1') == 'Gray' %}
                  {% set new1 = '808080' %}
                {% elif states('input_select.color_1') == 'White' %}
                  {% set new1 = 'ffffff' %}
                {% elif states('input_select.color_1') == 'Maroon' %}
                  {% set new1 = '800000' %}
                {% elif states('input_select.color_1') == 'Red' %}
                  {% set new1 = 'ff0000' %}
                {% elif states('input_select.color_1') == 'Purple' %}
                  {% set new1 = '800080' %}
                {% elif states('input_select.color_1') == 'Fuchsia' %}
                  {% set new1 = 'ff00ff' %}
                {% elif states('input_select.color_1') == 'Green' %}
                  {% set new1 = '008000' %}
                {% elif states('input_select.color_1') == 'Lime' %}
                  {% set new1 = '00ff00' %}
                {% elif states('input_select.color_1') == 'Olive' %}
                  {% set new1 = '808000' %}
                {% elif states('input_select.color_1') == 'Yellow' %}
                  {% set new1 = 'ffff00' %}
                {% elif states('input_select.color_1') == 'Navy' %}
                  {% set new1 = '000080' %}
                {% elif states('input_select.color_1') == 'Blue' %}
                  {% set new1 = '0000ff' %}
                {% elif states('input_select.color_1') == 'Teal' %}
                  {% set new1 = '008080' %}
                {% elif states('input_select.color_1') == 'Aqua' %}
                  {% set new1 = '00ffff' %}
                {% endif %}

                {{ 'ffff0502' }}{{new1}}{{
                states('sensor.custom_colors').split('ffff0502')[1][6:] }}
      - conditions:
          - condition: trigger
            id: color2
        sequence:
          - service: localtuya.set_dp
            data:
              device_id: eb3f15638dccc2fe59wdgj
              dp: 101
              value: >
                {% if states('input_select.color_2') == 'Black' %}
                  {% set new2 = '000000' %}
                {% elif states('input_select.color_2') == 'Silver' %}
                  {% set new2 = 'c0c0c0' %}
                {% elif states('input_select.color_2') == 'Gray' %}
                  {% set new2 = '808080' %}
                {% elif states('input_select.color_2') == 'White' %}
                  {% set new2 = 'ffffff' %}
                {% elif states('input_select.color_2') == 'Maroon' %}
                  {% set new2 = '800000' %}
                {% elif states('input_select.color_2') == 'Red' %}
                  {% set new2 = 'ff0000' %}
                {% elif states('input_select.color_2') == 'Purple' %}
                  {% set new2 = '800080' %}
                {% elif states('input_select.color_2') == 'Fuchsia' %}
                  {% set new2 = 'ff00ff' %}
                {% elif states('input_select.color_2') == 'Green' %}
                  {% set new2 = '008000' %}
                {% elif states('input_select.color_2') == 'Lime' %}
                  {% set new2 = '00ff00' %}
                {% elif states('input_select.color_2') == 'Olive' %}
                  {% set new2 = '808000' %}
                {% elif states('input_select.color_2') == 'Yellow' %}
                  {% set new2 = 'ffff00' %}
                {% elif states('input_select.color_2') == 'Navy' %}
                  {% set new2 = '000080' %}
                {% elif states('input_select.color_2') == 'Blue' %}
                  {% set new2 = '0000ff' %}
                {% elif states('input_select.color_2') == 'Teal' %}
                  {% set new2 = '008080' %}
                {% elif states('input_select.color_2') == 'Aqua' %}
                  {% set new2 = '00ffff' %}
                {% endif %}

                {{ 'ffff0502' }}{{
                states('sensor.custom_colors').split('ffff0502')[1][:6] }}{{
                new2 }}
    default: []
mode: single

To confirm the light color was changed to the desired color, I also made two sensors to read the scene's custom colors from the light itself.

- sensor:
      - name: "Custom Color 2"
        state: >
          {% if states('sensor.custom_colors').split('ffff0502')[1][6:] == '000000' %}
            {{ 'Black' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'c0c0c0' %}
            {{ 'Silver' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '808080' %}
            {{ 'Gray' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ffffff' %}
            {{ 'White' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '800000' %}
            {{ 'Maroon' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ff0000' %}
            {{ 'Red' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '800080' %}
            {{ 'Purple' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ff00ff' %}
            {{ 'Fuchsia' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '008000' %}
            {{ 'Green' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '00ff00' %}
            {{ 'Lime' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '808000' %}
            {{ 'Olive' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == 'ffff00' %}
            {{ 'Yellow' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '000080' %}
            {{ 'Navy' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '0000ff' %}
            {{ 'Blue' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '008080' %}
            {{ 'Teal' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][6:] == '00ffff' %}
            {{ 'Aqua' }}
          {% else %}
            {{ states('sensor.custom_colors').split('ffff0502')[1][6:] }}
          {% endif %}
- sensor:
      - name: "Custom Color 1"
        state: >
          {% if states('sensor.custom_colors').split('ffff0502')[1][:6] == '000000' %}
            {{ 'Black' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'c0c0c0' %}
            {{ 'Silver' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '808080' %}
            {{ 'Gray' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ffffff' %}
            {{ 'White' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '800000' %}
            {{ 'Maroon' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ff0000' %}
            {{ 'Red' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '800080' %}
            {{ 'Purple' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ff00ff' %}
            {{ 'Fuchsia' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '008000' %}
            {{ 'Green' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '00ff00' %}
            {{ 'Lime' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '808000' %}
            {{ 'Olive' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == 'ffff00' %}
            {{ 'Yellow' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '000080' %}
            {{ 'Navy' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '0000ff' %}
            {{ 'Blue' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '008080' %}
            {{ 'Teal' }}
          {% elif states('sensor.custom_colors').split('ffff0502')[1][:6] == '00ffff' %}
            {{ 'Aqua' }}
          {% else %}
            {{ states('sensor.custom_colors').split('ffff0502')[1][:6] }}
          {% endif %}

That produces these two entities:

Color Select Entities

Light Picker

Finally

Our light is finished and fully controllable locally. The next step is to block the light's internet access from your router.

Completed Light

Hope this helps some people! Let me know if you have any questions!

Link to Purchase

Affiliate Link:

XMCOSY+ Outdoor String Lights, 49Ft

Non-Affiliate Link:

XMCOSY+ Outdoor String Lights, 49Ft

Custom 3D Printed Mount for Fire Tablet (2015)

thehelpfulidiot — Fri, 26 Nov 2021 00:00:00 +0000

I have been enamored by the slew of Home Assistant Dashboard mounts lately and have wanted to make one for a while. I saw that the newer Amazon tablets have commercially available options, but my old 2015 Fire Tablet was on its own.

I decided to make one using my Creality Ender 3 Pro. I designed one with screw holes to fit a standard 2-gang box, an opening for the cord to exit posteriorly, and cut outs for the tablet's buttons.

Printing

Unfortunately, each one took about a day to print and I went through several drafts, slowly tweaking the design to fit perfectly. This slow process probably took weeks, but I finally came up with something that seemed to work.

Finished Frame

Cut outs for buttons and cord

Tablet in the frame- tight fit

Attaching a tablet to a wall is not hard, but doing it in a way to hide the cords is definitely more challenging. I know that the standard way is to run power to a new gang box which is why I designed my tablet frame to fit that scenario. However, I realized that I could probably do something a little simpler.

I just ran the USB cable behind the wall and made a seperate opening towards the floor so that at least, the cable would be hidden and not dangling from the tablet.

Future tablet location

Attached frame

Cord exit

And here is the final result:

Final product

In case anyone is interested, here is a copy of my STL file:

fire-tablet-mount

And finally, here is the cord I used and the brush wall plate:

Affiliate Link:

GELRHONR Micro USB Cable U Shape,180 Degree Angle USB 5 Pin Nylon Braided Fast Data Sync Charging Cord USB A to USB B Fast Charger Wire for Most Micro USB Devices - Black 4.9Ft/1.5M

Single Brush Wall Plate 2-Pack,Single Gang faceplate Cable Pass Through Insert Wall Socket for HDTV, HDMI, Network, Coaxial Cables, Speaker Wires White

Non-affiliate Link:

GELRHONR Micro USB Cable U Shape,180 Degree Angle USB 5 Pin Nylon Braided Fast Data Sync Charging Cord USB A to USB B Fast Charger Wire for Most Micro USB Devices - Black 4.9Ft/1.5M

Single Brush Wall Plate 2-Pack,Single Gang faceplate Cable Pass Through Insert Wall Socket for HDTV, HDMI, Network, Coaxial Cables, Speaker Wires White

Let me know if you have any questions. Thanks!

UPDATE 2: A Wired Sonoff Zigbee Alternative

thehelpfulidiot — Mon, 18 Oct 2021 00:00:00 +0000

ORIGINAL POST

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

The updates for this little device keep coming. Thanks to the folks leaving comments, we keep getting more information. I apologize for not noticing this myself, but I did not spend any time trying Zigbee2Tasmota on this device having played with it enough on the Sonoff hub.

A commenter, Ronald, noted that using Zigbee2Tasmota, attempting to pair Zigbee devices caused the hub to reboot. I confirmed this myself, and tried with both Zigbee firmware packages from the EmberZNet NCP UART EZSP firmware page.

I attempted to build a firmware using the development branch of Tasmota instead of the Release branch, but it totally bricked my coordinator. After re-soldering and flashing again, I tried the unofficial Tasmota32-zigbeebridge.bin mentioned on the Blakadder Template Repo. At the time of this writing, the Tasmota version is 9.5.0.9. Miraculously, I was able to pair devices as expected.

Device allowed to join

After flashing this firmware, you would set it up following the instructions from the initial post by changing the template and then disabling WiFi if desired.

To use ZHA, I would still recommend you follow the instructions from the first update and building a custom firmware so that the necessary rules to self-correct following a cold boot will work.

Hopefully, whatever fixes exist in the nightly build will make their way to the stable branch and a single firmware will be able to accomplish ZHA and Zigbee2Tasmota.

Please keep me updated with any issues or questions. Thanks!

ORIGINAL POST

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

**UPDATE 10/30: **The above issue was acknowledged on Github and apparently had something to do with the serial communication in the Tasmota firmware. As noted above, that is why the unofficial zigbeebridge firmware works. It containts a fix for the serial communication.

UPDATE: A Wired Sonoff Zigbee Alternative

thehelpfulidiot — Fri, 15 Oct 2021 00:00:00 +0000

ORIGINAL POST

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

So while the documentation in the initial post on how to compile and flash Tasmota to the ZB-GW03 works, there is a bug when using the Tasmotized device with Home Assistant.

After changing the device to a ZHA Bridge, a power disruption will prevent a reconnection to Home Assistant once power is restored. The only way to reconnect appears to be by switching the device template back to Zigbee2Tasmota, letting it start, and then back to the ZHA Bridge template. Afterwards it will reconnect as usual. A software initiated reboot DOES NOT cause the same issue.

I do not know enough to say exactly why this is the case, but my theory is that the Zigbee2Tasmota template initializes the Zigbee Radio when it boots up and maybe that is not happening with Home Assistant when the device is shut down in an unsafe state.

Fortunately, I have developed a work around which appears to work well and essentially fixes the problem without further intervention.

It requires two rules:

Turning on after an unsafe shutdown changes the device back to Zigbee2Tasmota
Once the device boots into Zigbee2Tasmota and the Zigbee radio is fully initialized, the device switches back to ZHA bridge.

In my somewhat brief testing, this has worked really well and the device reconnects as expected after an unsafe shutdown without any user intervention.

Unfortunately, to accomplish this, requires a bit more work than one would think. Tasmota only saves one user defined Module at a time, and I was not able to save a rule with template information, probably due to the fact that it is in JSON format. Therefore, I had to compile a new Tasmota firmware, with the two file modifications on the initial instruction page, plus two new modules, one for each template.

tasmota_template

To compile it yourself, you just need to replace tasmota_template.h with the following. Towards the end are the modules for ESP32 devices and you can see two additional modules for the ZB-GW03, one for ZHA and one for Zigbee2Tasmota.

Module 2 - ZHA_Bridge
Module 3 - Zigbee2Tasmota

If you prefer, you can just download the updated firmware below:

tasmota32-EWELINK-updated

Once you have the new firmware installed, you can use the following rules:

backlog rule1 on system#boot do backlog wifi 0; TCPStart 8888 endon; rule1 on
backlog rule2 on Info3#RestartReason=Vbat power on reset do module 3 endon; rule2 on
backlog rule3 on ZbState#Message=Started do module 2 endon; rule3 on

The first rule combines rules 1 and 2 from the initial post and sets up the Zigbee Web Server and turns off wifi. If you intend to use wifi (which I probably wouldn't recommend), just use the following instead for the first rule:

backlog rule1 on system#boot do TCPStart 8888 endon; rule1 on

Rule2 changes the module to Zigbee2Tasmota ONLY when the power is cut and the device undergoes an unsafe shutdown.

Rule3 as soon as the device boots to Zigbee2Tasmota and the Zigbee radio is fully initialized, it switched back to ZHA Radio at which point the device should be able to reconnect to Home Assistant as expected.

An issue has been opened on Github so please reply if you have experienced the same problem:

https://github.com/arendst/Tasmota/issues/13350

Please let me know about any additional issues or if you have any problems with the above solution.

ORIGINAL POST

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

Making an Automatic Email Backup - Part 2

thehelpfulidiot — Sat, 09 Oct 2021 00:00:00 +0000

UPDATE: PART 3 has instructions for email backup using pre-made docker images based on this post and Part 1.

<--- Part 1

Summary so far

Ok, so if you have gone through Part 1 already, you have Mbsync set up to sync all emails from your remote IMAP server and then Dovecot set up to host those emails on a local SMTP server, but without SSL.

That may be enough for many, but we can take things a step further. In this part, we will add self-signed SSL certificates for Dovecot, and we will install Roundcube. Roundcube is an open sourced webmail client. We will configure it to access our local email server and then to send emails through our remote server.

Now one thing I would highly highly recommend to get the most out of this setup is to install Wireguard on your network and to set up a split-tunnel on your devices so that wherever you are, you always have access to all your local services. There are likely many of these tutorials on the internet and the instructions may vary a bit with the exact Wireguard instance you decide to use. Once you have it set up though, you will have access to all home hosted services - Pihole is always blocking ads for me on my cell phone, even on mobile, I can access router settings, Proxmox, whatever.

In this case, I can also access my "local" IMAP server from anywhere in the world without exposing the IMAP server to the wide scary internet. Of course, if you wanted, I'm sure you could also expose whatever services you want using a reverse proxy, but to me that just seems like a little less secure.

Installing SSL

So in my home, I have created a root Certificate Authority and have many server certificates for the various services on my network, mostly used with Nginx Reverse Proxy. There are several ways to do this, but if you have Pfsense or OPNsense, it comes with a way to create and manage these certificates using a graphical format. Below is a link to the OPNsense website detailing the creation of these certificates and a great video by Allen Sampsell.

OPNsense documentation

Youtube video by Allen Sampsell

If you don't have OPNsense or PFsense, this is the website I previously followed. Everything is done by the command line so a little more work, but with good instructions:

Self-signed SSL certificate command line instructions

Now, pick a local DNS name for your IMAP server. For example, imap.thehelpfulidiot.lan. Use whatever router software you have to create a local DNS entry pointing to your email backup server. Once you have the server name, you can create the Server Certificate for the server.

By now you should have three things:

Certificate Authority Certificate
IMAP Server Certificate
IMAP Server Key

SSH into your email backup server.

Copy over your server certificate and keys. Go to the Dovecot directory and make a directory to store SSL certificates.

cd /etc/dovecot
sudo mkdir ssl
cd ssl

sudo nano email.pem

If your certificate has a "*.crt" extension, you will need to convert it to "*.pem" with the following command.

openssl x509 -in email.crt -out email.pem -outform PEM

In the new file that opens, copy and paste the contents of your server certificate file.

Ctrl+X, Y, enter.

Next, do the same with your key file.

sudo nano email.key

Paste the contents of your key file.

Ctrl+X, Y, enter.

Now your certificate and key files are at the following locations, respectively:

/etc/dovecot/ssl/email.pem
/etc/dovecot/ssl/email.key

Ok, so now we have to tell Dovecot where the files are and what its server name is.

cd /etc/dovecot/conf.d

sudo nano 10-auth.conf

Remove the line "disable_plaintext_auth = no" we placed in part 1 so that the default to not accept plaintext authentication will be reinstated.

Now let's edit the SSL file.

sudo nano 10-ssl.conf

Change "ssl = yes" to "ssl = required" towards the top.

Then you should see "ssl_cert" and "ssl_key" entries near the top as well. Change those to the appropriate file locations, but do not remove the "<" before each file path as this tells Dovecot to read the contents of the files and not just the file names.

ssl_cert = </etc/dovecot/ssl/email.pem
ssl_key = </etc/dovecot/ssl/email.key

Finally, we have to tell Dovecot what its sever name is so it knows to trust connections to it:

sudo nano /usr/share/dovecot/dovecot-openssl.cnf

Change the following line from:

commonName = @commonName@

to:

commonName = imap.thehelpfulidiot.lan

Now, assuming you have installed your root certificate on your devices, connecting to this IMAP server should be like connecting to any server.

Installing Roundcube

Next, wouldn't it be cool to be able to access this email from a nice Web GUI? Roundcube is a great way to do that and is something else which can be hosted internally. Even better, it comes as a Docker image.

I use Portainer to easily spin up docker compose files and recommend it to anyone for simple Docker management.

Create a new stack called "Roundcube" and use the following docker compose file, edited as necessary for your setup:

version: '2'

services:
  roundcubemail:
    image: roundcube/roundcubemail:latest
    container_name: roundcubemail
    restart: unless-stopped
    volumes:
      - /path/to/roundcube/www:/var/www/html
      - /path/to/roundcube/ssl:/var/ssl
    ports:
      - 9002:80
    depends_on:
      - "roundcubedb"
    environment:
      - ROUNDCUBEMAIL_DB_TYPE=pgsql
      - ROUNDCUBEMAIL_DB_HOST=roundcubedb # same as pgsql container name
      - ROUNDCUBEMAIL_DB_NAME=roundcube # same as pgsql POSTGRES_DB env name
      - ROUNDCUBEMAIL_DB_USER=roundcube # same as pgsql POSTGRES_USER env name
      - ROUNDCUBEMAIL_DB_PASSWORD=roundcube # same as pgsql POSTGRES_PASSWORD env name
      - ROUNDCUBEMAIL_DB_PORT=5432
      - ROUNDCUBEMAIL_SKIN=elastic
      - ROUNDCUBEMAIL_DEFAULT_HOST=ssl://<localserver>
      - ROUNDCUBEMAIL_DEFAULT_PORT=993
      - ROUNDCUBEMAIL_SMTP_SERVER=tls://<remoteserver>
      - ROUNDCUBEMAIL_SMTP_PORT=587
     
  roundcubedb:
    image: postgres:latest
    container_name: roundcubedb
    restart: unless-stopped
    volumes:
      - /path/to/db/postgres:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=roundcube
      - POSTGRES_USER=roundcube
      - POSTGRES_PASSWORD=roundcube

Save your root Certificate Authority to the SSL folder you have mapped for Roundcube.

Spin that up, but don't worry about logging in just yet. Go to the folder you used for '/path/to/www'.

There should now be a bunch of files and folders generated by Roundcube's first run.

cd config

Edit config.inc.php

sudo nano config.inc.php

you should see the following:

<?php
    $config['plugins'] = [];
    $config['log_driver'] = 'stdout';
    $config['zipdownload_selection'] = true;
    $config['des_key'] = 'some_key';
    include(__DIR__ . '/config.docker.inc.php');

Just below that, add the following:

    $config['enable_caching'] = false;
    $config['imap_conn_options'] = array(
      'ssl'         => array(
         'verify_peer'  => true,
         'verify_depth' => 3,
         'cafile'       => '/var/ssl/rootCA.pem',
         'allow_self_signed' => true,
         'verify_peer_name' => true,
       ),
     );
     $config['smtp_user'] = 'remote_user';
     $config['smtp_pass'] = 'remote_pass';

Ctrl+x, y, enter.

Almost done.

Now log into Roundcube using the credentials for your Linux username and password from your email backup container.

Roundcube Login Screen

Go to Settings --> Identities --> and select the User which is probably something like "thehelpfulidiot@imap.thehelpfulidiot.lan". If we send an email from roundcube, we don't want that to be the identity the recipients see.

Change the Display Name and Email to your actual name and email addresses.

Closing

That's it! Now you can log in to Roundcube as if you were using a commercial email service. You can view your emails neatly organized and send emails directly from your remote email address!

If you were so inclined, you could set up a reverse proxy and put Roundcube behind it and maybe something like Authelia and you could access your email inbox from anywhere. Personally, I'll leave something as sensitive as email safe behind my Wireguard server though.

Please let me know if you have any questions!

Making an Automatic Email Backup - Part 1

thehelpfulidiot — Mon, 04 Oct 2021 00:00:00 +0000

UPDATE: PART 3 has instructions for email backup using pre-made docker images based on this post and Part 2.

Intro

In the world of self-hosting, creating a personal email server always seemed like the Holy Grail of personal privacy. Unfortunately, with all the work that would go into maintaining it, and the high probability of failure when operated by someone as inexperienced as myself, it has never seemed like a viable option for me. However, I have wanted a system for some time whereby I can automatically backup my email in a standard format which could potentially be served to me through standard mail apps by hosting my own internal IMAP server.

After doing a fair amount of research, unfortunately, I could not find any suitable ready-made Docker solutions to accomplish this. However, I was able to create something fairly simple and elegant.

I am going to share my solution which, while it took me a lot of work, should be fairly easy to set up.

It requires three parts:

mbsync - a command line utility to synchronize IMAP servers
dovecot - IMAP server software
an email client

A major benefit of this set up could be for people with limited space in their email mailboxes. Instead of deleting emails to remain under 15gigs, you could essentially backup all emails to your home server, delete emails on the remote server, and still have easy, ready access to everything.

Initial Set Up

I have done this on Proxmox by creating a Ubuntu 20.04 LXC. I am using iSCSI for storage so that while the LXC is being run on my Proxmox machine, the data is being saved on my NAS where it has RAID protection, snapshots, and backups.

I will go through the process assuming Ubuntu LXC, but I imagine this should work with any Ubuntu installation.

I made my LXC with 2 cores and 1GB of RAM. Something I think that's important, is to make a non-root user with sudo privileges. This username and password will be the username and password for your IMAP server so you may want to make it relatively secure. To do so, enter the following commands:

sudo adduser <username>
sudo usermod -aG sudo <username>

Then you can sign out of the root username and log in to the LXC using your new credentials.

Afterwards, update the LXC to the latest with

sudo apt update && sudo apt upgrade -y

Mbsync

Mbsync is a very powerful IMAP syncing software. It downloads your IMAP emails into a Maildir format in which each email is its own file and the file structure of your mailbox disk directory mimics the mailbox itself. It is a command line tool, but was fortunately fairly easy to work with. It can do a lot and I'm sure I barely scratched the surface, but for my purposes, I set it up to download all new emails from the server, but to not propagate any deletions, and to not propagate any changes from my local instances back. This is a one way trip, but allowing deletions of this archive would defeat a major purpose of creating it.

Confusingly, to install mbsync, you have to use its legacy name, isync.

sudo apt install isync

Then it's important to make the configuration file or else the program will not run.

sudo nano ~/.mbsyncrc

and populate accordingly. Below is the template I use with a short explanation. There are many other configuration options if you are interested in further customizing it.

<strong># Set up according to your email provider. It will probably involve creating an application password.</strong>
IMAPAccount <email account>
Host <host name>
Port <port>
User <username>
Pass <password>
<strong># select one</strong>
SSLType < None | STARTTLS | IMAPS >
<strong># select one</strong>
SSLVersions < SSLv3 | TLSv1 | TLSv1.1 | TLSv1.2 | TLSv1.3 > 

<strong># I use fastmail and so my settings will reflect that but change as needed.
# I believe IMAPStore is relatively arbitrary and has to do with how settings are stored. <something>-remote appears to be convention.</strong>
IMAPStore fastmail-remote
<strong># pick a name</strong>
Account fastmail

<strong># This section describes the local storage
# pick a name similar to what you picked for IMAPStore</strong>
MaildirStore fastmail-local

<strong>#The root of your mailbox. By convention the Maildir directory is in the user's home folder.</strong>
Path /home/thehelpfulidiot/Maildir/
<strong>#The inbox seems to be a special folder in a mailbox so you need to specify it explicitly and then the other folders will be auto-created.</strong>
Inbox /home/thehelpfulidiot/Maildir/Inbox/
<strong># Adds additional subfolders to match IMAP folder names</strong>
SubFolders Verbatim

<strong># A channel defines how the sync will occur between the server and local storage.
# pick a name</strong>
Channel fastmail
<strong># use what you have for IMAPStore above between two :'s</strong>
Master :fastmail-remote:
<strong># use what you have for MaildirStore above between two :'s</strong>
Slave :fastmail-local:
<strong># You can control what gets synced. '*' syncs everything.</strong>
Patterns *
<strong># Do not remove messages marked for deletion</strong>
Expunge None
<strong># Do not remove deleted mailboxes</strong>
Remove None
<strong># use the arrival date and time from the IMAP server for the original message</strong>
CopyArrivalDate yes
<strong># You can see all the different ways to sync the mailboxes by going to the link above, but this will sync from remote to local without propagating deletions.</strong>
Sync PullNew
<strong># automatically create necessary mailboxes locally</strong>
Create Slave
<strong>#keeps the mailbox sync state files in the local mailbox</strong>
SyncState *

Press "ctrl+x", Y to save, enter.

Now make the mail directory you specified a moment ago. If using the same directory as I specified in the example config file, from your home directory, enter:

mkdir Maildir

Then run mbsync for the first time to download the emails.

mbsync -a

This will take a while if your email folder is like mine. Take a break and come back when it's done. If you set up your config file like mine, mbsync will recreate all your mail folders locally.

I got a socket error during the first sync, but luckily, mbsync can pick up from wherever it leaves off, so just run 'mbsync -a' as many times as you need to get the whole mailbox synced initially.

So now that your current mailbox is fully synced with your local archive, we need to set up a mechanism to pull new messages. As in all things, the archwiki page on mbsync has an excellent example of how to do this.

First, create the systemd unit. I think that placing it in the system directory and adding your specific User to the unit file will ensure that the process runs on boot without login, but as your user.

sudo nano /etc/systemd/system/mbsync.service

And paste this text, making sure to change the user:

[Unit]
Description=Mailbox synchronization service

[Service]
Type=oneshot
User=<user>
ExecStart=/usr/bin/mbsync -Va

Then we need to make a timer file so that this Unit is called periodically.

sudo nano /etc/systemd/system/mbsync.timer

And enter the following:

[Unit]
Description=Mailbox synchronization timer

[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=mbsync.service

[Install]
WantedBy=timers.target

Now activate the new files.

sudo systemctl daemon-reload
sudo systemctl enable mbsync.timer
sudo systemctl start mbsync.timer

Great, so now we have successfully backed up our IMAP account into a Maildir directory and we have set up automatic fetching. For some, this might be adequate, but the next thing I wanted to do was to make sure I had an easy way to consume the mail. Ideally, I wanted it to be viewable in any mail app. Next, we have to install Dovecot which is an IMAP server.

Dovecot

Dovecot is another really powerful piece of software used at super high levels, and I don't presume to be an expert with it, but I will show how it can be used in this situation as a local IMAP server.

So first, let's get it installed:

sudo apt install dovecot-imapd

Now we need to make some changes to the configuration.

sudo nano /etc/dovecot/conf.d/10-mail.conf

Locate the following line:

mail_location = mbox:~/mail:INBOX=/var/mail/%u

Comment it out and replace it with:

mail_location=maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX

Save and close the file.

Next, open 10-auth.conf.

Replace

 #disable_plaintext_auth = yes

with

disable_plaintext_auth = no

This will ensure you can connect to your server without SSL, but if you don't want that option. You can skip it and keep plaintext authentication disabled.

Now open 15-mailboxes.conf.

Under "namespace inbox {", add "inbox = yes" like so:

namespace inbox {
  inbox = yes

Go down farther and uncomment the blocks about the virtual "all" and "flagged" mailboxes so that it looks like this:

  # If you have a virtual "All messages" mailbox:
  mailbox virtual/All {
    special_use = \All
  #  comment = All my messages
  }

  # If you have a virtual "Flagged" mailbox:
  mailbox virtual/Flagged {
    special_use = \Flagged
  #  comment = All my flagged messages
  }

Start Dovecot.

sudo systemctl enable dovecot
sudo systemctl start dovecot

Dovecot is ready and your IMAP server should be functioning on your internal network, but without SSL. Since this is on your local network, that may or may not be adequate for you.

Viewing

Now if you use a mail app, you should be able to enter the IP address of your Mail Backup LXC in the server name and your Linux account credentials and your email will populate like any other server. For example, on my iPhone, I entered the following settings:

Home SMTP settings

You can basically make the name, email, and descriptions what you would normally use for your remote email server. It will not be able to authenticate of course and you will then have an opportunity to create separate username, passwords, and server identities for IMAP and SMTP. The IMAP host name is your LXC's IP address, the user name is your Linux username, and the password is your Linux password.

The SMTP configuration would be according to your remote server's SMTP configuration to ensure you can send emails as ususal.

You'll notice that attempting to connect from an iPhone (and I suspect from other systems) gives this warning:

SSL error

I will continue in a second part on how to take things a bit further. I will show you how to install self-signed certificates, and how to set up a local webmail application called Roundcube to view your backed-up mailbox and send emails through your remote server.

Conclusion

I hope that helps some folks. It took me quite a while figuring out these settings, but now that I have, I think its pretty straightforward. Please feel free to let me know if you have any questions or if I got something wrong.

Thanks for reading!

Continue to Part 2 --->

Controlling Home Assistant with an Old Remote

thehelpfulidiot — Sun, 03 Oct 2021 00:00:00 +0000

I, like many other I'm sure, have plenty of old infrared remotes laying around from TVs past. They were just collecting dust, but what if they could be useful again? Well thanks to the Flirc USB, a small USB IR receiver, those old remotes can be used to control anything your Home Assistant Box can control.

The Flirc USB

Banana for scale

The amazing thing about this device is its simplicity. First, you plug it into a PC, Mac, or Linux machine. Then download the software. The software lets you assign buttons to a virtual keyboard by selecting individual keys and firing your remote at it. Those codes are saved on the device itself, so it can be moved to your Home Assistant Operating System box and will be seen as just an ordinary keyboard. Except, instead of interacting by physically pressing keys, the key is "pressed" when it detects the corresponding infrared frequency.

Set Up

So let's walk through how this would be set up.

First, download the software from the website and plug in the USB device.

Flirc Software

There are a number of virtual keyboard or remotes you can select. I think selecting a standard keyboard makes the most sense to maximize the number of programmable keys.

Next, click on a key, and it will ask you to press the corresponding button on the remote. You do not need to remember which buttons are mapped to which keys because they will later be identified in Home Assistant by watching the event log.

Once you have recorded all the keys you want, you unplug the device from your computer and plug it into your Home Assistant box.

Home Assistant

Now the Flirc can be set up in Home Assistant using the Keyboard Remote integration. First, confirm it is recognized by Home Assistant by looking in the Supervisor's Hardware tab.

Flirc recognized in Hardware tab

Once you see it's recognized, add the following to your configuration.yaml and restart:

keyboard_remote:
  - device_name: 'flirc.tv flirc Keyboard'
    type: 'key_down'

Creating Automations:

Now you can make automations capable of anything your Home Assistant can do from the press of a button. To do so, you need to obtain the "key_code" for each programmed button. Go to the Developer Tools and then the Events tab. Listen for events from "keyboard_remote_command_received" and press the individual buttons. For example:

Example of keyboard_remote_command_received response

So now I know that my remote's "Netflix" button is mapped to keycode 50.

A corresponding automation trigger would look like this:

platform: event
event_type: keyboard_remote_command_received
event_data:
  device_name: flirc.tv flirc Keyboard
  key_code: 50

And from there it can do whatever you want- turn off lights, lock doors, turn on music, or even turn on your TV using your old remote!

Logitech Harmony

Now, I happen to be a big fan of the Harmony Remote and its Home Assistant integration. As far as I know, the integration is one way, meaning that you can fire commands through the Harmony but Home Assistant cannot receive commands. If instead of an old remote you wanted to use the Harmony, you could simply set up some dummy device and make sure the the Harmony Hub or an IR mini blaster will have line of sight to the Flirc Receiver. Set up the device in the same way and then you can integrate Home Assistant actions into your Harmony Hub Activity.

BONUS - MagiQuest Wands

Some of you may be familiar with MagiQuest. It is an interactive live role playing game at many tourist attractions. My family and I, specifically, encountered it at Great Wolf Lodge. To participate, you purchase (expensive) customized wands and can run around the hotel or tourist destination waving it at different objects.

MagiQuest Wand

Well the wand is actually just an infrared transmitter which is fired when the wand is flicked. Instead of bringing them home and having life-less souvenirs, you can integrate them into Home Assistant using the Flirc!

Conclusion

I hope this write-up was helpful. For only $19.95 on Amazon, I definitely think it is a good device at a good price with a lot of possibility. Please let me know if you have any questions.

Affiliate Link

FLIRC USB Universal Remote Control Receiver

Non-Affiliate Link

FLIRC USB Universal Remote Control Receiver

How to Use Zigbee2Tasmota with Home Assistant

thehelpfulidiot — Sat, 02 Oct 2021 00:00:00 +0000

To follow up my last post, integrating a newly flashed Tasmota Zigbee Coordinator with Home Assistant has a third option besides ZHA and Zigbee2MQTT; it can be used directly as a Zigbee Coordinator using its own amazing Tasmota Software.

The device itself uses a zigbee implementation called Zigbee2Tasmota, which is a full-fledged zigbee routing platform with the usual MQTT integration. Unfortunately, it is probably overlooked because its use is limited to the Tasmota console, but it may be an attractive option for some, especially for a potentially more reliable connection when relying on the WiFi radio of the Sonoff Zigbee Coordinator.

In this case, the Pros and Cons may have some subjectivity, but I will share why I was interested in using it, and the drawbacks associated with it.

First, as I stated in my first post, when I began my self-hosting journey seriously by purchasing a machine with the intention of running ProxMox, I did what I'm sure everyone did. I put EVERYTHING I possibly could onto this single machine. That included Pihole (my dns server), Untangled (my firewall), Home Assistant, Mosquitto, EVERYTHING. I thought it was so cool to have a single machine running so many discrete tasks so elegantly through ProxMox, but I found that a host reboot was disastrous and caused a very long downtime in my house. Everything went down and the entire boot process could take up to 10 minutes. This had an extremely negative impact on my WAF.

To solve this, I began splitting up my various services and really only have my plex server, my backup pihole, home assistant plus many other non-critical services on my ProxMox box now. I got a number of cheap Zotac mini-pcs from Amazon Warehouse and have begun using them to run baremetal services like OPNsense, zigbee/zwave radios, NVR, MQTT server, etc.

Back to my point, I did not initially enjoy turning my Tasmotized Zigbee Coordinator into a Home Assistant Zombie. When you integrate the Zigbee coordinator into ZHA, you lose all native Zigbee commands of the coordinator.

In an effort to further decentralize my smart home infrastructure, I would prefer for the zigbee coordinator to maintain functioning even if my Home Assistant instance goes down. Having said that, the device now becomes dependent on a functioning MQTT server so that is certainly a trade-off. I personally keep my MQTT server running on my NAS so that it is always up. I also have a second instance of Home Assistant which meant that I could only integrate the coordinator natively with one instance, whereas when the connection is over MQTT, it can connect to an unlimited number of Home Assistant Servers. I also think an MQTT server is more stable than a WiFi serial over TCP connection. I subjectively and anecdotally noticed a quicker reaction time with Zigbee2Tasmota vs ZHA.

Second, based on my testing and internet research, it is really not possible to change the Zigbee channel from 11 when integrating the Zigbee Coordinator in ZHA. I have a Hue Hub and WiFi channels 1, 6, and 11 throughout my home and so for me, the ideal channel was 20. With Zigbee2Tasmota, this is a trivial command.

zbconfig {"Channel":<enter channel>}

Third, I purchased a number of cheap Sonoff and Aqara sensors without realizing the issues I would have when trying to pair them with standard Zigbee coordinators. I had many drop offs requiring a physical reset on the device when on ZHA, but the devices all just seemed to work with Zigbee2Tasmota. Now, this could be related to my second point and while on ZHA, there just may have been too much interference, but changing the channels of my other devices was not a viable option for me.

But it isn't all positive. There are certainly plenty of downsides with Zigbee2Tasmota.

First, all commands are done through the console except Zigbee Join and viewing the Zigbee Map, as you can see below. I suppose that negative depends on the individual, but it is certainly a nice option to have an easy to use GUI nicely integrated into Home Assistant. However, I should note once devices are added, they are listed on the main Tasmota page along with their current state, connection strength, battery level, etc.

It is definitely nice for a quick glance or to troubleshoot.

Second, along with all Zigbee network controls requiring commands through the Tasmota terminal, adding devices into Home Assistant also required manually configured MQTT entities and Template entities. Again, may or may not be a big deal. I will show some examples of my devices below. Honestly, once one device was written, it was trivial to copy and paste an additional device.

Third, while sensors worked quickly and flawlessly, lights were a bit of a different story. You can only send ONE command at a time. So changing the hue/color and brightness requires two separate commands. In Home Assistant, this required me to get a little creative with a template light ON TOP of the manually configured MQTT light. Also, the light updates in Home Assistant were a little slow since it required possibly sending multiple commands to the hub and then receiving a state message back to home assistant to update the entity. This probably wasn't a huge deal, but as you can imagine, my testing was artificial and involved a lot of stress testing which is probably beyond real world usage.

Summary Comparison

Zigbee2Tasmota	ZHA
Functions independently from Home Assistant	Nice and user-friendly GUI
Easily change Zigbee channel	Automatic device/entity creation after pairing
Good compatibility with nontraditional Zigbee protocols	Better light bulb compatibility and control

Helpful Zigbee2Tasmota Options

Ok, so you've read about Zigbee2Tasmota so far and you want to give it a try. There were a few global Tasmota options I found very helpful in simplifying the administration of the network.

By default, every Zigbee device is communicated over MQTT using the same topic, tele/%topic%/SENSOR.

By setting SetOption89 to 1, each device is given their own topic, tele/%topic%/<device>/SENSOR. I found this much easier to keep the devices separated.

Each device is given a unique identifier when it joins the network, and that is used to initially populate the MQTT topic if you decided to make the change I suggested above, but there is fortunately a way to give friendly names to each device.

ZBName <device>,<new_friendly_name>

Once you do that, the device friendly name will display on the Tasmota home screen, and you can set SetOption112 to 1 so that the friendly name will be used in the MQTT topic instead of the unique identifier.

Ok, so now that we've done that, our Zigbee coordinator should be good to go. We have it connected to our MQTT broker and have tweaked the settings enough to hopefully add some devices.

Example Home Assistant Entities

Here are some examples I made to control various devices. Like I said above, the MQTT topic for each device will now be tele/Zigbee/<friendly_device_name>/SENSOR. I set my topic to zbbridge in Tasmota, so change yours accordingly along with the device names. I did my best to bold the device-specific names and options below.

Motion detector:

  - platform: mqtt
    name: <strong>master bath motion</strong>
    state_topic: "tele/<strong>zbbridge</strong>/<strong>master_bath_motion</strong>/SENSOR"
    value_template: "{{ value_json['ZbReceived']['<strong>master_bath_motion</strong>']['Occupancy'] }}"
    payload_on: 1
    payload_off: 0
    availability_topic: "tele/<strong>zbbridge</strong>/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "motion"

Contact sensor:

  - platform: mqtt
    name: <strong>half bath door sensor</strong>
    state_topic: "tele/<strong>zbbridge</strong>/<strong>half_bath_door</strong>/SENSOR"
    value_template: "{{ value_json['ZbReceived']['<strong>half_bath_door</strong>']['ZoneStatusChange'] }}"
    payload_on: 1
    payload_off: 0
    availability_topic: "tele/<strong>zbbridge</strong>/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "door"

Battery sensor:

This was a little trickier. A battery percentage is not sent with each device update message. I added a template to the MQTT entity to say, "if message has updated value, use that, otherwise, keep old value." Without that addition, each time an updated message came without the specific value, the entity in Home Assistant became undefined.

  - platform: mqtt
    name: <strong>half bath door battery</strong>
    state_topic: "tele/<strong>zbbridge</strong>/<strong>half_bath_door</strong>/SENSOR"
    value_template: >
      {% if value_json['ZbReceived']['<strong>half_bath_door</strong>']['BatteryPercentage'] > 0 %}
        {{ value_json['ZbReceived']['<strong>half_bath_door</strong>']['BatteryPercentage'] }}
      {% else %}
        {{ states(entity_id) }}
      {% endif %}
    availability_topic: "tele/<strong>zbbridge</strong>/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "battery"
    expire_after: 86400
    unit_of_measurement: '%'

Humidity and Temperature Sensors:

The same is true for the Aqara Humidity and Temperature Sensor I use. Each message did not have ever variable and so it was important to only update the MQTT entity when the message had the necessary information. Also, to note, the temperature was reported in Celsius to Tasmota and so I had to add a freedom conversion to my MQTT entity for Fahrenheit.

  - platform: mqtt
    name: <strong>refrigerator temperature sensor</strong>
    state_topic: "tele/<strong>zbbridge</strong>/<strong>refrigerator_temperature_sensor</strong>/SENSOR"
    value_template: >
      {% if value_json['ZbReceived']['<strong>refrigerator_temperature_sensor</strong>']['Temperature'] > 0 %}
        {{ (value_json['ZbReceived']['<strong>refrigerator_temperature_sensor</strong>']['Temperature']|float + 1.8) + 32 }}
      {% else %}
        {{ states(entity_id) }}
      {% endif %}
    availability_topic: "tele/<strong>zbbridge</strong>/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "temperature"
    expire_after: 86400
    unit_of_measurement: 'F'

  - platform: mqtt
    name: <strong>refrigerator humidity sensor</strong>
    state_topic: "tele/<strong>zbbridge</strong>/<strong>refrigerator_temperature_sensor</strong>/SENSOR"
    value_template: >
      {% if value_json['ZbReceived']['<strong>refrigerator_temperature_sensor</strong>']['Humidity'] > 0 %}
        {{ value_json['ZbReceived']['<strong>refrigerator_temperature_sensor</strong>']['Humidity'] }}
      {% else %}
        {{ states(entity_id) }}
      {% endif %}
    availability_topic: "tele/<strong>zbbridge</strong>/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    qos: 0
    device_class: "humidity"
    expire_after: 86400
    unit_of_measurement: '%'

Lights:

Ok, so far things have been relatively straightforward. Here is where they get tricky. I am in no way saying that this is the easiest or best way to accomplish light control. This is what I did which seemed to work. I did not attach any colored lights (thank god as I can only imagine that would add to the complication), but I did attach a number of white tunable lights.

Like I said above, the lights can only receive a single command at a time. Luckily, it is possible to group lights and send 1 command for multiple lights. To do so, use this following command:

ZbSend {"device":"IKEA_Light","Send":{"AddGroup":100}}

Replace the device with either the friendly name or unique identifier. The group number is a unique, arbitrary integer for each group. As far as I know, it is not possible to name the groups from within Tasmota and their status is not displayed on the front page.

First I will show how I handled a single light. It required both an mqtt entity and a template entity. The MQTT entity basically brought the device into home assistant, and the template entity made its control simpler by allowing me to change color temp and brightness at the same time by breaking down the single home assistant command into two separate MQTT commands. The MQTT device, for me, basically just sat there, hidden, and the Template light is how I interacted with the light.

Single Light:

   - platform: template
     lights:
       guest_bedroom_lamp_template:
         level_template: >
           {% if state_attr('light.guest_bedroom_lamp','brightness')|int >0 %}
             {{ state_attr('light.guest_bedroom_lamp','brightness') }}
           {% else %}
             0
           {% endif %}
         value_template: "{{ states('light.guest_bedroom_lamp') }}"
         temperature_template: >
           {% if state_attr('light.guest_bedroom_lamp','color_temp')|int >0 %}
             {{ state_attr('light.guest_bedroom_lamp','color_temp') }}
           {% else %}
             0
           {% endif %}
         turn_on:
           - service: light.turn_on
             data_template:
               entity_id: light.guest_bedroom_lamp
               brightness: "{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}"
           - service: light.turn_on
             data_template:
               entity_id: light.guest_bedroom_lamp
               color_temp: "{{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }}"
         turn_off:
           - service: light.turn_off
             data_template:
               entity_id: light.guest_bedroom_lamp
         set_level:
         - service: light.turn_on
           data_template:
             brightness: "{{ brightness|int }}"
             entity_id: light.guest_bedroom_lamp
         set_temperature:
         - service: script.change_brightness_and_color_temp
           data_template:
             light_ct: light.guest_bedroom_lamp
             brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
         - service: light.turn_on
           data_template:
             color_temp: "{{ color_temp|int }}"
             entity_id: light.guest_bedroom_lamp
         max_mireds_template: 370
         min_mireds_template: 153

   - platform: mqtt
     schema: template
     name: "Guest Bedroom Lamp"
     command_topic: "cmnd/zbbridge/ZbSend"
     state_topic: "tele/zbbridge/guest_room_lamp/SENSOR"
     command_on_template: >
       {"device": "guest_room_lamp","send":{
       {%- if brightness is defined -%}
       "Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness }}{%- endif -%}
       {%- elif color_temp is defined -%}
       "CT": {{ color_temp }}
       {%- else -%}
       "Power": 1
       {%- endif -%}
       }}
     command_off_template: '{"device": "guest_room_lamp","send":{"Power":0}}'
     state_template: >
       {% if value_json['ZbReceived']['guest_room_lamp']['Power'] == 1 %}
         on
       {% elif value_json['ZbReceived']['guest_room_lamp']['Power'] == 0 %}
         off
       {% else  %}
         {{ states('light.guest_bedroom_lamp') }}
       {% endif %}
     brightness_template: >
       {% if value_json['ZbReceived']['guest_room_lamp']['Dimmer'] is defined %}
         {% if value_json['ZbReceived']['guest_room_lamp']['Dimmer'] == 255 %}
           254
         {% else %}
           {{ value_json['ZbReceived']['guest_room_lamp']['Dimmer'] }}
         {% endif %}
       {% else %}
         {{ state_attr('light.guest_bedroom_lamp','brightness') }}
       {% endif %} 
     color_temp_template: >
       {% if value_json['ZbReceived']['guest_room_lamp']['CT'] is defined %}
         {% if value_json['ZbReceived']['guest_room_lamp']['CT'] > 370 %}
           370
         {% elif value_json['ZbReceived']['guest_room_lamp']['CT'] < 153 %}
           153
         {% else %}
           {{ value_json['ZbReceived']['guest_room_lamp']['CT'] }}
         {% endif %}
       {% else %}
         {{ state_attr('light.guest_bedroom_lamp','color_temp') }}
       {% endif %} 
     availability_topic: "tele/zbbridge/LWT"
     payload_available: "Online"
     payload_not_available: "Offline"
     optimistic: true
     qos: 0
     max_mireds: 370
     min_mireds: 153

So basically, Guest Bedroom Lamp is the mqtt light, and Guest bedroom lamp template is (obviously), the template light. The only reason I made a template light was so I could change the temperature and brightness at the same time. For a template light, if the brightness variable is defined, the set_level script is called. However, if brightness and color_temp are both defined, the color_temp script is called. To deal with this, I made a separate script which basically turns the light on at the desired brightness first if brightness is defined and then sends the color_temp command. The script which you can see is called from my template script is:

alias: change brightness and color temp
sequence:
  - choose:
      - conditions:
          - condition: template
            value_template: '{{ brightness != -1 }}'
        sequence:
          - service: light.turn_on
            data_template:
              entity_id: '{{ light_ct }}'
              brightness: '{{ brightness }}'
    default: []
mode: parallel
max: 100

As you can see, my lights are also customized with the min and max color_temp and to use circadian lighting (which is an awesome HACS integration).

Group Lights

Controlling a group of lights was also a little tricky. I set up the same MQTT and template entities for each light, but the group light required its own template entity.

  - platform: template
    lights:
      kids_bath_1:
        friendly_name: "kids_bath_1"
        level_template: >
          {% if state_attr('light.kids_bath_1_mqtt','brightness')|int >0 %}
            {{ state_attr('light.kids_bath_1_mqtt','brightness') }}
          {% else %}
            0
          {% endif %}
        value_template: "{{ states('light.kids_bath_1_mqtt') }}"
        temperature_template: >
          {% if state_attr('light.kids_bath_1_mqtt','color_temp')|int >0 %}
            {{ state_attr('light.kids_bath_1_mqtt','color_temp') }}
          {% else %}
            0
          {% endif %}
        turn_on:
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_1_mqtt
              brightness: "{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}"
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_1_mqtt
              color_temp: "{{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }}"
        turn_off:
          - service: light.turn_off
            data_template:
              entity_id: light.kids_bath_1_mqtt
        set_level:
        - service: light.turn_on
          data_template:
            brightness: "{{ brightness|int }}"
            entity_id: light.kids_bath_1_mqtt
        set_temperature:
        - service: script.change_brightness_and_color_temp
          data_template:
            light_ct: light.kids_bath_1_mqtt
            brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
        - service: light.turn_on
          data_template:
            color_temp: "{{ color_temp|int }}"
            entity_id: light.kids_bath_1_mqtt
        max_mireds_template: 370
        min_mireds_template: 153

  - platform: mqtt
    schema: template
    name: "kids bath 1 mqtt"
    command_topic: "cmnd/zbbridge/ZbSend"
    state_topic: "tele/zbbridge/kids_bath_1/SENSOR"
    command_on_template: >
      {"device": "kids_bath_1","send":{
      {%- if brightness is defined -%}
      "Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness }}{%- endif -%}
      {%- elif color_temp is defined -%}
      "CT": {{ color_temp }}
      {%- else -%}
      "Power": 1
      {%- endif -%}
      }}
    command_off_template: '{"device": "kids_bath_1","send":{"Power":0}}'
    state_template: >
      {% if value_json['ZbReceived']['kids_bath_1']['Power'] == 0 %}
        off
      {% elif value_json['ZbReceived']['kids_bath_1']['Power'] == 1 %}
        on
      {% else  %}
        {{ states('light.kids_bath_1_mqtt') }}
      {% endif %}
    brightness_template: >
      {% if value_json['ZbReceived']['kids_bath_1']['Dimmer'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_1']['Dimmer'] == 255 %}
          254
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_1']['Dimmer'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_1_mqtt','brightness') }}
      {% endif %} 
    color_temp_template: >
      {% if value_json['ZbReceived']['kids_bath_1']['CT'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_1']['CT'] > 370 %}
          370
        {% elif value_json['ZbReceived']['kids_bath_1']['CT'] < 153 %}
          153
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_1']['CT'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_1_mqtt','color_temp') }}
      {% endif %} 
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    optimistic: false
    qos: 0
    max_mireds: 370
    min_mireds: 153
    
  - platform: template
    lights:
      kids_bath_2:
        friendly_name: "kids_bath_2"
        level_template: >
          {% if state_attr('light.kids_bath_2_mqtt','brightness')|int >0 %}
            {{ state_attr('light.kids_bath_2_mqtt','brightness') }}
          {% else %}
            0
          {% endif %}
        value_template: "{{ is_state('light.kids_bath_2_mqtt','on') }}"
        temperature_template: >
          {% if state_attr('light.kids_bath_2_mqtt','color_temp')|int >0 %}
            {{ state_attr('light.kids_bath_2_mqtt','color_temp') }}
          {% else %}
            0
          {% endif %}
        turn_on:
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_2_mqtt
              brightness: "{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}"
          - service: light.turn_on
            data_template:
              entity_id: light.kids_bath_2_mqtt
              color_temp: "{{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }}"
        turn_off:
          - service: light.turn_off
            data_template:
              entity_id: light.kids_bath_2_mqtt
        set_level:
        - service: light.turn_on
          data_template:
            brightness: "{{ brightness|int }}"
            entity_id: light.kids_bath_2_mqtt
        set_temperature:
        - service: script.change_brightness_and_color_temp
          data_template:
            light_ct: light.kids_bath_2_mqtt
            brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
        - service: light.turn_on
          data_template:
            color_temp: "{{ color_temp|int }}"
            entity_id: light.kids_bath_2_mqtt
        max_mireds_template: 370
        min_mireds_template: 153

  - platform: mqtt
    schema: template
    name: "kids bath 2 mqtt"
    command_topic: "cmnd/zbbridge/ZbSend"
    state_topic: "tele/zbbridge/kids_bath_2/SENSOR"
    command_on_template: >
      {"device": "kids_bath_2","send":{
      {%- if brightness is defined -%}
      "Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness }}{%- endif -%}
      {%- elif color_temp is defined -%}
      "CT": {{ color_temp }}
      {%- else -%}
      "Power": 1
      {%- endif -%}
      },"endpoint":1}
    command_off_template: '{"device": "kids_bath_2","send":{"Power":0}}'
    state_template: >
      {% if value_json['ZbReceived']['kids_bath_2']['Power'] == 0 %}
        off
      {% elif value_json['ZbReceived']['kids_bath_2']['Power'] == 1 %}
        on
      {% else  %}
        {{ states('light.kids_bath_2_mqtt') }}
      {% endif %}
    brightness_template: >
      {% if value_json['ZbReceived']['kids_bath_2']['Dimmer'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_2']['Dimmer'] == 255 %}
          254
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_2']['Dimmer'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_2_mqtt','brightness') }}
      {% endif %} 
    color_temp_template: >
      {% if value_json['ZbReceived']['kids_bath_2']['CT'] is defined %}
        {% if value_json['ZbReceived']['kids_bath_2']['CT'] > 370 %}
          370
        {% elif value_json['ZbReceived']['kids_bath_2']['CT'] < 153 %}
          153
        {% else %}
          {{ value_json['ZbReceived']['kids_bath_2']['CT'] }}
        {% endif %}
      {% else %}
        {{ state_attr('light.kids_bath_2_mqtt','color_temp') }}
      {% endif %} 
    availability_topic: "tele/zbbridge/LWT"
    payload_available: "Online"
    payload_not_available: "Offline"
    optimistic: false
    qos: 0
    max_mireds: 370
    min_mireds: 153
    
  - platform: template
    lights:
      kids_bath:
        friendly_name: "kids_bath"
        level_template: "{{ (state_attr('light.kids_bath_1_mqtt','brightness')|int + state_attr('light.kids_bath_2_mqtt','brightness')|int )/2 }}"
        value_template: "{{ is_state('light.kids_bath_2_mqtt','on') and is_state('light.kids_bath_1_mqtt','on') }}"
        temperature_template: "{{ (state_attr('light.kids_bath_1_mqtt','color_temp')|int + state_attr('light.kids_bath_2_mqtt','color_temp')|int )/2 }}"
        turn_on:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ (state_attr('switch.circadian_lighting_circadian_lighting','brightness')|float*(255/100)|float)|int }}{%- endif -%}}}
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"CT": {{ ((1000000)|float/state_attr('sensor.circadian_values','colortemp')|float)|int }} }}
        turn_off:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: '{"group": "101","send":{"Power":0}}'
        set_level:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"Dimmer": {%- if brightness == 255 -%}254{%- else -%}{{ brightness|int }}{%- endif -%}}}
        set_temperature:
          - service: script.change_brightness_and_color_temp_group
            data_template:
              group: 101
              brightness: "{% if brightness is defined %}{{ brightness|int }}{% else %}-1{% endif %}"
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "101","send":{"CT": {{color_temp}} }}
        max_mireds_template: 370
        min_mireds_template: 153

This assumes an arbitrarily defined group number of 101 and uses a similar script to control both brightness and color_temp at the same time.

alias: change brightness and color temp group
sequence:
  - choose:
      - conditions:
          - condition: template
            value_template: '{{ brightness != -1 }}'
        sequence:
          - service: mqtt.publish
            data:
              topic: cmnd/zbbridge/ZbSend
              payload_template: >
                {"group": "{{ group }}","send":{"Dimmer": {%- if brightness ==
                255 -%}254{%- else -%}{{ brightness|int }}{%- endif -%}}}
    default: []
mode: parallel
max: 100

Conclusion:

As you can see, Zigbee2Tasmota is a fairly powerful platform, incredibly operating on a system with very limited resources. It is not for everyone, but could be enticing primarily for people with the Tasmotized Sonoff Zigbee Bridge. Connecting a WiFi device to Home Assistant through serial over TCP may not be the best or most stable setup. Instead allowing the coordinator to function independently and sharing the device information over MQTT might be more desirable.

On the other hand, if you have Tube's wired zigbee coordinator or the ZB-GW03 wired coordinator, you may be better off using ZHA for simplicity sake. Regarding Zigbee2MQTT, it should be noted that the EZSP radio found in the Sonoff Zigbee Adapter and the ZB-GW03 is not finalized and may be buggy.

Links to adapters:

Sonoff Zigbee

Affiliate link:

SONOFF ZBBridge Smart Zigbee Bridge Hub, WI-FI & Zigbee Dual-protocol Supporting, APP Control and Multi-device Management

Non-affiliate link:

SONOFF ZBBridge Smart Zigbee Bridge Hub, WI-FI & Zigbee Dual-protocol Supporting, APP Control and Multi-device Management

Eachen ZB-GW03

Tube's ZB Coordinator

Let me know if you have any questions. Thanks for reading!

Edit: changed instances of zigbee router to the correct terminology zigbee coordinator thanks to /u/chick_repellent.

Playing a sound directly from Home Assistant Operating System

thehelpfulidiot — Sat, 02 Oct 2021 00:00:00 +0000

This may sound like it should've been a trivial task to accomplish but took me way longer to solve than I anticipated. As I said it my first post, I started my Home Automation journey with things like Nest. This included the (now killedbygoogle) Nest Secure with Nest Detects. I did enjoy a very pleasing *DING *each time a door opened. As time went on, I tried to distance myself as much as possible from the cloud.

Integrating them into Home Assistant was a major PITA though. I had to purchase the amazing Starling Home Hub, and use Homekit to create an automation where each door open and door close event would trigger an automation to control an MQTT switch through Homebridge which would then pass the information into Home Assistant. This worked pretty well for the most part, but once the batteries started to die in the Nest Detects, it seemed like the perfect opportunity to replace the door sensors.

I just purchased cheap Aqara Door Sensors and mounted them on my door. Obviously, the open/close information was passed far more easily into Home Assistant and I was able to simplify my work flow, but one thing I did miss was the DING.

DING

I wanted something as simple as possible, and to me, the obvious solution was to connect a speaker to my Home Assistant box's 3.5mm jack and just play the sound. This was wayyyy more complicated than I expected.

Home Assistant's addon store has a very bizarre version of VLC. It seems like it is feature incomplete and has almost no options. I couldn't figure out how to use it.

VLC from Official Add-on Store

Fortunately, there exists another version of VLC from a repository belonging to rodripf. Simply add the following address to the add-on repository list to install it:

https://github.com/rodripf/hassio-local-vlc

Once installed, it can be configured for telnet access. This is what I found to be the easiest and most reliable way to play a sound. Set the following options as desired.

Once that is set up, it is necessary to add a vlc_telnet entity which is done through yaml.

media_player:
  - platform: vlc_telnet
    name: vlc
    host: 6872d8d8-local-vlc
    password: <secret password>

Edit the entry as you see fit, but the important thing is the host, which identifies the Home Assistant container, should be written as above.

After a reboot, "media_player.vlc" should exist and through this, you can play whatever sound you want, locally. Although, I had trouble playing the music unless I uploaded it to the "/share" directory.

Once you've uploaded your sound file to the /share directory, the following will play it:

service: media_player.play_media
target:
  entity_id: media_player.vlc
data:
  media_content_id: /share/chime.wav
  media_content_type: music

It is important to make the media_content_type "music" as that is the only media type currently playable by the vlc_telnet integration.

You can go to a web gui of your VLC instance by going to your homeassistanturl:9892 and using the password you created in the addon settings page.

It's important to make sure your volume here is set to something reasonable. At first, I could not get any sound to play since the volume defaulted to 0.

You can also set the volume through Home Assistant as you would for any media player.

service: media_player.volume_set
target:
  entity_id: media_player.vlc
data:
  volume_level: 0.35

Hope that helps. Seems like a fairly complex solution for a fairly simply problem, but like I said, this was the only way I could figure to do something like this from within Home Assistant Operating System (obviously other installations in standard operating systems can probably do this trivially).

The nice thing about this as opposed to the Nest solution, is that now I have control over when the door dings and when it doesn't. For example, my ding automation does not fire when my kids are asleep. Now I don't have to worry about waking them when I run out to the car.

Thanks for reading.

A Wired Sonoff Zigbee Alternative

thehelpfulidiot — Wed, 29 Sep 2021 00:00:00 +0000

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

UPDATE 12/5: We've come full circle. The amazing developers of Tasmota have fixed the issue causing serial port disconnections to the zigbee radio with Tasmota 10. Now, the below instructions can be followed, without relying on the previous update posts from 10/15 and 10/18. The uploaded precompiled Tasmota firmware below has been updated to the newest version as of today.

Many of us have seen the Sonoff Zigbee Hub and how easy it is to flash to Tasmota thanks to many awesome videos and my personal favorite by DigiblurDIY. The device is amazing and works well, but some things like your Zigbee bridge shouldn't rely on WiFi.

Recently, another amazing individual going by Tube has created a homemade Franken Zigbee router using an ESP32 board, ethernet, zigbee, and a 3D printed case. It comes preflashed with ESPHome and is instantly compatible with Home Assistant ZHA or Zigbee2MQTT. However, each one of these devices is being made by hand and supply cannot meet demand.

I came across another wired zigbee hub using an ESP32 - the EACHEN Zigbee Bridge Pro. For only $30, I thought I would give it a try. It shipped from China, but arrived fairly quickly after only a couple of weeks.

The hub

An ethernet port, a USB-C power port, and a reset pin hole

The device is probably the same size as the Sonoff Zigbee Router. It is easy enough to disassemble though since there are no screws and the bottom can simply be pried off.

The board

The board pops right out. It is now ripe for flashing and you can see some solder still on mine. My experience with Tasmota so far had been with mainly Tuya-convert and flashing the Sonoff Zigbee router which did not require soldering. This was my first time soldering and was way more fun than I expected.

Now before we get to soldering and flashing, we need to compile our own Tasmota firmware. We need to make sure Tasmota has access to the Ethernet port and the Zigbee Radio. As in all things Tasmota, Blakadder has a page devoted to this device and instructions. The instructions were excellent, but assumed a certain level of knowledge to compile the firmware. Having not done this before, this part took me the longest. The tasmota32-zigbeebridge precompiled firmware mentioned on the site did not provide ethernet access or else this process would have been a lot easier.

After much trial and error, the easiest way I found to compile this firmware was by going to GitPod. After you log in, you are taken to a VS code window prepopulated with the latest Tasmota build.

The instructions from Blakadder's website say to edit two files- user_config_override.h and platform_override.ini.

'Platform_override.ini' is located in the root directory - /workspace/Tasmota. Open the file, scroll to the bottom, and add:

[env:tasmota32-EWELINK]
extends                 = env:tasmota32
board_build.f_cpu       = 240000000L
board_build.f_flash     = 40000000L
build_flags             = ${env:tasmota32_base.build_flags} -DFIRMWARE_EWELINK

Now find the 'user_config_override.h' file which is located in the tasmota subdirectory (/workspace/Tasmota/tasmota). Open that file, erase all its contents and replace with:

#ifdef FIRMWARE_EWELINK
  #warning **** Build: FIRMWARE FOR EWELINK ZB-GW03-V1.2 ****

  #undef  SERIAL_LOG_LEVEL
  #define SERIAL_LOG_LEVEL LOG_LEVEL_NONE

  #define USE_ZIGBEE
  #undef  USE_ZIGBEE_ZNP
  #define USE_ZIGBEE_EZSP
  #define USE_UFILESYS
  #define USE_ZIGBEE_EEPROM // T24C512A
  #define USE_TCP_BRIDGE
  #undef  USE_ZIGBEE_CHANNEL
  #define USE_ZIGBEE_CHANNEL 11 // (11-26)
  
  #define USE_ETHERNET
  #undef  ETH_TYPE
  #define ETH_TYPE 0 // ETH_PHY_LAN8720
  #undef  ETH_CLKMODE
  #define ETH_CLKMODE 3 // ETH_CLOCK_GPIO17_OUT
  #undef  ETH_ADDRESS
  #define ETH_ADDRESS 1 // PHY1
#endif

Ok, so those two parts were relatively easy, but the next part on how to compile took me probably longer than it should have to figure it out. In the terminal at the bottom of the screen, while in the /workspace/Tasmota directory, type the following:

platformio run -e tasmota32-EWELINK

Now wait patiently for the new firmware to compile. Once it does, it can be found in /workspace/Tasmota/build_output/firmware. Download 'tasmota32-EWELINK.bin.

tasmota32-EWELINK

I compiled this on 10/7/2021, Tasmota32 9.5.0. I promise that I did not tamper with it (and would not know how to tamper with it if I wanted), but you should probably not download firmware from some random guy on the internet.

This is Tasmota 10 firmware compiled 12/5/21 using the above method. It should work as intended without the previously described work-arounds.

NOTE: As reported by erkoc, my new tasmota firmware based on v10 was still giving him the same issue with reboots losing connection to ZHA without manual intervention. He forked Tasmota32 and made sure the serial communication patch was included. He has instructions on how to compile your own or you can simply download his precompiled version.

https://github.com/vahempio/Tasmota-for-eWeLink

Ok, now that we have the firmware, it's time to start flashing. Since we have an ESP32. I do not believe we can use Tasmotizer. I have Windows and used ESP-Flasher-Windows-x64. Open it up, choose your serial port, the firmware we just compiled.

Now the fun part with soldering. Below I have a photo of the ESP32 board and the corresponding connection on my Serial Adapter (I use this one).

ESP32 board - labels point to corresponding pin on Serial Converter

Serial Converter - labels point to corresponding pin on ESP32

Now Flash.

Once you finish, it's a matter of setting up Tasmota and a few more finishing touches.

Set up Tasmota through the access point first and add this Template by issuing the following command:

backlog Template {"NAME":"ZB-GW03-V1.2","GPIO":[0,0,3552,0,3584,0,0,0,5793,5792,320,544,5536,0,5600,0,0,0,0,5568,0,0,0,0,0,0,0,0,608,640,32,0,0,0,0,0],"FLAG":0,"BASE":1} ; module 0

From this website, select ncp-uart-sw-6.7.8_115200.ota. Go to the upgrade firmware screen to update the zigbee radio. The device will automatically recognize this as firmware for the zigbee radio and will apply it accordingly.

Now try plugging the device in to Ethernet. I noticed it was sort of strange and took a couple reboots and unplugging and plugging the cable back in, but I saw messages about setting up the Eth connection in the console. There was no real way to change Ethernet settings like you can with WiFi, but I noticed two DHCP listings in my router. Once Ethernet is set up, you can disconnect WiFi with the following, but do not do the following until Ethernet is set up! Make sure the first command turning WiFi off works first before applying the subsequent rule which will disable it on each reboot. The first command by itself will not persist across reboots.

Wifi 0
backlog rule2 on system#boot do Wifi 0 endon ; rule2 1

If something went wrong and now you are locked out of the device because Ethernet isn't working and WiFi is down, you can try Tasmota device recovery.

The device is now set up to use with Zigbee2Tasmota. For Home Assistant, there are a few more steps.

To integrate with Home Assistant, we must remap the Zigbee Tx and Zigbee Rx pins directly to TCP Tx and TCP Rx. This will disable Zigbee2Tasmota, but will allow the device to integrate with Home Assistant. Use the following command to update the template:

backlog template {"NAME":"ZHA-bridge","GPIO":[0,0,5472,0,5504,0,0,0,5793,5792,320,544,5536,0,5600,0,0,0,0,5568,0,0,0,0,0,0,0,0,608,640,32,0,0,0,0,0],"FLAG":0,"BASE":1} ; module 0

Notice the change in the third and fifth positions. Afterwards, it will be possible to start the TCP Server which is what ZHA needs to connect to the coordinator.

Enter the following from the console to turn on the webserver for the Zigbee radio:

backlog rule1 on system#boot do TCPStart 8888 endon ; rule1 1 ; tcpstart 8888

Finally, you can jump into Home Assistant and set up the device for ZHA exactly how it's (masterfully) described by DigiBlur.

https://www.digiblur.com/2020/07/how-to-use-sonoff-zigbee-bridge-with.html

Thanks for reading. I hope this helps fill in the blanks for some folks like myself who are not as familiar with the inner workings of Tasmota. Please let me know if you have any questions.

Thanks to DigiBlur and Blakadder and all the other folks who did the hard work in getting this device to work.

Alternative: a single template, no toggling

A reader working through this guide found a cleaner approach that avoids the ZHA/Z2T toggling described above entirely. Rather than the ZHA-bridge template, use:

backlog template {"NAME":"Zig_Bridge32","GPIO":[1,1,5472,1,5504,1,1,1,5793,5792,320,544,5536,1,5600,1,0,1,1,5568,0,1,1,1,0,0,0,0,608,640,32,1,1,3552,3584,1],"FLAG":0,"BASE":1}; module 0;

The reasoning: the ZHA-bridge template has no Zigbee Tx/Rx GPIOs at all, so Tasmota never resets the EZSP chip at boot. Mapping Zigbee Rx and Tx to otherwise unused GPIOs means the reset does happen, and the TCP connection works across a cold start. The Zigbee2Tasmota UI is still visible but unused.

For Zigbee2MQTT, adding this to your config makes it reconnect within a minute of a bridge reboot:

availability:
  active:
    timeout: 1

Link to Purchase

Affiliate Link:

AliExpress.com Product - EACHEN EWeLink Zigbee Bridge Pro (Zbbridge Pro)

eWelink ZigBee 3.0 Smart Wired Wireless Gateway Hub

Non-Affiliate Link:

AliExpress.com Product - EACHEN EWeLink Zigbee Bridge Pro (Zbbridge Pro)

eWelink ZigBee 3.0 Smart Wired Wireless Gateway Hub

EACHEN Website:

EACHEN eWeLink Zigbee Bridge Pro (zbbridge pro)

CORRECTION 10/2: added steps to remap zigbee Tx and Rx to TCP.

CORRECTION 10/7: made changes based on feedback from /u/Vertigo722. Edited platformIO text. Changed some order of setup.

CORRECTION 10/14: hard reset of the coordinator breaks connection with ZHA until toggled back to Z2T and then back again. Using this firmware and the rules below automatically toggle back and forth only on cut power. Hopefully this workaround fixes things for now.

tasmota32-EWELINK-1

UPDATE 10/15: READ IF YOU PLAN TO USE HUB WITH ZHA

UPDATE 10/18: READ IF YOU PLAN TO USE HUB WITH Zigbee2Tasmota

About

thehelpfulidiot — Wed, 29 Sep 2021 00:00:00 +0000

As someone without a tech background, my introduction to the world of self hosting and home automation began as a slow and bumpy ride with many false starts. Since beginning seriously in 2019, I am proud of what I have accomplished with OPNsense, Home Assistant, Proxmox, Docker, to name a few. Here is a tech blog, not from a computer scientist or an engineer, but from an ophthalmologist.

The Birth of a Smart Home

thehelpfulidiot — Wed, 29 Sep 2021 00:00:00 +0000

When the first Google Home came out in 2016, I briefly thought about buying it, but decided I had no use for it. Shortly after, my brother-in-law bought me two Chinese Tuya White-label smart plugs from Amazon. I plugged a couple of lamps into them and was hooked. Google home followed shortly after. At the time, we were renting and so my ability to "smartify" our home was limited. I replaced many light bulbs with Sengled bulbs, changed out smoke detectors for Nest, but that was pretty much it.

In 2019 we bought our first home. I knew from the beginning that building a smart home would be an obsession. During the year leading up to the move, I kept my eye on deals and snatched up some of the essentials- Hue light bulbs, Nest smoke detectors, Nest lock, Nest thermostats, and maybe a few others. Anything that could be replaced by an electronic wireless device was on the chopping block.

At this point I knew nothing about Linux or Home Assistant. The best smart home options were the well-known cloud versions. I set up a few automations with SmartThings, but that was it.

After a few months, I came across a Reddit Post about Mozilla Webthings. It was an incredibly simple smart home solution which could be installed on a raspberry pi. I drove straight to Micro Center to pick up the necessary equipment and got started. This solution was game changing but I quickly reached it's limitations and the never-ending quest for smart home supremacy began.

Below is a diagram of my current setup:

Network Diagram

Initially, I migrated everything to the fewest number of boxes as possible. This scared me since there was only a single point of failure. More recently, I have purchased very cheap celeron computers from Amazon Warehouse and have used a few to host various services.

I hope to share some lessons learned as someone who entered this knowing nothing and after a lot of hard work, I can honestly say I know more than nothing, but still very little.

Thanks for reading.