Integrating wmBus devices into iobroker

After my quite expensive MH-Collector (identical with the easy.MUC from solvimus) died (it survived just a little longer that warranty protects), I decided to collect my wmBus devices‘ data with some home brewn solution. I’m also the owner of a Ubiquity US-24-250W, so the descision to go with PoE supply is quite an easy descision. So what lies closer than using a Raspberry Pi 3B+ with PoE hat and an USB-wmBus stick?

No sooner said than done, I bought the parts, installed raspbian and an iobroker slave. The iobroker master ist now running on a Debian 10 buster VM on my tiny HP Proliant server… How to install the iobroker slave can be found here.

wmBus Hardware

For me, the appropriate hardware was the IMST iM871A-USB (you can buy it directly from IMST or from tekmodul). This wmBus stick provides a serial interface (e.g. /dev/ttyUSB0), is quite cheap and supported by most open source wmBus software. But here comes the tricky part. There are quite some paths you can go, but for me, using the Messhelden heat cost allocators, I found myself in a very frustrating situation. These devices stick to the OMS standard for most of the telegram, but unfortunately do some very shitty stuff at slot 2 and 3, so many decoders fall out of sync just after the first data slot.

wmBus Software

After trying different ioborker adapters (like iobroker.wm-bus) and also deamon solutions (like wmbusmeters), sending the data to some MQTT broker (a server is easy to rise up in iobroker), I ended with the iobroker.wmbus (beware of the dash, it is not the same as above). Somehow the author of this adapter managed to come up with the inconsistencies, I even could not decode manually, looking at each single bit and byte of the wmBus telegram.

After attaching the iM871A-USB stick to the Pi and placing it at some location where it can receive all meters you are interested in, you need to install and configure the iobroker adapter iobroker.wmbus.

Adapter Configuration

The configuration is also quite easy and should look like the following:

It could also be, that your slaves need other modes to be received. One widespread mod for battery driven devices is also mode C. Unfortunately, a single stick can not receive multiple modes. But usually you only run devices with a single mode. Another important setting is the baud rate. For the IMST device, it needs to be 57600. The stick contains some serial converter that attaches the IMST module with a real serial connection.

Add Encrypted Meters

After finishing configuration and starting up the adapter, it is time to have a look into the log. There you will see, if the adapter started up correctly. If it did, you soon should see a line that says „Updated device state:<MANUF>-<ID>“ or an error saying, that it could not decrypt a telegram due to missing decryption key. If this occurs, go to the adapter configuration again. There you should see a new entry with a key „UNKNOWN“. Place the correct key there and push „Save“.

The follwoing telegram of that device should be decrypted correctly and a new state will be created within the object tree of iobroker.

If you see other unencrypted devices that pollute your object tree or your log with encryption failed messages, simply put them below „Blocked Devices“ tab in the adapters configuration. My Pi can see at least 20 unencrypted Techem water meters and heat cost allocators.

Let’s encrypt (also on wmBus)

I don’t know, how they can survive in a time of GDPR (General Data Protection Regulation), but they still have no hurry to encrypt their telegrams with a device-unique key. I think it is a security issue, when burglars can easily find people that do not heat in wintertime or have no water demand currently. But at least, Techem sticks closely to the OMS. If you rent a flat, that still has unencrypted wmBus meters, I would definitely claim to get encrypted meters. Even if encryption of wmBus has some weaknesses, it is by far better than plaintext.

Merging the Contents of Two InfluxDBs

Eveer had the problem that data runs into two different influx databases and you want to merge the data into a single one? You wonder, why this can happen? Then just think about migrating some data aquisition project from one server to another without a downtime by spooling the aquired data into both DBs for some time or simply setting up a fresh system after the old one is dying slowly because of low performance. This happened to me with my smart home system, running iobroker and some influxdb on it. The old one ran on a ODROID-HC1 with only 2GB RAM. The new one is a Debian 10.0 VM on my brand new HP Proliant running XCP-NG. (OK, it’s not new, it’s used HW, but for private use, it is a monster 🙂 )

For sure, there exist great tools from the influx inventor, but this is way to much for a small project to set up. So I decided to go the easy way of first doing the absolutely necessary (bring up the new system) and later pull in the historical data.

Backing up and restoring iobroker is well documented and works smooth and will not be part of this post. For influx, I just setup a new instance with a copy of the /etc/influx.conf and started the service. Then I started iobroker and everything worked like before, except the availability of the historical data.

After endless searching throughout the web and many tries to export the DB as CSV, JSON,…, I found a set of scripts from ETZ, that helpt me half the way up and down the backup-restore-process.

The key to success in the end was, to restore the DB on the new server not as the original one, but to rename it. In my case, I called the new historical DB on the new server iobroker_old.

Now I had all the data I want to import at least on the new instance, but how could iobroker find it? It won’t! I needed to do some internal import within influxdb to migrate the iobroker_old into the iobroker database.

To export the DB from the old system, just issue (on the old system):

$ backup-influxdb
$ scp /var/backups/influxdb/2019.... <myuser>@<newserver>:~

On the new server, then do:

$ restore-influxdb-database-online \
    ~/2019... \
    iobroker iobroker_old

The command to my success was (with help from GIST, before you run the command, doing a snapshot of the system may be a good advise):

$ influx -database iobroker_old -execute \
   'SELECT * INTO "iobroker"."global".:MEASUREMENT FROM "iobroker_old"."global"./.*/ GROUP BY *'

To make this succeed, I needed to raise the physical RAM of the VM to 16 GB. With 4 GB it simply swallowed the swap and ran into some timeout. Afterwards, I turned it back to 4 GB and everything runs fine.

How to build a Smart Home

Since everbody complains about smart homes are vendor lock in, too expensive, giving you data for free to some suspecting cloud provider,… I need to preset my smart home solution to you, that can be completely running inside your „four walls“ and does not need any third party, if you do not want it to. Additionally, there is NO vendor lock in. Everything can be open source and connect to almost any existing smart home appliance.

What I’m talking about? It’s iobroker.

If you already own a Homematic System from EQ-3 (ELV in older ages), iobroker is a must-have. BTW: If you already own a Charly, you can install iobroker on it without addional hardware, but be advised to install a USB drive with enough storage for the data that accumulates when years pass by. I would not suggest to use a flash based drive, since it needs a tradeoff between amount of data lost when power cuts and the endurance of your drive. Write cycles will be quite often…

Welp, where to start? I would first get some low-power and reliable mini PC. The solution I selected is from hardkernel and called ODROID HC-1 (in Germany best bought at Pollin for 60€). This is a little ARM board running ubuntu linux and providing space for a single 2.5″ HDD. I selected a WD Red 1TB drive for that purpose. The linux OS itself needs to be put on a Micro-SD card.

I would suggest to also buy the following:

  • Power supply (or a PoE+ adapter if you own a PoE+-capable switch)
  • Micro-SD card with at least 8GB (best with ubuntu preinstalled)
  • The serial cord from hardkernel
  • The Battery (for RTC)
  • A hard disk for all the data your smart home will collect and
  • A little case against the dust, that definitely will render you appliance as very attractive 😉

Start installing your software. First you need to install the Ubuntu OS, provided by hardkernel (if not already installed).

When everything arrived, connect it to your network, find out the IP address or use the serial cord and start installing everything you need (supposing, you do all under root, if not, prepend a sudo where needed):

Prepare your hard disk

dmesg  # Find out, which is your HDD, assuming /dev/sda
fdisk /dev/sda
# Create 2 Partitions
# 8 GB with swap
# Remainder with ext4 or something else (I prefer btrfs)
# write partition table with entering 'w'
mkswap /dev/sda1
swapon /dev/sda2 
mkfs.ext4 /dev/sda2
mkdir /var/data
blkid /dev/sda2   # Take the UUID-part
echo "UUID=<the-above> /var/data   ext4 errors=remount-ro,noatime 0 1" \
  >> /etc/fstab
mount /var/data
mkdir /var/data/iobroker
mkdir /opt/iobroker /var/lib/influxdb
echo "/opt/iobroker/ /var/data/iobroker  none bind" \
  >> /etc/fstab
echo "/var/data/influxdb/ /var/lib/influxdb none bind" \
  >> /etc/fstab

Installing node.js

sudo apt-get install python build-essential curl
mkdir src
cd src
wget https://nodejs.org/dist/v10.15.3/node-v10.15.3.tar.gz
tar xzvf node-v10.15.3.tar.gz
cd node-v10.15.3.tar.gz
./configure --without-snapshot
make
./node -v  # If version is returned than 'make' was OK 
make install

Installing InfluxDB

curl -sL https://repos.influxdata.com/influxdb.key | \
   sudo apt-key add - source /etc/lsb-release echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update && sudo apt-get install influxdb sudo systemctl unmask influxdb.service sudo systemctl start influxdb 

Installing Go language

apt install git golang-go  # needed to build a newer go
cd /usr/lib
git clone https://go.googlesource.com/go
cd go
git checkout go1.12.5
cd src
./all.bash
cd $HOME
cd /usr/bin
rm go
ln -s ../lib/go-1.12 go
cd $HOME
cat >>.profile <<HERE
export GOROOT=/usr/lib/go
export GOPATH=$HOME/go
[ -d $GOPATH ] || mkdir $GOPATH
[ -d $GOPATH/bin ] || mkdir $GOPATH/bin
export PATH=$GOPATH/bin:$PATH
HERE

Installing yarn

see: https://yarnpkg.com/en/docs/install#debian-stable

Installing Chronograph

For creating your own querys to InfluxDB (e.g. with node red), it is easiest to have some good InfluxDB interface, best to be used in a browser…

With go and yarn, it is as easy as boiling water:

go get github.com/influxdata/chronograf
cd $GOPATH/src/github.com/influxdata/chronograf
make
go install github.com/influxdata/chronograf/cmd/chronograf

Now you can start chronograph

./chronograph

Now visit the chronograph web interface by browsing to http://<host>:8888. A Wizard will welcome you. Enter the appropriate data to access your InfluxDB. Youcan skip the Kapacitor question. After finishing the wizard, you should be able to see the following on chronographs Config tab.

When this is done, issue your first query on the Explore tab (use the datapoint selector on the lower half of the page and select an appropriate date from the date picker in the upper right corner:

Install iobroker

COMING SOON…