Archiv September 2019

Merging the Contents of Two InfluxDBs

Eveer had the problem that data runs into two different influx databases and you want to merge the data into a single one? You wonder, why this can happen? Then just think about migrating some data aquisition project from one server to another without a downtime by spooling the aquired data into both DBs for some time or simply setting up a fresh system after the old one is dying slowly because of low performance. This happened to me with my smart home system, running iobroker and some influxdb on it. The old one ran on a ODROID-HC1 with only 2GB RAM. The new one is a Debian 10.0 VM on my brand new HP Proliant running XCP-NG. (OK, it’s not new, it’s used HW, but for private use, it is a monster 🙂 )

For sure, there exist great tools from the influx inventor, but this is way to much for a small project to set up. So I decided to go the easy way of first doing the absolutely necessary (bring up the new system) and later pull in the historical data.

Backing up and restoring iobroker is well documented and works smooth and will not be part of this post. For influx, I just setup a new instance with a copy of the /etc/influx.conf and started the service. Then I started iobroker and everything worked like before, except the availability of the historical data.

After endless searching throughout the web and many tries to export the DB as CSV, JSON,…, I found a set of scripts from ETZ, that helpt me half the way up and down the backup-restore-process.

The key to success in the end was, to restore the DB on the new server not as the original one, but to rename it. In my case, I called the new historical DB on the new server iobroker_old.

Now I had all the data I want to import at least on the new instance, but how could iobroker find it? It won’t! I needed to do some internal import within influxdb to migrate the iobroker_old into the iobroker database.

To export the DB from the old system, just issue (on the old system):

$ backup-influxdb
$ scp /var/backups/influxdb/2019.... <myuser>@<newserver>:~

On the new server, then do:

$ restore-influxdb-database-online \
    ~/2019... \
    iobroker iobroker_old

The command to my success was (with help from GIST, before you run the command, doing a snapshot of the system may be a good advise):

$ influx -database iobroker_old -execute \
   'SELECT * INTO "iobroker"."global".:MEASUREMENT FROM "iobroker_old"."global"./.*/ GROUP BY *'

To make this succeed, I needed to raise the physical RAM of the VM to 16 GB. With 4 GB it simply swallowed the swap and ran into some timeout. Afterwards, I turned it back to 4 GB and everything runs fine.

Headless Rescue System over SSH

Again, I stumbled over some problem… I need to draw some backup disk image of a headless bare metal server, to run a risky update of atlassian tools. And yes, it was one of the old servers in my network, not beeing virtualized.

I tried a lot of different solutions, including debian (and derivatives) with automated installer and it’s network-console, but it was not leading to a working ssh access. So I searched a lot and came to ploplinux. Just put it on a USB-Stick and run it. It will start up with ssh server and a default password for root (root:ploplinux).

To find your server in your network, just do (on another Linux machine):

nmap -sn | grep ^Nmap > network_snapshot.txt

and when ploplinux was booted (its a one-liner…)

nmap -sn | grep ^Nmap > network_rescue.txt && \
      diff network_snapshot.txt network_rescue.txt | grep ^\>

To find the IP address of the running ploplinux. It will show somewhat like:

> Nmap scan report for (   # Some other machine
> Nmap scan report for                  # This is ploplinux
> Nmap done: 256 IP addresses (39 hosts up) scanned in 7.85 seconds 

Then I logged in over SSH on as root using ploplinux as the password and was able to do what I need.

Mounting a NTFS drive (some large USB drives use it) needed the command mount.ntfs-3g <drive> <mountpoint> to work…

BTRFS-Backup (using squashfs)

Taking a raw image of some partition with BTRFS is not as convenient as it seems. To get some image of the containing filesystem, it is easier, to mount it directly and run mksquashfs on it. It then is easily mountable in any linux that supports squashfs. To create the backup, run the following commands, replacing sda1 with the device your filesystem to backup resides in. If you connected a large USB drive for putting your backup on, cd to there…

cd <where_some_space_is_available>
mkdir /mnt/tmp
mount -o ro /dev/sda1 /mnt/tmp
mksquashfs /mnt/tmp backup.squashfs

To test you backup-image:

mkdir /mnt/test
mount -o loop backup.squashfs /mnt/test

and examine the contents of your mounted filesystem tree. More information about squashing filesystems can be found here or here. If everything is fine, the backup is finished… Don’t forget to eject your USB drive cleanly, to not corrupt data with unwritten cache. Therefore, unmount your USB drive or use eject <usb_device> to do it.

Happy plopping 😉

STM32 UART Continuous Receive with Interrupt

My last post is quite some time ago, due to vacations and high workload. But now I encountered some problem within an embedded project, I want to share the solution with you. Continuously receive data using interrupts on UART is complicated (or even impossible) in HAL. Most approaches I found crawling the internet are using the LL library to achieve this and many discussions around HAL do not end in satisfaction. Some work around the problems with dirty approaches (e.g. changing the HAL code itself), other step back from interrupt and use a polling approach.

To be honest, the high levels of HAL do not offer such a solutions. Instead, it offers functions to receive a special amount of data using a non-blocking interrupt approach, handling all the difficulties with tracking the state in the instance stucture (huartX) and entering a callback for the diverse states of the reception/transmission, e.g.
void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart) or
void HAL_UART_RxHalfCpltCallback(UART_HandleTypeDef *huart)

Using HAL_UART_Receive_IT (not recommended)

A nearby approach without touching HAL code itself is, to call HAL_UART_Receive_IT(&huart3, &rxbuf, 1) once after initalization and at the end of the RxCpltCallback, to retrigger the reception, but this leads to some undesired lock (possibly a HAL-Bug), when transmitting data using HAL_StatusTypeDef HAL_UART_Transmit_IT(UART_HandleTypeDef *huart, uint8_t *pData, uint16_t Size), which could also not be the desired behaviour and the beginning of endless debug sessions and frustration.

Simply Enable the IRQ

The best solution in my opinion instead is really simple. Don’t use the high level receive functions at all for the continuous RX behaviour, since you do not want to receive a special amount of data but be called at each reception. So, configure the UART with interrupt in CubeMX and after it’s initalization, enable the interrupt itself, never calling the HAL_UART_Receive_IT or any other UART receive function (it will disable the IT after finishing).

In the section of the appropriate instance in void HAL_UART_MspInit(UART_HandleTypeDef* uartHandle), add the following line of code:


In stm32xxx_it.c do:

void USART3_IRQHandler(void)  {    
    return;  // To avoid calling the handler at all 
             // (in case you want to save the time)
    /* USER CODE END USART3_IRQn 1 */ 

The return statement will avoid calling the HAL IRQ handler. I did not try during transmit, but it seems not disturbing anything. If you plan to use the HAL_UART_Receive_IT functions in parallel, you could try to put your code below the handler. I did not test it, but there is a good chance that it works.

Since this approach only touches the user code functions, none of your code will be destroyed by code re-generation of CubeMX.

This is all you need… Happy UART processing 😉

If Timestamping is Needed

Simple Millisecond Timestamps

If you want to trigger on inactive time durations (some serial protocols use it as a synchronisation condition), save a timestamp (e.g. HAL_GetTick()) within the UART-RX-Interrupt and look at the difference to the previous one (subtract the duration of a byte to get the real inactive time).

High Resolution Timestamps

If sub-milli-second resolution is required, run a timer with a prescaler of desired resolution and take the counter value of the timer instead of the tick counter. (you can get it with __HAL_TIM_GET_COUNTER(&htimX)).

Hope this helps in your next project using UART 🙂