Just Another Hobby – 3D Printing

When it comes to leasure time, there are not many activities, that are more exciting than 3D printing. After you printed diverse things from Thingiverse or any other 3D printing gallery, there is a great wish, deeply buried in your mind, to design stuff yourself to get the real excitement — and what is more obvious than upgrading your 3D printer with parts printed by itself/ yourself 😛

If this was the right appetizer, look how my Ender3 got upgraded with a full metal frontend or take a look at my Thingiverse profile with some other designs. If you have any questions, especially the decisions, e.g. why I designed the new hotend exactly as it is (a wide monster hotend) or suggestions, how to improve it, do not hesitate to comment!



I’m not sure, if I’m simply a problem magnet or why some stuff does not work as described… Here is another case. The tool I tend to use can be found here at STM’s site.

When I connected a piece of custom designed hardware with my laptops USB port with BOOT0 tied to VCC, it immediately showed up a new USB-Device called „STM32 Bootloader“ in group „USB Devices“. I was really happy about that and started the DFuSe Demo Software from STM. But hey, it could not find an appropriate device. What the heck???

After some digging through the web, I found different suggestions and problem solutions, but none of it worked.The simple solution was, to uninstall the device’s driver in „Device Manager“, unconnect and reconnet again. A different device showed up just a goup above the previous one: „USB Controller“.

Welp, as soon as the device showed up, also the DFuSe Demo Software recognized the device and stepped into life.

To create a DFU file is well described in all the other resources on the web. Just use the DFU file manager that was installed along with the DFuSe Demo Software to create a DFU file out of e.g. a hex file.

Then press the lower „Choose…“ button to select the generated DFU file and press „Upgrade“… That’s all. Your device has a new Firmware on it.

Have fun!

STM32 FreeRTOS and printf

After some more coding, I found some more issues with FreeRTOS and printf, not being solved by my fix below. If you need to get it fixed completely, look at that forums post: ST Community
and the website of Dave Nadler: newlib and FreeRTOS
In my current project, I replaced the newlib-nano-printf implementation by adding github:mpaland/printf as a git submodule to my project and including the printf.h (it overwrites the printf-library function with macro defines) in my topmost header file.

This will be a very short post. If you experience hard faults when using printf (this happens mostly, when using floats) and you already ticked the appropriate settings in the project’s properties…

…don’t waste your time digging through assembler instructions with instruction stepping (like I did) just to realize, that memory management is broken when using FreeRTOS. It is simply a bug in CubeMX-generated source files. Locate your _sbrk-function (either in syscalls.c or in sysmem.c) and change it to the following:

caddr_t _sbrk(int incr)
  extern char end asm("end");
  static char *heap_end;
  char *prev_heap_end,*min_stack_ptr;
  if (heap_end == 0) 
    heap_end = &end; prev_heap_end = heap_end; 
  /* Use the NVIC offset register to locate the main stack pointer. */
  /* Locate the STACK bottom address */
  min_stack_ptr = (char*)(*(unsigned int *)*(unsigned int *)0xE000ED08);
  min_stack_ptr -= MAX_STACK_SIZE; 
  if (heap_end + incr > min_stack_ptr) {
    errno = ENOMEM;
    return (caddr_t) -1;
  heap_end += incr; 
  return (caddr_t) prev_heap_end;

For what _sbrk does, have a look here.

If you want to digg a bit deeper, here are some websites dealing with this problem:

ESP32-EVB and platform.io – Yet Another ESP32 tutorial

What the heck? Aren’t there enough toturials out there about ESP32? I believe: Yes, too many. And there are too many that struggle with setting up the arduino environment for non-arduino hardware. But there is a straight-forward solution that works out of the box: platform.io or also known as PIO.

And there are enought tutorials about installing platform.io. It is as easy as you can think. Install Visual Studio Code from Microsoft and install the PIO plugin in VSCode.

Then create a new project VSCode -> New Project and select Olimex ESP32-EVB with Arduino Platform. After PIO downloaded all dependencies and configured your project, we are ready to go!

Hello World – Relay Toggle

Open main.cpp and edit it, so it will look like that:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  pinMode(relay1pin, OUTPUT);

void loop() {
  digitalWrite(relay1pin, HIGH);
  digitalWrite(relay1pin, LOW);

…connect your Olimex ESP32-EVB and hit the upload button:

Done! Your Relay should toogle every 10 seconds.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.


Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code.

Extend your code to look like this:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  pinMode(relay1pin, OUTPUT);

void loop() {
  Serial.printf("Switch on\r\n");
  digitalWrite(relay1pin, HIGH);
  Serial.printf("Switch off\r\n");
  digitalWrite(relay1pin, LOW);

Add two lines to your platfromio.ini file. You will find the correct COM port in terminal when uploading the new code to your device.

In my case it is COM6 and platformio.ini looks like this:

platform = espressif32
board = esp32-evb
framework = arduino
monitor_port = COM6
monitor_speed = 115200

Upload your code again after changing the ini file. If it does not print your output automatically to your terminal, start the monitor manually:

From now on, you should see the printf output in your terminal every time you upload your code again:

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or whatever. But now, you have some great arduino IDE without arduino IDE 😛

Have fun coding!

Doxygen – Tips and Tricks

LaTeX non-interactive

To make LaTeX skip some errors without user interaction, you can add the option --interaction=nonstopmode to the pdflatex call. Easiest way to do so, is changing the LATEX_COMMAND_NAME in your Doxyfile.

LATEX_CMD_NAME = „latex –interaction=nonstopmode“

Do not forget the double quotation marks. Otherwise doxygen will remove the space and the command in your make.bat will fail.

If you now want to generate the code, step into your doxygen-generated latex folder (designated by LATEX_OUTPUT option in Doxyfile) and execute make.bat (on Windows) or make all (on *nix).

Adding a favicon to html output

Adding a favicon to html output, you need to specify it in a custom header and include the original image in HTML, as described here. To extract the default header file:

doxygen -w html headerFile

Add the follwing line to in headerFile within the html header

<link rel="shortcut icon" href="favicon.png" type="image/png">

And add your headerFile and the image to the HTML_EXTRA_FILES in your Doxyfile. Its path is relative to your Doxyfile.

HTML_HEADER = headerFile
HTML_EXTRA_FILES = some_rel_path/favicon.png

Now you can generate your html documentation with some favicon in place.

PDF output destination

Did you ever search for the PDF file, doxygen (or better the Makefile in latex) generates? I just added an option to doxygen, copying the refman.pdf to a location of your choice. (Hopefully it soon get’s merged and released).

If you want to test it out? Compile doxygen from my doxygen fork and add the following option to the Doxyfile of your project.

PDF_DST_FILE = ../MyGenerated.pdf

The destination is relative to your Makefile in your doxygen latex folder. As soon as make finished it’s job, the PDF is just in the same folder, the latex folder resides in.

That’s all. Enjoy generating software documentation with doxygen

FreeRTOS debugging on STM32 – CPU usage


Since the information about FreeRTOS debugging with STM32CubeIDE is sparse and ST is not yet providing the task list view (that was part of the Atollic TrueStudio), here is, how you get it by installing a plugin from freescale and adding the approprite stuff to your code. I assume, you already have a project with FreeRTOS setup and running…

Adding the plugins

First start STM32CubeIDE and go to Help -> Install New Software…

Then add an Update Site by clicking the „Manage“-Button. Here you need to add the update site from freescale. And yes, NXP/Freescales plugin works with STM’s CubeIDE 🙂


„Apply and Close“ and select the new site to „Work with“

Select the FreeRTOS Task Aware Debugger for GDB.

And click Next… Follow the Wizard until complete and after installation, restart your STM32CubeIDE.

Configuring the FreeRTOS project

Now add a timer and configure a reasonably a high tick rate (e.g. I used TIM13 of my STM32F469, running with 180 MHz HCLK, 90 MHz APB1 Timer clock and a timer counter period of 899 -> 100 kHz resolution).

Enable the interrupt

And in Middleware -> FreeRTOS, enable the run-time stats

If you like, you can also enable RECORD_STACK_HIGH_ADDRESS. Sometimes this is quite useful and avoids the little warning symbol in stack usage column of task list view.

Now regenerate your project…

Adjusting the code

Now it’s time to adjust your code for collecting the stats. Add a line for starting the timer in IT-mode by adding a function in some user code section in main.c.

volatile unsigned long ulHighFrequencyTimerTicks;

void configureTimerForRunTimeStats(void) {
  ulHighFrequencyTimerTicks = 0;

unsigned long getRunTimeCounterValue(void) {
  return ulHighFrequencyTimerTicks;

In stm32f4xx_it.c, add the following lines to the appropriate user sections

extern volatile unsigned long ulHighFrequencyTimerTicks;


void TIM8_UP_TIM13_IRQHandler(void)

If you are compiling with optimization levels above -O0, you also need to fix a bug (it is one in my opinion) in freertos tasks.c.

There are two possibilities:

  1. Switch of optimizations for tasks.c by right clicking on the file in project browser and changing the compiler optimization to -O0
  2. Change the line in tasks.c adding a volatile (see picture)

The problem with solution 2 is, that you need to do it after each STM32CubeMX code generation again. But there is a 3rd solution, that makes solution 2 persistend (until you update the MCU package).

Go to `%HOMEPATH%\STM32Cube\Repository\STM32Cube_FW_F4_V1.25.0\Middlewares\Third_Party\FreeRTOS\Source\` and edit the file like in solution 2, adding a volatile statement.

When you regenerate your project from CubeMX, it will include the correct line.

Profiling in action

Now after you put everything in place, it is time to run your code. Start the project in debugging mode, make the FreeRTOS/Task List view visible and let it run for some seconds. Then hit the pause button. The task list will collect the information from your target (from GDB) and show it nicely:

If the Task List view complains about FreeRTOS not have being detected, restart STM32CubeIDE and it should show up again.

Edit: During my last weeks of uing this Eclipse plugin, I had some problems seeing all tasks in Task Analyzer. In fact, the FreeRTOS functions to print the run-time statistics show them, while the plugin doesn’t. Also the data seems to be currupted sometimes within the plugin. So I would suggest, to better use the FreeRTOS internal stuff: https://www.freertos.org/rtos-run-time-stats.html


The information was collected from these links:

Penmount PCI Touch Controllers And I2C – Lost In Space


After I worked quite a lot with Touchnetix touch controllers some month ago, I now had a project using a PM2204 from Pemount (Salt). The datasheet of Touchnetix controllers (disclosed only with NDA) consists of several hundred pages defining a huge amount of objects for configuration and infromation retrieval purposes. In the end, you access these objects through dynamic register sets… But this is another story. This is a story about simplicity 😛


With penmount, you get the direct opposite of the Touchnetix. No documentation amywhere and only 6 bytes of data through UART or I2C. Yes, thats right. And its a one way communication (Edit: It can be 2-way, but even the linux kernel driver for UART ignores this fact.). Nothing to configure, nothing you can do wrong… With I2C, you send a read request to the address of the PM2204 and you will receive 6 bytes, when there has been a touch event. If not, you will receive 6 times 0xEE.

So, the best approach is, to watch out for an interrupt and when it occurs, polling these 6 bytes. They contain the event (1 byte), the position (2 x 2 byte) and a checksum (1 byte). And here is the piece for decoding it (some spices for error checking HAL should be added…):

uint32_t total;
uint8_t buf[6];

HAL_I2C_Master_Receive(&i2c1, 0x70, buf, 6, 100);

btn = buf[0] & 0x40;
xpos = ((buf[2] << 8) | buf[1]) * SCREEN_X_SIZE / 2048;
ypos = ((buf[4] << 8) | buf[3]) * SCREEN_Y_SIZE / 2048;
checksum = buf[5];

for (int i = 0; i < 5; i++) 
  total += buf[i]; 

if (checksum == (unsigned char) ~(total & 0xff))

How I digged through it? I found the linux driver using UART communication here and just tried, if I2C behaves the same… after hours of trying to access registers like on a memory device…. 🙁

Thats all. Have fun with Penmount!


Sometimes, using CubeMX and the HAL, there is something missing. For SDRAM, it is the command sequence that need to be issued after initializing the FMC module. The SDRAM itself also needs some information on timing and refresh, so FMC and SDRAM getting friends.

Digging around the web again mostly shows the low-level solutions and hardly no solutions using CubeMX and HAL. Even the examples of the STM32F4 MCU package has only very irritating examples that could not have been generated by CubeMX. It seems to be again my turn. I already found a community post, that exactly points out the problem with it. The problem with the solution is, that it does not integrate safely into CubeMX generated code when simply copy pasted.

But first we look at the callback tree for initializing the SDRAM.

HAL callback structure for SDRAM

MX_FMC_Init()                 fmc.c
   HAL_SDRAM_Init()           stm32f4xx_hal_sdram.c
      HAL_SDRAM_MspInit()     fmc.c (stm32f4xx_hal_sdram.c)
         HAL_FMC_MspInit()    fmc.c
      FMC_SDRAM_Init()        stm32f4xx_ll_fmc.c
      FMC_SDRAM_TimingInit()  stm32f4xx_ll_fmc.c

And the bug of CubeMX is, not to provide

  • a callback or user section at the end of MX_FMC_Init() or
  • a user section at the end of fmc.c (or at least somewhere after SDRAM handler definition)
  • a callback in FMC_SDRAM_TimingInit()

Welp, without having any of these, we need to cheat a bit. The SDRAM handler is already declared external by CubeMX in fmc.h, which in turn is included in fmc.c itself. Honestly, I find this not very clean, because it exposes the internal data stuctures to the outer code. But this is only the opinion of a clean code enthusiast… Since we have neither a callback nor a top level user code section in fmc.c, following the declaration of the SDRAM handler, we go this dirty way. Just below the peripheral initialization, there is a user code section generated by CubeMX. There we add our init sequence function call.


The init sequence function, we will add in a user code section of fmc.c looks like this (it is for SDRAM part IS42S32800G-6BLI).

define SDRAM_MODEREG_BURST_LENGTH_1             ((uint16_t)0x0000)
define SDRAM_MODEREG_BURST_LENGTH_2             ((uint16_t)0x0001)
define SDRAM_MODEREG_BURST_LENGTH_4             ((uint16_t)0x0002)
define SDRAM_MODEREG_BURST_LENGTH_8             ((uint16_t)0x0004)
define SDRAM_MODEREG_BURST_TYPE_SEQUENTIAL      ((uint16_t)0x0000)
define SDRAM_MODEREG_BURST_TYPE_INTERLEAVED     ((uint16_t)0x0008)
define SDRAM_MODEREG_CAS_LATENCY_2              ((uint16_t)0x0020)
define SDRAM_MODEREG_CAS_LATENCY_3              ((uint16_t)0x0030)
define SDRAM_MODEREG_OPERATING_MODE_STANDARD    ((uint16_t)0x0000)
define SDRAM_MODEREG_WRITEBURST_MODE_SINGLE     ((uint16_t)0x0200)

void MX_SDRAM1_InitSequence(uint32_t RefreshCount, uint32_t timeout)
  __IO uint32_t tmpmrd = 0;
  static FMC_SDRAM_CommandTypeDef Command;

  /* Step 1: Configure a clock configuration enable command */
  Command.CommandMode            = FMC_SDRAM_CMD_CLK_ENABLE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 2: Insert 100 us minimum delay */
  // Inserted delay is equal to 1 ms due to systick time base unit (ms) 

  /* Step 3: Configure a PALL (precharge all) command */
  Command.CommandMode            = FMC_SDRAM_CMD_PALL;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 4: Configure an Auto Refresh command */
  Command.CommandMode            = FMC_SDRAM_CMD_AUTOREFRESH_MODE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 8;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 5: Program the external memory mode register */
  tmpmrd = (uint32_t)SDRAM_MODEREG_BURST_LENGTH_1 |\
  Command.CommandMode            = FMC_SDRAM_CMD_LOAD_MODE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = tmpmrd;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 6: Set the refresh rate counter */
  HAL_SDRAM_ProgramRefreshRate(&hsdram1, RefreshCount);

Just to make it complete, here are the settings for FMC for the specific part, taken from MX_FMC_Init(), generated by CubeMX:

hsdram1.Instance = FMC_SDRAM_DEVICE;

/* hsdram1.Init */ 
hsdram1.Init.SDBank             = FMC_SDRAM_BANK1; 
hsdram1.Init.ColumnBitsNumber   = FMC_SDRAM_COLUMN_BITS_NUM_8;
hsdram1.Init.RowBitsNumber      = FMC_SDRAM_ROW_BITS_NUM_12; 
hsdram1.Init.MemoryDataWidth    = FMC_SDRAM_MEM_BUS_WIDTH_32; 
hsdram1.Init.InternalBankNumber = FMC_SDRAM_INTERN_BANKS_NUM_4; 
hsdram1.Init.CASLatency         = FMC_SDRAM_CAS_LATENCY_1; 
sdram1.Init.WriteProtection     = FMC_SDRAM_WRITE_PROTECTION_DISABLE; 
hsdram1.Init.SDClockPeriod      = FMC_SDRAM_CLOCK_PERIOD_2; 
hsdram1.Init.ReadBurst          = FMC_SDRAM_RBURST_DISABLE; 
hsdram1.Init.ReadPipeDelay      = FMC_SDRAM_RPIPE_DELAY_0; 

/* SdramTiming */
SdramTiming.LoadToActiveDelay    = 2;
SdramTiming.ExitSelfRefreshDelay = 7;
SdramTiming.SelfRefreshTime      = 4;
SdramTiming.RowCycleDelay        = 7;
SdramTiming.WriteRecoveryTime    = 3;
SdramTiming.RPDelay              = 2;
SdramTiming.RCDDelay             = 2;

Even if SDRAM_HandleTypeDef hsdram1; is declared below our init function, it can be used in the upper user code section of fmc.c, because it is declared external in fmc.h, which is included just ahead of the user code section. Again not very clean, but it works.

My Two Cents About HAL…

And for all of you crying that HAL is way to oversized for STM32 projects and everybody needs to know about all registers of the devices. Maybe you are right, but this is no excuse for writing bad code. In my opinion, HAL does a very good job at keeping cohesion high and coupling low. The main problems with it originate from CubeMX templates, that break some of the rules for good coding. Would it use more techniques like first class abstract data types and ensure, the extension points (user code sections, callbacks) are sufficient, it would guide more people to a well designed architecture. Currently, programmers are forced to break the boundaries too often. The same applies e.g. for my other post about UART continuous receive

But instead of coding low-level code that is badly evolvable (can not be extended, reused, adjusted or understand easily), I think it is better to find a clean solution to keep most of CubeMX’s code as is and use the extension points it provides. I believe it is of much higher value having projects all integrating with CubeMX nicely and therefore can be understood easily by other parties (e.g. a new team mate or the guy taking over your project), than reinventing the wheel over and over again using LL functions or direct register access. STM32 have enough space and speed to cope with a little overhead. And exactly here applies another rule of thumb from clean coding guidelines: Never try to speed optimize without profiling. This means, use a profiler to identify the critical pieces of your code and only if needed, invest time to optimize the critical parts.

Installing BigBlueButton on Your Dedicated Server


After struggling with a dedicated server from Strato Webhosting, running ubuntu 18.04 and playing around with schroot to get some ubuntu 16.04 environment, I gave up with this solution. The systemd hurdle was much to high to be taken within the available time I currently have.

Finding the right server

The first act was to find a suitable dedicated server throughout various providers from cheap to expensive. There have been only a few but hard requirements: German location, 4+ Cores, 8+ GB RAM, 100+ GB SSD, 400+ MBit internet connection, 1+ TB Traffic. And the hardest requirement: Ubuntu 16.04

At hetzner I found a german provider. A bit expensive in general, but there is also some server auction section where you could find really valuable servers with great support and a nice price tag. They start at around 30€/month and they include almost everything you could dream about.

Long story short, I ordered a dedicated server from hetzner, located in Germany (Falkenstein, FSN1), and installed some ubuntu 16.04 using the rescue system. The automated installer of the rescue console offers (possibly) all available Linux derivatives. From the Hetzner Robot you can get already a hand full of supported distributions, but through the rescue system, there are many supported distros more to be chosen, not talking about the available unsupported ones.

The whole procedure took around 20-30 minutes from customer registration through ordering the server to get it running. The hetzner-wiki is crystal clear, the installer is easy, the whole stuff is rock stable.


Setting up the server

To setup the server, you usually need to wait a few minutes, until the server shows up in the so-called Hetzner Robot. Here you can choose to start the rescue system with your SSH pubkey to be deployed (details at hetzner wiki)…

…and then do an automated reset.

Then log in the rescue system with SSH and enter installimage.

This command issued, a menu will appear and guide you through the install procedure of the operationg system of your choice (for BigBlueButton you need ubuntu-16-04-minimal). Amazingly, there is a ton of hetzner supported systems, but a megaton of unsupported but available derivatives.

Welp, this menuconfig-style tool is a bit old-school, but it serves it’s purpose so well… I’m really fascinated. And within 5 minutes, the system is poured on your server’s hard disk. When finished, type reboot and the server will boot the new operating system.

Setting up BigBlueButton…

…with Let’s Entcrypt SSL, Greenlight and (almost) the whole configuration.

Just do a system update/upgrade…

apt update
apt upgrade

… and then you can start over installing bbb…

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh |\
    bash -s -- -v xenial-220 -s bbb.example.com \
    -e info@example.com -g

That’s it… When you change your Greenlight config in ~/greenlight/.env (e.g. for enabling Google OAuth2), just follow this procedure:

cd ~/greenlight
vi .env   # do there what you want or need
docker-compose down
docker-compose up -d

If you see a 404 Error when loading your page (https://bbb.example.com), just give it 30 seconds (or more if you did not follow the system requirements 🙂 ) to come up and enjoy your conferences.

My first real world experiences

After some days, I had a „real world“ video conference with 20 attendees, all using audio and 14 of them video. It was the first virtual classroom meeting of my daughter, related to my School’s out post. Most of the attendees had been crystal clear without interruptions. Some of them had minor audio and/or video stuttering and two of them I could hardly understand. OK, there have been a few that had problems with their own hardware (video and audio), but this was not related to BBB.

The stuttering connection originates mainly from WiFi connections, loosing and delaying data packets. So, if you don’t have a chance to get wired, get as close as you can to your WiFi hotspot. There can also be a big difference between cheap consumer equipment and the professional one. I run some UniFi based installation from ubiquity networks that simply outperforms the FritzBox WiFi in every aspect (speed, reliability, configurability, VLAN capability, … and so on)

Also watching htop for a while gives some interesting insights what BBB expects from your server. With 20 attendees and many having the microphone turned on and talking, the load rises astonishingly. My quad-core core-i7 (8 virtual HT CPUs, see server details) was pushed up to 30% per cpu during this meeting. This mostly originated from freeswitch and kurento-media-server:

This means two things:

  1. It takes quite a lot of power to mix audio in realtime.
  2. The freeswitch code seems to make very efficient use of multiple processors

That’s both, good and bad news for large installations. Bigger hardware with more cores is a constructive solution. But honestly, if you are planning to put more than 100 concurrent users on a server in a production environment, you should think about high-availabilty solutions (keywords: AWS Elastic Load Balancer, AWS RDS with failover and availability zones,…). But then, the bbb-install.sh approach get’s to it’s limits. However, it serves well as a starting point to get up a test system and to understand BBB’s and Greenlight’s architecture.

Some words about the client

When you know Zoom Meetings, which is a bit bullheaded about installing an executable and you need to rick it to get the browser version of the client, you will find BBB a real pleasure.

To use the BBB server, there is nothing needed beside a web browser. It supports audio, video and screen sharing (screen or application based) and it supports uploading a presentation or watching a web video (youtube, vimeo,..) together while the moderator controls the content. I tested it already with a few friends and we found out, that it works best with Chrome and Firefox on Windows and Linux. But even on smartphones it runs fine, as long as you have a high quality internet connection.

What next?

BigBlueButton advises to not install any other stuff beside BBB (and maybe Greenlight) on the Server because of the realtime audio processing in FreeSwitch, every little delay can destroy the user experience of your video conference.

In fact, if you have enough cores and RAM and running on a NVMe-RAID 1, you can for sure install other web applications on your server, if you don’t do heavy number crunching. If you can live with the risk, that the other applications potentially influence the conference quality, there is no technical reason not to do so.

The configuration of nginx looks very clear and straight forward. You should be able to add another site (in /etc/nginx/sites-available) and activate it (soft-link it to /etc/nginx/sites-enabled/). The only thing I would advise is to run it on another subdomain (otherapp.example.com). For more encapsulation and also with a bit more effort, you can go with a second IP or whatever you like…

Backup thoughts

With a dedicated server, you should keep your backup always in mind. It is not simply creating a snapshot like in virtualized environments. If you build up on e.g. ext4, your backup might brake. If you want to create an online snapshot backup of your system block devices, better stick with an LVM base for your block devices or, if you like the bleeding edge, with btrfs.

Doing it the safe way

The safest way for sure is to stop all services and take a snapshot then. The most secure is to boot some rescue system, fetch the raw disk image snapshot and then boot back in the system. This is safe, but it could mean, that you need to do it manually. Hetzner is providing information for its Robot-API (the one that Hetzner Robot is using), but it could mean quite some work to implement it… and you need a second server (or a RaspberryPi at your home) that steers the backup process using the Robot-API and SSH commands (intersting thought BTW, maybe I’ll try it and write some new post about it one day)…

Doing it the less safe way

But there is also a quite safe way of doing your backup with all services running. There are five different parts to be considered.

  1. The system environment (e.g. /boot –> block device based)
  2. The Home Directores (e.g. /root –> file based)
  3. The static system files (e.g. /usr/share –> file based)
  4. The variable system files (e.g. /var/lib –> file based)
  5. The database directories (dynamic –> dump based)

With this said, the process should be quite clear.

Firstly, you should backup your whole system (finished installing your services) on block device level using a rescue system, just to make sure, you can easily revert to a running state or bind-mount the block device snapshot to compare with your file based backups.

Secondly, dump your database files to place a like root’s home directory, so the content gets somewhat like static content. Usually transaction based databases dump a consistent state. It depends on the software, how well the transactions are formed, but usually this should at least keep the database itself intact.

Thirdly, backup the files from all other directories (exluding temporary ones) using a efficient approach (e.g. rsync, rsnapshot or borg) over to a safe place.

And last but not least, check your backup consientiously, or better, try a restore. For a production environment, a regular and automated restore test is indispensable. Being honestly, for a real production environment with hundreds of users, a simple dedicated server is not the right way to go.

Doing it the Chuck Norris way

A quite risky, but mostly working way, is to run a snapshot tool simply on the running system. As it is stated on the borg page, most directories are stable enough to be backed up using an rsync approach. If you do your first backup (that takes quite long) with all services stopped and then only do incremental backups (being lightning fast), there is a low chance of e.g. a home directory changing during the backup process. To avoid getting inconsistent databases, just do a new dump of your databases beforehand (e.g. to backup user’s home). You can even prevent more of the risky parts (using LVM, btrfs) to lower the chance for unexpected changes. And moreover you can stop the relevant services.

For the docker containers (postgres and greenlight), you need to keep in mind, that it is a good practice for docker services to store their data in a bind mount that is kept in the host system. This means, that the thoughts on the database dumps can not be relaxed and may get even more complicated.

Installing Moodle on the BBB-Server

Next step is to install Moodle on this server and migrate all the users and courses from the old one (see my post about it’s installation) to the new one (stay tuned).

Have fun with BBB and Greenlight 🙂

How To Debug Hardware-Faults on Your Dedicated Server

While installing some stuff on a dedicated server at Strato, I encountered a problem with the server two times. While having no clue what happened during the first time, I was prepared during the second. I kept a connection to the serial port proxy of my provider open, to see the most low level messages (kprints).

And you won’t believe, after 13295 seconds of uptime (3 hours 41 minutes), the kernel had a panic:

This is a great indication to run a Hardware-Test and to address the support of your hosting provider.