Home

STM32 USB-DFU

I’m not sure, if I’m simply a problem magnet or why some stuff does not work as described… Here is another case. The tool I tend to use can be found here at STM’s site.

When I connected a piece of custom designed hardware with my laptops USB port with BOOT0 tied to VCC, it immediately showed up a new USB-Device called „STM32 Bootloader“ in group „USB Devices“. I was really happy about that and started the DFuSe Demo Software from STM. But hey, it could not find an appropriate device. What the heck???

After some digging through the web, I found different suggestions and problem solutions, but none of it worked.The simple solution was, to uninstall the device’s driver in „Device Manager“, unconnect and reconnet again. A different device showed up just a goup above the previous one: „USB Controller“.

Welp, as soon as the device showed up, also the DFuSe Demo Software recognized the device and stepped into life.

To create a DFU file is well described in all the other resources on the web. Just use the DFU file manager that was installed along with the DFuSe Demo Software to create a DFU file out of e.g. a hex file.

Then press the lower „Choose…“ button to select the generated DFU file and press „Upgrade“… That’s all. Your device has a new Firmware on it.

Have fun!

STM32 FreeRTOS and printf with floats

This will be a very short post. If you experience hard faults when using printf with floats and you already ticked the appropriate settings in the project’s properties…

…don’t waste your time digging through assembler instructions with instruction stepping (like I did) just to realize, that memory management is broken when using FreeRTOS. It is simply a bug in CubeMX-generated source files. Locate your _sbrk-function (either in syscalls.c or in sysmem.c) and change it to the following:

caddr_t _sbrk(int incr)
{
  extern char end asm("end");
  static char *heap_end;
  char *prev_heap_end,*min_stack_ptr;
  if (heap_end == 0) heap_end = &end; prev_heap_end = heap_end; 
    /* Use the NVIC offset register to locate the main stack pointer. */
    /* Locate the STACK bottom address */
    min_stack_ptr = (char*)(*(unsigned int *)*(unsigned int *)0xE000ED08);
    min_stack_ptr -= MAX_STACK_SIZE; if (heap_end + incr > min_stack_ptr) {
    errno = ENOMEM;
    return (caddr_t) -1;
  }
  heap_end += incr; return (caddr_t) prev_heap_end;
}

For what _sbrk does, have a look here.

If you want to digg a bit deeper, here are some websites dealing with this problem:

ESP32-EVB and platform.io – Yet Another ESP32 tutorial

What the heck? Aren’t there enough toturials out there about ESP32? I believe: Yes, too many. And there are too many that struggle with setting up the arduino environment for non-arduino hardware. But there is a straight-forward solution that works out of the box: platform.io or also known as PIO.

And there are enought tutorials about installing platform.io. It is as easy as you can think. Install Visual Studio Code from Microsoft and install the PIO plugin in VSCode.

Then create a new project VSCode -> New Project and select Olimex ESP32-EVB with Arduino Platform. After PIO downloaded all dependencies and configured your project, we are ready to go!

Hello World – Relay Toggle

Open main.cpp and edit it, so it will look like that:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

…connect your Olimex ESP32-EVB and hit the upload button:

Done! Your Relay should toogle every 10 seconds.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code.

Extend your code to look like this:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  Serial.begin(115200);
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  Serial.printf("Switch on\r\n");
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  Serial.printf("Switch off\r\n");
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

Add two lines to your platfromio.ini file. You will find the correct COM port in terminal when uploading the new code to your device.

In my case it is COM6 and platformio.ini looks like this:

[env:esp32-evb]
platform = espressif32
board = esp32-evb
framework = arduino
monitor_port = COM6
monitor_speed = 115200

Upload your code again after changing the ini file. If it does not print your output automatically to your terminal, start the monitor manually:

From now on, you should see the printf output in your terminal every time you upload your code again:

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or whatever. But now, you have some great arduino IDE without arduino IDE 😛

Have fun coding!

Doxygen – Tips and Tricks

LaTeX non-interactive

To make LaTeX skip some errors without user interaction, you can add the option --interaction=nonstopmode to the pdflatex call. Easiest way to do so, is changing the LATEX_COMMAND_NAME in your Doxyfile.

LATEX_CMD_NAME = „latex –interaction=nonstopmode“

Do not forget the double quotation marks. Otherwise doxygen will remove the space and the command in your make.bat will fail.

If you now want to generate the code, step into your doxygen-generated latex folder (designated by LATEX_OUTPUT option in Doxyfile) and execute make.bat (on Windows) or make all (on *nix).

Adding a favicon to html output

Adding a favicon to html output, you need to specify it in a custom header and include the original image in HTML, as described here. To extract the default header file:

doxygen -w html headerFile

Add the follwing line to in headerFile within the html header

<link rel="shortcut icon" href="favicon.png" type="image/png">

And add your headerFile and the image to the HTML_EXTRA_FILES in your Doxyfile. Its path is relative to your Doxyfile.

HTML_HEADER = headerFile
HTML_EXTRA_FILES = some_rel_path/favicon.png

Now you can generate your html documentation with some favicon in place.

PDF output destination

Did you ever search for the PDF file, doxygen (or better the Makefile in latex) generates? I just added an option to doxygen, copying the refman.pdf to a location of your choice. (Hopefully it soon get’s merged and released).

If you want to test it out? Compile doxygen from my doxygen fork and add the following option to the Doxyfile of your project.

PDF_DST_FILE = ../MyGenerated.pdf

The destination is relative to your Makefile in your doxygen latex folder. As soon as make finished it’s job, the PDF is just in the same folder, the latex folder resides in.

That’s all. Enjoy generating software documentation with doxygen

FreeRTOS debugging on STM32 – CPU usage

Introduction

Since the information about FreeRTOS debugging with STM32CubeIDE is sparse and ST is not yet providing the task list view (that was part of the Atollic TrueStudio), here is, how you get it by installing a plugin from freescale and adding the approprite stuff to your code. I assume, you already have a project with FreeRTOS setup and running…

Adding the plugins

First start STM32CubeIDE and go to Help -> Install New Software…

Then add an Update Site by clicking the „Manage“-Button. Here you need to add the update site from freescale. And yes, NXP/Freescales plugin works with STM’s CubeIDE 🙂

http://freescale.com/lgfiles/updates/Eclipse/KDS

„Apply and Close“ and select the new site to „Work with“

Select the FreeRTOS Task Aware Debugger for GDB.

And click Next… Follow the Wizard until complete and after installation, restart your STM32CubeIDE.

Configuring the FreeRTOS project

Now add a timer and configure a reasonably a high tick rate (e.g. I used TIM13 of my STM32F469, running with 180 MHz HCLK, 90 MHz APB1 Timer clock and a timer counter period of 899 -> 100 kHz resolution).


Enable the interrupt

And in Middleware -> FreeRTOS, enable the run-time stats

If you like, you can also enable RECORD_STACK_HIGH_ADDRESS. Sometimes this is quite useful and avoids the little warning symbol in stack usage column of task list view.

Now regenerate your project…

Adjusting the code

Now it’s time to adjust your code for collecting the stats. Add a line for starting the timer in IT-mode by adding a function in some user code section in main.c.

volatile unsigned long ulHighFrequencyTimerTicks;

void configureTimerForRunTimeStats(void) {
  ulHighFrequencyTimerTicks = 0;
  HAL_TIM_Base_Start_IT(&htim13);
}

unsigned long getRunTimeCounterValue(void) {
  return ulHighFrequencyTimerTicks;
}

In stm32f4xx_it.c, add the following lines to the appropriate user sections

/* USER CODE BEGIN EV */
extern volatile unsigned long ulHighFrequencyTimerTicks;
/* USER CODE END EV */

[...]

void TIM8_UP_TIM13_IRQHandler(void)
{
  /* USER CODE BEGIN TIM8_UP_TIM13_IRQn 0 */
  ulHighFrequencyTimerTicks++;
  /* USER CODE END TIM8_UP_TIM13_IRQn 0 */
  HAL_TIM_IRQHandler(&htim13); 
  /* USER CODE BEGIN TIM8_UP_TIM13_IRQn 1 */
  /* USER CODE END TIM8_UP_TIM13_IRQn 1 */
}

If you are compiling with optimization levels above -O0, you also need to fix a bug (it is one in my opinion) in freertos tasks.c.

There are two possibilities:

  1. Switch of optimizations for tasks.c by right clicking on the file in project browser and changing the compiler optimization to -O0
  2. Change the line in tasks.c adding a volatile (see picture)

The problem with solution 2 is, that you need to do it after each STM32CubeMX code generation again. But there is a 3rd solution, that makes solution 2 persistend (until you update the MCU package).

Go to `%HOMEPATH%\STM32Cube\Repository\STM32Cube_FW_F4_V1.25.0\Middlewares\Third_Party\FreeRTOS\Source\` and edit the file like in solution 2, adding a volatile statement.

When you regenerate your project from CubeMX, it will include the correct line.

Profiling in action

Now after you put everything in place, it is time to run your code. Start the project in debugging mode, make the FreeRTOS/Task List view visible and let it run for some seconds. Then hit the pause button. The task list will collect the information from your target (from GDB) and show it nicely:

If the Task List view complains about FreeRTOS not have being detected, restart STM32CubeIDE and it should show up again.

Citations

The information was collected from these links:

Penmount PCI Touch Controllers And I2C – Lost In Space

Introduction

After I worked quite a lot with Touchnetix touch controllers some month ago, I now had a project using a PM2204 from Pemount (Salt). The datasheet of Touchnetix controllers (disclosed only with NDA) consists of several hundred pages defining a huge amount of objects for configuration and infromation retrieval purposes. In the end, you access these objects through dynamic register sets… But this is another story. This is a story about simplicity 😛

TL;DR

With penmount, you get the direct opposite of the Touchnetix. No documentation amywhere and only 6 bytes of data through UART or I2C. Yes, thats right. And its a one way communication (Edit: It can be 2-way, but even the linux kernel driver for UART ignores this fact.). Nothing to configure, nothing you can do wrong… With I2C, you send a read request to the address of the PM2204 and you will receive 6 bytes, when there has been a touch event. If not, you will receive 6 times 0xEE.

So, the best approach is, to watch out for an interrupt and when it occurs, polling these 6 bytes. They contain the event (1 byte), the position (2 x 2 byte) and a checksum (1 byte). And here is the piece for decoding it (some spices for error checking HAL should be added…):

uint32_t total;
uint8_t buf[6];

HAL_I2C_Master_Receive(&i2c1, 0x70, buf, 6, 100);

btn = buf[0] & 0x40;
xpos = ((buf[2] << 8) | buf[1]) * SCREEN_X_SIZE / 2048;
ypos = ((buf[4] << 8) | buf[3]) * SCREEN_Y_SIZE / 2048;
checksum = buf[5];

for (int i = 0; i < 5; i++) 
  total += buf[i]; 

if (checksum == (unsigned char) ~(total & 0xff))
  DO_SUCCESS_STUFF;
else
  RAISE_AN_ERROR;

How I digged through it? I found the linux driver using UART communication here and just tried, if I2C behaves the same… after hours of trying to access registers like on a memory device…. 🙁

Thats all. Have fun with Penmount!

STM32CubeMX and SDRAM

Sometimes, using CubeMX and the HAL, there is something missing. For SDRAM, it is the command sequence that need to be issued after initializing the FMC module. The SDRAM itself also needs some information on timing and refresh, so FMC and SDRAM getting friends.

Digging around the web again mostly shows the low-level solutions and hardly no solutions using CubeMX and HAL. Even the examples of the STM32F4 MCU package has only very irritating examples that could not have been generated by CubeMX. It seems to be again my turn. I already found a community post, that exactly points out the problem with it. The problem with the solution is, that it does not integrate safely into CubeMX generated code when simply copy pasted.

But first we look at the callback tree for initializing the SDRAM.

HAL callback structure for SDRAM

MX_FMC_Init()                 fmc.c
   HAL_SDRAM_Init()           stm32f4xx_hal_sdram.c
      HAL_SDRAM_MspInit()     fmc.c (stm32f4xx_hal_sdram.c)
         HAL_FMC_MspInit()    fmc.c
      FMC_SDRAM_Init()        stm32f4xx_ll_fmc.c
      FMC_SDRAM_TimingInit()  stm32f4xx_ll_fmc.c

And the bug of CubeMX is, not to provide

  • a callback or user section at the end of MX_FMC_Init() or
  • a user section at the end of fmc.c (or at least somewhere after SDRAM handler definition)
  • a callback in FMC_SDRAM_TimingInit()

Welp, without having any of these, we need to cheat a bit. The SDRAM handler is already declared external by CubeMX in fmc.h, which in turn is included in fmc.c itself. Honestly, I find this not very clean, because it exposes the internal data stuctures to the outer code. But this is only the opinion of a clean code enthusiast… Since we have neither a callback nor a top level user code section in fmc.c, following the declaration of the SDRAM handler, we go this dirty way. Just below the peripheral initialization, there is a user code section generated by CubeMX. There we add our init sequence function call.

MX_SDRAM1_InitSequence(SDRAM_REFRESH_COUNT, SDRAM_TIMEOUT);

The init sequence function, we will add in a user code section of fmc.c looks like this (it is for SDRAM part IS42S32800G-6BLI).

define SDRAM_MODEREG_BURST_LENGTH_1             ((uint16_t)0x0000)
define SDRAM_MODEREG_BURST_LENGTH_2             ((uint16_t)0x0001)
define SDRAM_MODEREG_BURST_LENGTH_4             ((uint16_t)0x0002)
define SDRAM_MODEREG_BURST_LENGTH_8             ((uint16_t)0x0004)
define SDRAM_MODEREG_BURST_TYPE_SEQUENTIAL      ((uint16_t)0x0000)
define SDRAM_MODEREG_BURST_TYPE_INTERLEAVED     ((uint16_t)0x0008)
define SDRAM_MODEREG_CAS_LATENCY_2              ((uint16_t)0x0020)
define SDRAM_MODEREG_CAS_LATENCY_3              ((uint16_t)0x0030)
define SDRAM_MODEREG_OPERATING_MODE_STANDARD    ((uint16_t)0x0000)
define SDRAM_MODEREG_WRITEBURST_MODE_PROGRAMMED ((uint16_t)0x0000)
define SDRAM_MODEREG_WRITEBURST_MODE_SINGLE     ((uint16_t)0x0200)

void MX_SDRAM1_InitSequence(uint32_t RefreshCount, uint32_t timeout)
{
  __IO uint32_t tmpmrd = 0;
  static FMC_SDRAM_CommandTypeDef Command;

  /* Step 1: Configure a clock configuration enable command */
  Command.CommandMode            = FMC_SDRAM_CMD_CLK_ENABLE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 2: Insert 100 us minimum delay */
  // Inserted delay is equal to 1 ms due to systick time base unit (ms) 
  HAL_Delay(1);

  /* Step 3: Configure a PALL (precharge all) command */
  Command.CommandMode            = FMC_SDRAM_CMD_PALL;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 4: Configure an Auto Refresh command */
  Command.CommandMode            = FMC_SDRAM_CMD_AUTOREFRESH_MODE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 8;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 5: Program the external memory mode register */
  tmpmrd = (uint32_t)SDRAM_MODEREG_BURST_LENGTH_1 |\
    SDRAM_MODEREG_BURST_TYPE_SEQUENTIAL |\
    SDRAM_MODEREG_CAS_LATENCY_2 |\
    SDRAM_MODEREG_OPERATING_MODE_STANDARD |\
    SDRAM_MODEREG_WRITEBURST_MODE_SINGLE;
  Command.CommandMode            = FMC_SDRAM_CMD_LOAD_MODE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = tmpmrd;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 6: Set the refresh rate counter */
  HAL_SDRAM_ProgramRefreshRate(&hsdram1, RefreshCount);
}

Just to make it complete, here are the settings for FMC for the specific part, taken from MX_FMC_Init(), generated by CubeMX:

hsdram1.Instance = FMC_SDRAM_DEVICE;

/* hsdram1.Init */ 
hsdram1.Init.SDBank             = FMC_SDRAM_BANK1; 
hsdram1.Init.ColumnBitsNumber   = FMC_SDRAM_COLUMN_BITS_NUM_8;
hsdram1.Init.RowBitsNumber      = FMC_SDRAM_ROW_BITS_NUM_12; 
hsdram1.Init.MemoryDataWidth    = FMC_SDRAM_MEM_BUS_WIDTH_32; 
hsdram1.Init.InternalBankNumber = FMC_SDRAM_INTERN_BANKS_NUM_4; 
hsdram1.Init.CASLatency         = FMC_SDRAM_CAS_LATENCY_1; 
sdram1.Init.WriteProtection     = FMC_SDRAM_WRITE_PROTECTION_DISABLE; 
hsdram1.Init.SDClockPeriod      = FMC_SDRAM_CLOCK_PERIOD_2; 
hsdram1.Init.ReadBurst          = FMC_SDRAM_RBURST_DISABLE; 
hsdram1.Init.ReadPipeDelay      = FMC_SDRAM_RPIPE_DELAY_0; 

/* SdramTiming */
SdramTiming.LoadToActiveDelay    = 2;
SdramTiming.ExitSelfRefreshDelay = 7;
SdramTiming.SelfRefreshTime      = 4;
SdramTiming.RowCycleDelay        = 7;
SdramTiming.WriteRecoveryTime    = 3;
SdramTiming.RPDelay              = 2;
SdramTiming.RCDDelay             = 2;

Even if SDRAM_HandleTypeDef hsdram1; is declared below our init function, it can be used in the upper user code section of fmc.c, because it is declared external in fmc.h, which is included just ahead of the user code section. Again not very clean, but it works.

My Two Cents About HAL…

And for all of you crying that HAL is way to oversized for STM32 projects and everybody needs to know about all registers of the devices. Maybe you are right, but this is no excuse for writing bad code. In my opinion, HAL does a very good job at keeping cohesion high and coupling low. The main problems with it originate from CubeMX templates, that break some of the rules for good coding. Would it use more techniques like first class abstract data types and ensure, the extension points (user code sections, callbacks) are sufficient, it would guide more people to a well designed architecture. Currently, programmers are forced to break the boundaries too often. The same applies e.g. for my other post about UART continuous receive

But instead of coding low-level code that is badly evolvable (can not be extended, reused, adjusted or understand easily), I think it is better to find a clean solution to keep most of CubeMX’s code as is and use the extension points it provides. I believe it is of much higher value having projects all integrating with CubeMX nicely and therefore can be understood easily by other parties (e.g. a new team mate or the guy taking over your project), than reinventing the wheel over and over again using LL functions or direct register access. STM32 have enough space and speed to cope with a little overhead. And exactly here applies another rule of thumb from clean coding guidelines: Never try to speed optimize without profiling. This means, use a profiler to identify the critical pieces of your code and only if needed, invest time to optimize the critical parts.

Installing BigBlueButton on Your Dedicated Server

Introduction

After struggling with a dedicated server from Strato Webhosting, running ubuntu 18.04 and playing around with schroot to get some ubuntu 16.04 environment, I gave up with this solution. The systemd hurdle was much to high to be taken within the available time I currently have.

Finding the right server

The first act was to find a suitable dedicated server throughout various providers from cheap to expensive. There have been only a few but hard requirements: German location, 4+ Cores, 8+ GB RAM, 100+ GB SSD, 400+ MBit internet connection, 1+ TB Traffic. And the hardest requirement: Ubuntu 16.04

At hetzner I found a german provider. A bit expensive in general, but there is also some server auction section where you could find really valuable servers with great support and a nice price tag. They start at around 30€/month and they include almost everything you could dream about.

Long story short, I ordered a dedicated server from hetzner, located in Germany (Falkenstein, FSN1), and installed some ubuntu 16.04 using the rescue system. The automated installer of the rescue console offers (possibly) all available Linux derivatives. From the Hetzner Robot you can get already a hand full of supported distributions, but through the rescue system, there are many supported distros more to be chosen, not talking about the available unsupported ones.

The whole procedure took around 20-30 minutes from customer registration through ordering the server to get it running. The hetzner-wiki is crystal clear, the installer is easy, the whole stuff is rock stable.

TL;DR

Setting up the server

To setup the server, you usually need to wait a few minutes, until the server shows up in the so-called Hetzner Robot. Here you can choose to start the rescue system with your SSH pubkey to be deployed (details at hetzner wiki)…

…and then do an automated reset.

Then log in the rescue system with SSH and enter installimage.

This command issued, a menu will appear and guide you through the install procedure of the operationg system of your choice (for BigBlueButton you need ubuntu-16-04-minimal). Amazingly, there is a ton of hetzner supported systems, but a megaton of unsupported but available derivatives.

Welp, this menuconfig-style tool is a bit old-school, but it serves it’s purpose so well… I’m really fascinated. And within 5 minutes, the system is poured on your server’s hard disk. When finished, type reboot and the server will boot the new operating system.

Setting up BigBlueButton…

…with Let’s Entcrypt SSL, Greenlight and (almost) the whole configuration.

Just do a system update/upgrade…

apt update
apt upgrade

… and then you can start over installing bbb…

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh |\
    bash -s -- -v xenial-220 -s bbb.example.com \
    -e info@example.com -g

That’s it… When you change your Greenlight config in ~/greenlight/.env (e.g. for enabling Google OAuth2), just follow this procedure:

cd ~/greenlight
vi .env   # do there what you want or need
docker-compose down
docker-compose up -d

If you see a 404 Error when loading your page (https://bbb.example.com), just give it 30 seconds (or more if you did not follow the system requirements 🙂 ) to come up and enjoy your conferences.

My first real world experiences

After some days, I had a „real world“ video conference with 20 attendees, all using audio and 14 of them video. It was the first virtual classroom meeting of my daughter, related to my School’s out post. Most of the attendees had been crystal clear without interruptions. Some of them had minor audio and/or video stuttering and two of them I could hardly understand. OK, there have been a few that had problems with their own hardware (video and audio), but this was not related to BBB.

The stuttering connection originates mainly from WiFi connections, loosing and delaying data packets. So, if you don’t have a chance to get wired, get as close as you can to your WiFi hotspot. There can also be a big difference between cheap consumer equipment and the professional one. I run some UniFi based installation from ubiquity networks that simply outperforms the FritzBox WiFi in every aspect (speed, reliability, configurability, VLAN capability, … and so on)

Also watching htop for a while gives some interesting insights what BBB expects from your server. With 20 attendees and many having the microphone turned on and talking, the load rises astonishingly. My quad-core core-i7 (8 virtual HT CPUs, see server details) was pushed up to 30% per cpu during this meeting. This mostly originated from freeswitch and kurento-media-server:

This means two things:

  1. It takes quite a lot of power to mix audio in realtime.
  2. The freeswitch code seems to make very efficient use of multiple processors

That’s both, good and bad news for large installations. Bigger hardware with more cores is a constructive solution. But honestly, if you are planning to put more than 100 concurrent users on a server in a production environment, you should think about high-availabilty solutions (keywords: AWS Elastic Load Balancer, AWS RDS with failover and availability zones,…). But then, the bbb-install.sh approach get’s to it’s limits. However, it serves well as a starting point to get up a test system and to understand BBB’s and Greenlight’s architecture.

Some words about the client

When you know Zoom Meetings, which is a bit bullheaded about installing an executable and you need to rick it to get the browser version of the client, you will find BBB a real pleasure.

To use the BBB server, there is nothing needed beside a web browser. It supports audio, video and screen sharing (screen or application based) and it supports uploading a presentation or watching a web video (youtube, vimeo,..) together while the moderator controls the content. I tested it already with a few friends and we found out, that it works best with Chrome and Firefox on Windows and Linux. But even on smartphones it runs fine, as long as you have a high quality internet connection.

What next?

BigBlueButton advises to not install any other stuff beside BBB (and maybe Greenlight) on the Server because of the realtime audio processing in FreeSwitch, every little delay can destroy the user experience of your video conference.

In fact, if you have enough cores and RAM and running on a NVMe-RAID 1, you can for sure install other web applications on your server, if you don’t do heavy number crunching. If you can live with the risk, that the other applications potentially influence the conference quality, there is no technical reason not to do so.

The configuration of nginx looks very clear and straight forward. You should be able to add another site (in /etc/nginx/sites-available) and activate it (soft-link it to /etc/nginx/sites-enabled/). The only thing I would advise is to run it on another subdomain (otherapp.example.com). For more encapsulation and also with a bit more effort, you can go with a second IP or whatever you like…

Backup thoughts

With a dedicated server, you should keep your backup always in mind. It is not simply creating a snapshot like in virtualized environments. If you build up on e.g. ext4, your backup might brake. If you want to create an online snapshot backup of your system block devices, better stick with an LVM base for your block devices or, if you like the bleeding edge, with btrfs.

Doing it the safe way

The safest way for sure is to stop all services and take a snapshot then. The most secure is to boot some rescue system, fetch the raw disk image snapshot and then boot back in the system. This is safe, but it could mean, that you need to do it manually. Hetzner is providing information for its Robot-API (the one that Hetzner Robot is using), but it could mean quite some work to implement it… and you need a second server (or a RaspberryPi at your home) that steers the backup process using the Robot-API and SSH commands (intersting thought BTW, maybe I’ll try it and write some new post about it one day)…

Doing it the less safe way

But there is also a quite safe way of doing your backup with all services running. There are five different parts to be considered.

  1. The system environment (e.g. /boot –> block device based)
  2. The Home Directores (e.g. /root –> file based)
  3. The static system files (e.g. /usr/share –> file based)
  4. The variable system files (e.g. /var/lib –> file based)
  5. The database directories (dynamic –> dump based)

With this said, the process should be quite clear.

Firstly, you should backup your whole system (finished installing your services) on block device level using a rescue system, just to make sure, you can easily revert to a running state or bind-mount the block device snapshot to compare with your file based backups.

Secondly, dump your database files to place a like root’s home directory, so the content gets somewhat like static content. Usually transaction based databases dump a consistent state. It depends on the software, how well the transactions are formed, but usually this should at least keep the database itself intact.

Thirdly, backup the files from all other directories (exluding temporary ones) using a efficient approach (e.g. rsync, rsnapshot or borg) over to a safe place.

And last but not least, check your backup consientiously, or better, try a restore. For a production environment, a regular and automated restore test is indispensable. Being honestly, for a real production environment with hundreds of users, a simple dedicated server is not the right way to go.

Doing it the Chuck Norris way

A quite risky, but mostly working way, is to run a snapshot tool simply on the running system. As it is stated on the borg page, most directories are stable enough to be backed up using an rsync approach. If you do your first backup (that takes quite long) with all services stopped and then only do incremental backups (being lightning fast), there is a low chance of e.g. a home directory changing during the backup process. To avoid getting inconsistent databases, just do a new dump of your databases beforehand (e.g. to backup user’s home). You can even prevent more of the risky parts (using LVM, btrfs) to lower the chance for unexpected changes. And moreover you can stop the relevant services.

For the docker containers (postgres and greenlight), you need to keep in mind, that it is a good practice for docker services to store their data in a bind mount that is kept in the host system. This means, that the thoughts on the database dumps can not be relaxed and may get even more complicated.

Installing Moodle on the BBB-Server

Next step is to install Moodle on this server and migrate all the users and courses from the old one (see my post about it’s installation) to the new one (stay tuned).

Have fun with BBB and Greenlight 🙂

How To Debug Hardware-Faults on Your Dedicated Server

While installing some stuff on a dedicated server at Strato, I encountered a problem with the server two times. While having no clue what happened during the first time, I was prepared during the second. I kept a connection to the serial port proxy of my provider open, to see the most low level messages (kprints).

And you won’t believe, after 13295 seconds of uptime (3 hours 41 minutes), the kernel had a panic:

This is a great indication to run a Hardware-Test and to address the support of your hosting provider.

School’s out – How to Tame Your Children

Introduction – Historic Reasons for this Post

Here in Germany, executive decided to lock all public life down to a minimum (only system relevant shops are allowed to be opened and for each state in germany , there are different exceptions) and it seems, that all countries do the same. This means, many people need to work in their home office and if you have kids, they „join you at work“. And as if that’s not enough, they refuse to learn anything for school. Parents are simply not made to guide their kids through school stuff.

After some days of lock down the teacher of my daughter sent around a voicemail, telling the kids she misses all of them and they should do some tasks at least until easter holidays (from 2020-04-04 to 2020-04-19) start. I was completely excited, how my little girl changed. She stopped every activity and listened to the teacher as she would never listen to me, telling her the tasks she needs to do. But as soon as it came, the magic was over…

Welp, what can I do as an engineer to help myself and my girl out of this…? Right! Help the school to raise a digital classroom. Technically no problem, but in germany, most problems are not technical. Most problems arise due to privacy protection and simply the reluctance of official employees to deal with the work. But this shall not be the topic of this post. The solution is, to be very bullheaded and try to explain the situation based on sience. (Corona is just now starting, german video).

Short side notice: In Bavaria we have a system called Mebis that was intended to complement the daily school with digital information and courses. Teachers even can collect tasks and share courses. Unfortunately, there are no means of communication for whole classes or at least a simple video and/or audio distribution. The highest level of group communication is a chat for 6 persons at max. So, the system needs to be steered by the parents for the little ones (grade one to four).

This means, that all information for our children was distributed by Email or some WhatsApp group (inofficially, because of the privacy laws). Lately, the teacher also has setup a class in Anton-App. But even this is not a good solution for the little ones…

Then, WHAT Would be the Solution?

So, what would you do as a technician, if you expect this lockdown to continue after the easter holidays (as it is a logical conclusion from the above video)? Right! Evaluate what exists out there and find a solution that fulfills the requirements for the low grade scholars:

  • Audio/Video communication (virtual class room)
  • Distribution and collection of task sheets/images (not digital courses, little ones can not operate the keyboard efficiently)
  • Easy to setup courses
  • Complies with GDPR (DSGVO) –> Runs on own server or in a german cloud
  • Easy to use
  • Cheap or free to use

TL;DR

I quickly found a digital classroom soultion that fulfills the requirements: Moodle (BTW: there is also moodlecloud, but this does not comply with GDPR). I decided to run it on my home server (HP Proliant Gen 7 + 100M DSL). So the architecture is quite simple: Reverse Proxy VM as exposed host and Moodle VM connected to internal proxy net (10.0.0.0/16).

My proxy it using Nginx in a very minimal configuration and Moodle is protected and accessed through the proxy from the outside world. The proxy net is also only connected to the services that shall connect to the public internet.

The installation of Moodle is straight forward and following the how-to (German how-to) leads to the the expected results. As database I used MariaDB and Nginx as the webserver.

Beside MariaDB and Nginx, you need to install php-fpm. Which additional php modules you will need is perfectly evaluated during moodle web setup and you should simply keep a console open to run sudo apt install <required-module>.

If you want to run all this through a reverse proxy, don’t delay its configuration to a point in time after the moodle web setup. The setup will determine the url it was accessed and you will need to change this somewhere in the config afterwards…

One of the problems was, to get Moodle running smoothly through the reverse proxy. It had some minor problems to resolve the CSS, JS and all PHP indirectly served files. Here is my proxy configuration to make it work (I assume you already did your SSL config and certificates correctly, hint: wildcard certificates for your domain make everything much easier).

Nginx proxy config (for reverse proxying the Moodle host)

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name moodle.<mydomain>;
  include snippets/my_ssl.conf; 
  include snippets/ssl-params.conf; 
  root /var/www/html/; 
  location / { 
    proxy_set_header X-Forwarded-Host $host:$server_port; 
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
    proxy_pass http://10.0.2.20:80; 
    client_max_body_size 100M;
    proxy_set_header Host $host; 
  }
}

Nginx conf on Moodle host

server {
  listen 80 default_server;
  listen [::]:80 default_server; 
  root /var/www/html; 
  index index.html index.htm index.nginx-debian.html index.php;
  server_name _; 
  location / { 
    try_files $uri $uri/ =404; 
  }
  location ~ \.php/.*$ { 
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/var/run/php/php7.3-fpm.sock; 
  } 
  location ~ \.php$ { 
    include snippets/fastcgi-php.conf; 
    fastcgi_pass unix:/var/run/php/php7.3-fpm.sock; 
  }
}

You only need to unzip a Moodle distro package to /var/www/html and as soon as you point to https://moodle.<your-domain>, it will show up the setup process. Now follow the guide and install the packages Moodle needs.

If you keep everything at the default values, it will only be possible to upload files up to 2M in size. This is not sufficient, if you expect parents to upload photos of the completed task sheets and also sometimes too less for PDFs or images you want to share. Therefore, you need to raise the limit for Nginx and php-fpm. This is done on Debian 10 buster in:

  • /etc/php/7.3/fpm/php.ini
    • upload_max_filesize = 50M
    • post_max_size
  • /etc/nginx/nginx.conf (section http)
    • client_max_body_size 50M;

On the proxy, this is already included in above config.

Lessons Learned

Some of the dependencies, moodle and it’s plugins have are not hard during installation, but they will show up when you use moodle for a while. If you just don’t want to know, why these dependencies are needed, you simply need to do (as root or with sudo)

apt install unoconv ghostscript graphviz 

A bug in the onuconv release 0.7 (which is deployed by debian 10) prevent the unoconv to create it’s own listener and to create some tempfiles. To circumvent this, we simply create a little wrapper script:

vi /usr/bin/unoconv_wrapper.sh

Put in the following content:

#!/bin/bash
HOME=/tmp /usr/bin/unoconv $@

And make it executable.

chmod 0755 /usr/bin/unoconv_wrapper.sh

Now enable unoconv and adjust the path settings in the moodle unoconv configuration to point to the wrapper /usr/bin/unoconv_wrapper.sh

And afterwards, enable the unoconv document converter in website administration (search for unoconv to get the settings).

If you want to use the PDF annotation for all supported file formatts, you need to celect it in Feedback-Types of the Activity

You can also select it in the default settings, so you don’t need to change it for every activity you add to a course. Unoconv then converts different types of files to PDF, and as a trainer, you can simply annotate these PDFs online. If you don’t use unoconv, you need to download e.g. each and every image or other document to your PC and send back your comments in a different way to the student/scholar. Also as a file upload or as a textual comment.

Happy teaching

As soon as your nginx runs, you possibly wish to use your Moodle. One of the first things I did is installing the BigBlueButton-addon for video conferencing. Blindsidenetworks is thankfully providing a free conferencing server for moodle. This means, that all the audio and video traffic is not flooding your server (this is, why I can run it at home). If you wish to use an own on-premise solution, I would give OpenMeetings a try.

Conclusion

The setup of Moodle is quite simple, even if there are little pitfalls with reverse proxy configuration. It also needs some time to get up and running, using the system. But creating simple courses and putting in some PDF files is really dead easy. The higher levels are to concentrate the results (trainer task) and to add little extra candy (e.g. Badges) to motivate children to learn.

All this said, I wish you happy teaching and I would be happy to receive comments on it.

Here a little hint, how a course could look like with content (Tasks in PDFs and a place to put the pictures of the task sheet in).

Update

KW 17 – Three weeks after introducing Moodle

After three weeks and a (partly) official video conference using BigBlueButton (see my other post about installing BBB), More and more children tend to register for this platform. And it seems to be the only available platform, where all that stuff is organized well, corrections can be easily done and even rating is possible. Additionally, with the moodle android and iOS app, the camera can be easily used to commit the scholars/students work within seconds. No scanning, not transportation of finished tasks,…