Startseite

STM32CubeMX and SDRAM

Sometimes, using CubeMX and the HAL, there is something missing. For SDRAM, it is the command sequence that need to be issued after initializing the FMC module. The SDRAM itself also needs some information on timing and refresh, so FMC and SDRAM getting friends.

Digging around the web again mostly shows the low-level solutions and hardly no solutions using CubeMX and HAL. Even the examples of the STM32F4 MCU package has only very irritating examples that could not have been generated by CubeMX. It seems to be again my turn. I already found a community post, that exactly points out the problem with it. The problem with the solution is, that it does not integrate safely into CubeMX generated code when simply copy pasted.

But first we look at the callback tree for initializing the SDRAM.

HAL callback structure for SDRAM

MX_FMC_Init()                 fmc.c
   HAL_SDRAM_Init()           stm32f4xx_hal_sdram.c
      HAL_SDRAM_MspInit()     fmc.c (stm32f4xx_hal_sdram.c)
         HAL_FMC_MspInit()    fmc.c
      FMC_SDRAM_Init()        stm32f4xx_ll_fmc.c
      FMC_SDRAM_TimingInit()  stm32f4xx_ll_fmc.c

And the bug of CubeMX is, not to provide

  • a callback or user section at the end of MX_FMC_Init() or
  • a user section at the end of fmc.c (or at least somewhere after SDRAM handler definition)
  • a callback in FMC_SDRAM_TimingInit()

Update: With latest MCU Package (at least for STM32F4), ST provides a user code section at the end of MX_FMC_Init(). That’s great and avoids the dirty hacking described here 😛

Welp, without having any of these, we need to cheat a bit. The SDRAM handler is already declared external by CubeMX in fmc.h, which in turn is included in fmc.c itself. Honestly, I find this not very clean, because it exposes the internal data stuctures to the outer code. But this is only the opinion of a clean code enthusiast… Since we have neither a callback nor a top level user code section in fmc.c, following the declaration of the SDRAM handler, we go this dirty way. Just below the peripheral initialization, there is a user code section generated by CubeMX. There we add our init sequence function call.

MX_SDRAM1_InitSequence(SDRAM_REFRESH_COUNT, SDRAM_TIMEOUT);

The init sequence function, we will add in a user code section of fmc.c looks like this (it is for SDRAM part IS42S32800G-6BLI).

define SDRAM_MODEREG_BURST_LENGTH_1             ((uint16_t)0x0000)
define SDRAM_MODEREG_BURST_LENGTH_2             ((uint16_t)0x0001)
define SDRAM_MODEREG_BURST_LENGTH_4             ((uint16_t)0x0002)
define SDRAM_MODEREG_BURST_LENGTH_8             ((uint16_t)0x0004)
define SDRAM_MODEREG_BURST_TYPE_SEQUENTIAL      ((uint16_t)0x0000)
define SDRAM_MODEREG_BURST_TYPE_INTERLEAVED     ((uint16_t)0x0008)
define SDRAM_MODEREG_CAS_LATENCY_2              ((uint16_t)0x0020)
define SDRAM_MODEREG_CAS_LATENCY_3              ((uint16_t)0x0030)
define SDRAM_MODEREG_OPERATING_MODE_STANDARD    ((uint16_t)0x0000)
define SDRAM_MODEREG_WRITEBURST_MODE_PROGRAMMED ((uint16_t)0x0000)
define SDRAM_MODEREG_WRITEBURST_MODE_SINGLE     ((uint16_t)0x0200)

void MX_SDRAM1_InitSequence(uint32_t RefreshCount, uint32_t timeout)
{
  __IO uint32_t tmpmrd = 0;
  static FMC_SDRAM_CommandTypeDef Command;

  /* Step 1: Configure a clock configuration enable command */
  Command.CommandMode            = FMC_SDRAM_CMD_CLK_ENABLE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 2: Insert 100 us minimum delay */
  // Inserted delay is equal to 1 ms due to systick time base unit (ms) 
  HAL_Delay(1);

  /* Step 3: Configure a PALL (precharge all) command */
  Command.CommandMode            = FMC_SDRAM_CMD_PALL;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 4: Configure an Auto Refresh command */
  Command.CommandMode            = FMC_SDRAM_CMD_AUTOREFRESH_MODE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 8;
  Command.ModeRegisterDefinition = 0;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 5: Program the external memory mode register */
  tmpmrd = (uint32_t)SDRAM_MODEREG_BURST_LENGTH_1 |\
    SDRAM_MODEREG_BURST_TYPE_SEQUENTIAL |\
    SDRAM_MODEREG_CAS_LATENCY_2 |\
    SDRAM_MODEREG_OPERATING_MODE_STANDARD |\
    SDRAM_MODEREG_WRITEBURST_MODE_SINGLE;
  Command.CommandMode            = FMC_SDRAM_CMD_LOAD_MODE;
  Command.CommandTarget          = FMC_SDRAM_CMD_TARGET_BANK1;
  Command.AutoRefreshNumber      = 1;
  Command.ModeRegisterDefinition = tmpmrd;
  /* Send the command */
  HAL_SDRAM_SendCommand(&hsdram1, &Command, timeout);

  /* Step 6: Set the refresh rate counter */
  HAL_SDRAM_ProgramRefreshRate(&hsdram1, RefreshCount);
}

Just to make it complete, here are the settings for FMC for the specific part, taken from MX_FMC_Init(), generated by CubeMX:

hsdram1.Instance = FMC_SDRAM_DEVICE;

/* hsdram1.Init */ 
hsdram1.Init.SDBank             = FMC_SDRAM_BANK1; 
hsdram1.Init.ColumnBitsNumber   = FMC_SDRAM_COLUMN_BITS_NUM_8;
hsdram1.Init.RowBitsNumber      = FMC_SDRAM_ROW_BITS_NUM_12; 
hsdram1.Init.MemoryDataWidth    = FMC_SDRAM_MEM_BUS_WIDTH_32; 
hsdram1.Init.InternalBankNumber = FMC_SDRAM_INTERN_BANKS_NUM_4; 
hsdram1.Init.CASLatency         = FMC_SDRAM_CAS_LATENCY_1; 
sdram1.Init.WriteProtection     = FMC_SDRAM_WRITE_PROTECTION_DISABLE; 
hsdram1.Init.SDClockPeriod      = FMC_SDRAM_CLOCK_PERIOD_2; 
hsdram1.Init.ReadBurst          = FMC_SDRAM_RBURST_DISABLE; 
hsdram1.Init.ReadPipeDelay      = FMC_SDRAM_RPIPE_DELAY_0; 

/* SdramTiming */
SdramTiming.LoadToActiveDelay    = 2;
SdramTiming.ExitSelfRefreshDelay = 7;
SdramTiming.SelfRefreshTime      = 4;
SdramTiming.RowCycleDelay        = 7;
SdramTiming.WriteRecoveryTime    = 3;
SdramTiming.RPDelay              = 2;
SdramTiming.RCDDelay             = 2;

Even if SDRAM_HandleTypeDef hsdram1; is declared below our init function, it can be used in the upper user code section of fmc.c, because it is declared external in fmc.h, which is included just ahead of the user code section. Again not very clean, but it works.

My Two Cents About HAL…

And for all of you crying that HAL is way to oversized for STM32 projects and everybody needs to know about all registers of the devices. Maybe you are right, but this is no excuse for writing bad code. In my opinion, HAL does a very good job at keeping cohesion high and coupling low. The main problems with it originate from CubeMX templates, that break some of the rules for good coding. Would it use more techniques like first class abstract data types and ensure, the extension points (user code sections, callbacks) are sufficient, it would guide more people to a well designed architecture. Currently, programmers are forced to break the boundaries too often. The same applies e.g. for my other post about UART continuous receive

But instead of coding low-level code that is badly evolvable (can not be extended, reused, adjusted or understand easily), I think it is better to find a clean solution to keep most of CubeMX’s code as is and use the extension points it provides. I believe it is of much higher value having projects all integrating with CubeMX nicely and therefore can be understood easily by other parties (e.g. a new team mate or the guy taking over your project), than reinventing the wheel over and over again using LL functions or direct register access. STM32 have enough space and speed to cope with a little overhead. And exactly here applies another rule of thumb from clean coding guidelines: Never try to speed optimize without profiling. This means, use a profiler to identify the critical pieces of your code and only if needed, invest time to optimize the critical parts.

Penmount PCI Touch Controllers And I2C – Lost In Space

Forword

I would suggest to use UART to talk to the Penmount Touch. This works much better since you can simply trigger on some UART interrupt (6 Byte and timeout fits well). But after collecting some experience, I would strongly suggest to check touch functionality from time to time by requesting the version of the touch controller. We had a batch of touches working well for a long time across but the latest batch seems to hang sometimes.

Introduction

After I worked quite a lot with Touchnetix touch controllers some month ago, I now had a project using a PM2204 from Pemount (Salt). The datasheet of Touchnetix controllers (disclosed only with NDA) consists of several hundred pages defining a huge amount of objects for configuration and infromation retrieval purposes. In the end, you access these objects through dynamic register sets… But this is another story. This is a story about simplicity 😛

TL;DR

With penmount, you get the direct opposite of the Touchnetix. No documentation anywhere and only 6 bytes of data through UART or I2C. Yes, thats right. And its a one way communication (Edit: It can be 2-way, but even the linux kernel driver for UART ignores this fact.). Nothing to configure, nothing you can do wrong… With I2C, you send a read request to the address of the PM2204 and you will receive 6 bytes, when there has been a touch event. If not, you will receive 6 times 0xEE.

So, the best approach is, to watch out for an interrupt and when it occurs, polling these 6 bytes. They contain the event (1 byte), the position (2 x 2 byte) and a checksum (1 byte). And here is the piece for decoding it (some spices for error checking HAL should be added…):

uint32_t total;
uint8_t buf[6];

HAL_I2C_Master_Receive(&i2c1, 0x70, buf, 6, 100);

btn = buf[0] & 0x40;
xpos = ((buf[2] << 8) | buf[1]) * SCREEN_X_SIZE / 2048;
ypos = ((buf[4] << 8) | buf[3]) * SCREEN_Y_SIZE / 2048;
checksum = buf[5];

for (int i = 0; i < 5; i++) 
  total += buf[i]; 

if (checksum == (unsigned char) ~(total & 0xff))
  DO_SUCCESS_STUFF;
else
  RAISE_AN_ERROR;

How I digged through it? I found the linux driver using UART communication here and just tried, if I2C behaves the same… after hours of trying to access registers like on a memory device…. 🙁

Stay tuned for Penmount-Touch and UART, which is working best for us…

Thats all. Have fun with Penmount!

Angular Web App on ESP32

In progress…

TL;DR (take me to the battle field)

Introduction and Motivation

In some recent project, we added some ESP32 to replace wired communication with a BLE solution. Additionally, the customer wants to have some administrative web application to configure the device more easily and to get more detailed information in service.

Creating a simple web ab is not a hard task for an embedded C developer. But to create a beatiful or at least not shitty looking web application is just another game. When you look more closely on the available space on some ESP32, you will realize, that there is at most 16 MB of space to live with.

If you create a simple WiFi AP on some ESP32 4MB version with ESP-IDF, you can easily see, that there is not much left for BLE and APP.

Build statistics for a simple WiFi AP with only a hello world index.html on some lolin32 board

Unfortunately, I only have the 4 MB version available, which complicates the task even more. And no, I can not simply change to 16MB, since a few hundred units are already in operation, waiting for some firmware update to provide WiFi service. You know how this works… Deliver the hardware as early as possible and hand features in later 😛

But this means, there is much headroom for optimization and not much space for resources and code. If you have a look at web application frameworks (e.g. react) and build some basic apps, they all come with megabytes of resources in the production build output folder. If you build a very basic hello-world angular application, you will find a few 100 kilobytes of resources after building it, to be deployed on a static web space. This is also just small enough, to be fitted in the flash.

Now, let’s talk about the solution…

Prerequisites

First of all, you will need to create an angular and an ESP-IDF project. This is easily done, if you already installed Visual Studio Code with PlatformIO.

To create an angular project, just follow the official guide. If you installed npm correctly, you can issue the commands in your Visual Studio Code Terminal, to fire up the project.

ng new esp-on-angular

After these steps, you have two separate projects. You can easily setup a script to copy the angular production build over to the ESP project, do it manually or link it in using git submodules. But this is not part of this tutorial. We simply copy over the website content manually from the angular project output folder.

In addition to npm, VSCode and PlatformIO, you could need some basic Linux system, when working on Windows, if you intend to use the automation script I’l provide later. On my machine, I use MSYS2/MinGW to get the CLI tools I need: bash, find, gzip, stat

Getting Started

Firmware Web Content Integration

While it is quite easy to simply handling incoming requests and responding to it, like you would do for some RESTful Web Service (which is the usual case for IoT stuff like the ESP32), integrating plain files is usually a more complex task. Thanks to the ESP-IDF, this is not the case.

In general, there are two options. Adding the files to a SPIFFS partition and reading it from there. Another solution is to embed the files in your firmware object. The latter one is what we will use in this tutorial, since it is more dense (in terms of size), althought it is less flexible and requires a bit more lumber in your config.

For it to work, simply add the files to board_build.embed_files environment in your platform.ini file of your ESP project. I usually put these ressource files into a separate folder called res.

board_build.embed_txtfiles = 
    res/index.html
board_build.embed_files =
    res/logo.png

When you did so, you can link the content easily with the following lines in your code, PlatformIO will care for it to be a valid symbol and to be linked in:

extern const uint8_t index_html_start[] asm("_binary_index_html_start");
extern const uint8_t index_html_end[]   asm("_binary_index_html_end");
extern const uint8_t logo_start[] asm("_binary_logo_png_start");
extern const uint8_t logo_end[]   asm("_binary_logo_png_end");

When you want to use that data in your http handler, just use the pointers declared above:

httpd_resp_set_type(httpRequest, "text/html");
httpd_resp_send(httpRequest, (const char*) 
    index_html_start, HTTPD_RESP_USE_STRLEN);

…or for your png image file…

httpd_resp_set_type(httpRequest, "image/png");
httpd_resp_send(httpRequest, (const char*) 
    logo_start, logo_start - logo_end);

I personally use only the binary version, since it can also process text files and makes the code more universal. You will later see, that there will not be much of a text left 🙂

The symbols of your file object (the part in the asm function) is built by concatenating _binary_ with the name of your file with all special chars replaced by underscores _ and the suffix _start and _end respectively. In short: _binary_<FILENAMEREPL>_start and _binary_<FILENAMEREPL>_end.

Welp, that is quite fine, but the files still take up quite some space and we do not have a lot. When using the 4M version of the ESP32, we simply can not allow it to do so. So, what can we do further to reduce the size of our resources?

You could ask: Why not push the images and fonts to a publically accessible location in the wolrd wide web? Because some of us want to make it self contained. The reason for me have been simply the security and safety requirements of the project: Keep the system closed to the outside world, provide some WiFi Access Point and serve all files that are needed yourself. Do not depend on internet in general. If you have other requirements, kick your ESP in your local WiFi and access it from there or use the AP/Client mode, where the ESP connects to a wifi as a client, provides a WiFi AP and acts as a router between these two networks. But be warned, don’t expect too much performance from the latter option.

Welp, back to the size problem… What to do to make it fit?

Some quirks to shrink it down

Optimizing the ESP-IDF libraries and build scripts to free up some flash, is maybe quite an intensive task and needs a deep understanding of the IDF internals. So, we will first try to find some other quirks, to reduce the size of our own big blobs.

Angular – Stripping the unneeded

While angular itself is really tiny, it links in some fonts from google (Remeber, we want to be self contained). This is usually not needed, because devices nowadays have some replacement fonts installed, we can easily delete the links from the generated index.html and rely on the local resources of the client.

<link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">

Zero Effort Zipping

While Angular already minimizes most of it’s output files, it would be really awesome to compress the resources somehow. But integrating a compression library and using it, stacks up again…

Then I had an flash of thought. Once upon a time, web speed analyzis tools started to raise warnings, when your website content was not delivered in a compressed format. Nowadays, webspace providers enable brotli and/or gzip compresson by default and there are plenty of tutorials available about WordPress and how to enable compression. All modern web browsers support gzip compression by default and offer it as a supported content encoding sheme.

Welp, but what this has to do with my web application content. Shall the ESP compress all content ahead of the transfer? Of course not! We simply compress all files ahead of linking the firmware and simply fake the content type to be the original while setting the content encoding to gzip. This way, the ESP is completely liberated from compression and we can reduce the size of our firmware as much as possible.

Do I need to compress all files manually and somehow save the original content type? Not really. I wrote a simple bash script, that does most of the work for you and creates some header with a look-up table (LUT) for the code to access it easily. A tiny URI-router will search it within the LUT and pick all information it needs from it.

Putting the pieces together

The Angular Project

The ESP Project

After we created the ESP project as described in my other post, we will start implementing a simple web service. This is done with

Compile Ceph (master) on ARM (32-Bit)

I gave up on getting Ceph run on ARM 32 bit. It was a huge effort to fix the types, that diverge when switching from 64 to 32 bit. The development of Ceph is simply to fast to cope with. Since the developers decided to drop all tests for 32 bit builds, the needed fixes are too many for a single person to hunt after it.

TODO: Test this all on a virgin armhf system (raspberry, odroid hc1/2/XU4,
) and complete the TODOs for openssl and phantomjs (and the sass-dependency). Maybe with the new master tree, it is not needed to build it outside the ceph repo.

First install prerequisites:

sudo apt install python-pip build-essential libgmp-dev \
libmpfr-dev libmpc-dev reprepro

Install nodejs from nodejs.org

curl -sL https://deb.nodesource.com/setup_11.x | sudo -E bash - sudo apt-get install -y nodejs
sudo npm install -g npm

Then prepare a swap partition (you will need it 😉 )

dd if=/dev/zero of=/<some-hdd-path>/swapfile \
bs=1M count=8192 progress=status
mkswap /<some-hdd-path>/swapfile
swapon /<some-hdd-path>/swapfile

Then we should install some dependencies

sudo apt install libgmp-dev libmpfr-dev libmpc-dev ruby

Now install a new GCC that supports C++17.

wget https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.xz
tar xfJ gcc-8.2.0.tar.xz
cd gcc-8.2.0
./configure # for armhf
# ./configure --disable-multilib # for x86_64/arm64
make

Building ceph with do_cmake, building a debian package with make-debs.sh or simply build packages using another compiler than the debian default one (6.3.0) requires you to change the default compiler e.g. to gcc-8.2.0 for the whole system:

sudo update-alternatives --install /usr/bin/cc cc /usr/local/gcc-8.2/bin/gcc-8.2 50
sudo update-alternatives --install /usr/bin/c++ c++ /usr/local/gcc-8.2/bin/g++-8.2 50

Checkout OpenSSL-1.0.2-stable (seems also necessary for armhf), PhantomJS, compile and install it:

cd /opt/GIT
git clone git@github.com:openssl/openssl.git
cd openssl
git checkout OpenSSL-1_0_2-stable
...TODO...
# Following seems only necessary on arm
# (or all platforms wihtout precompiled binary)
cd /opt/GIT
git clone git@github.com:ariya/phantomjs.git
cd phantomjs
...TODO...
sudo LD_LIBRARY_PATH=/opt/openssl_build_stable/lib/ \
deploy/package.sh --bundle-libs

Add the following to build.py (at L:244, just after PlatformOptions.extend)

phantom_openssl = os.getenv("PHANTOM_OPENSSL_PATH", "")
if phantom_openssl != "":
openssl = os.putenv("OPENSSL_LIBS", "-L" + phantom_openssl + "/lib -lssl -lcrypto")
openssl_include = "-I" + phantom_openssl + "/include"
openssl_lib = "-L" + phantom_openssl + "/lib"
platformOptions.extend([openssl_include, openssl_lib])
print("Using OpenSSL at %s" % phantom_openssl)

Then install it to /opt

Build and compile Ceph

git clone git@github.com:the78mole/ceph.git
cd ceph
git checkout wip-32-bit-arm-fixes
./install-deps.sh
./do_cmake_arm32.sh # for armhf
# ./do_cmake.sh # for x86_64/amd64 or arm64
cd build
make -j4
# if it gets really slow due to swapping, break an do make -j1
# or use the scheduler-script from link below

Here you can find a rudimentary (but working) script that suspends compilers processes based on total compilers memory consumption. Running it through ‚watch‘-tool you can start e.g. 8 tasks and when memory limit is reached, it will suspend (kill -TSPT) the youngest tasks in sense of user space runtime.
https://github.com/the78mole/scripts/blob/master/linux/bash/schedule_compile.sh

Now do…

cd ..   # Back to ceph base dir
./make-debs-arm32.sh # for armhf
# ./make-debs.sh # fox x86_64/amd64 or arm64

If you encounter problems with setuptools (Exception –> TypeError: unsupported operand type(s) for -= ‚Retry‘ and ‚int‘) try to get a more recent version of python pip with the following commands and rerun make-debs-arm32.sh.

apt-get remove python-pip python3-pip
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
python3 get-pip.py

If I forgot anything to make it work, feel free to write some comment…

How to Build A Private Storage Cluster (with Ceph)

Finally, I did not succeed in getting Ceph running on a 32 bit ARM. There have been to many issues in the code (especially incompatible datatypes) and issues with GCC and the 3 GB RAM limit for 32 bit platforms. I’m now focusing on using a 64 bit ARM and developing a dedicated HW for it. So, stay tuned…

Since my NAS (a QNAP TS-419P II) get more and more buggy, especially with non-working Windows shares and the painfully low processing power of the integrated ARM single core, wished something like a SAN for myself. But SAN is quite expensive, the peripherial hardware (Switches, UPS,…) not included. So I decided to skip a few levels and build up a NAS 2.0 storage cluster based on open source ceph using low-budget ODROID HC2 (Octa-Core 4 x Cortex-A15 + 4 x Cortex-A7) from Hardkernel as the work horse to create storage nodes. To make it even more dense, you can use the ODROID HC1 that is just the same but for 2.5″ disks (be aware of the power supply: HC2 = 12V, HC1 = 5V !!!).

If you don’t need a SATA drive (e.g. for the controlling nodes of the cluster: mgr, metadata, nfs, cifs,…), you can use the MC1, MC1 solo, XU4 or XU4Q.

If you want to go with x86 instead of ARM, the ODROID H2 looks like a great alternative, but it will also be a bit more expensive (e.g. RAM is not included).

In fact, installing ceph will be much less pain, going for 64-bit x86 than going with ARM 32 bit. I decided to go with ARM 32, because I want to build up the most energy efficient cluster, to maximize scale out capabilities also in sense of my private budget.

To build up the cluster, I currently use 4 x ODRID HC2 with WD Red 4 TB drives (WD40EFRX), also installing the ceph non-OSD-services distributed accross this little cluster. The BOM for my test cluster is as follows:

If powering up the cluster in sequence (not all at once), you could reduce the power requirements of the supply component a lot (currently 12V/2A per node, 5V/4A for HC1). I will dive into this topic a bit deeper in future. I think, it can be done in software by delaying the spin up through some bootarg. Nevertheless, an optimum solution would be to have a power distribution unit for switching and measuring the supply current and also providing some UPS capabilities on the low voltage path. Additionally, current measuments could give you the ability to regulate the power through e.g. cpufreq to optimize the efficiency of the cluster and the power supply.

To generate the debian packages for installing ceph on the nodes, follow the instructions here. When you have built the debian packages, move them over to some http(s) server, to be easily accessible by your nodes.

Your Own Debian Package Repository

To be accessible, a Debian Package Repository needs to be placed in a webserver’s directory accessible at least in your own network. It is best practice to secure this repository with SSL, since Debian APT more or less expects this… So first we start with creating a (self signed CA). Later, if needed, you can easily replace the certifciate by an official one or let an authorithy also sign your server certificate.

Generating a CA for SSL

This part is based on the tutorial here. First, we will simply use self signed certificates, since it is much easier and faster than using officially signed certificates. We will then place the CA in the cert storage of our linux OS, to make it trust ourself. 😉

mkdir ~/CA
cd ~/CA
# Generate the CA key
openssl genrsa -out ca.key 4096
# Generate the CA certificate, here, you can leave the CN empty
openssl req -new -x509 -key ca.key -days 366 -out ca.crt
# Make it unaccessible by other users
chmod 700 ca.key
# Generate a certificate configuration
wget https://raw.githubusercontent.com/the78mole/scripts/master/templates/configs/ssl/cert.conf -O example.org.conf
# Edit the configuration
vi example.org.conf
# Create a server certificate key and the signing request (not the yet cert)
openssl req -new -out example.org.csr -config example.org.conf
# Create the public key
openssl rsa -in example.org.key -pubout -out example.org.pubkey
# Sign the CSR with your CA and create the certificate
openssl x509 -req -in example.org.csr -CA ca.crt -CAkey ca.key -CAcreateserial -extensions my_extensions -extfile example.org.conf -days 366 -out example.org.crt

To get the alternative DNS names and IPs added to the certificate, you need to specify the config file as an extensions and point to the config section, where the extensions are located. This is because the extensions in the CSR get ignored by openssl when signing and you need to specify it explicitly.

After generating the certificate, you need to import it, where you need it to be accepted (Browser, APT). For testing, it is best to try with a browser. Some tutorial can be found here (it’s german, so use google translator, to read in english) and here. Use the shell of your desktop Debian system.

scp <CA_HOST>:/<PATH_TO_CA>/ca.crt example_ca.pem
sudo cp example_ca.pem /usr/local/share/ca-certificates/
sudo update-ca-certificates

To add the certificate to your browser, e.g. chromium

sudo apt install libnss3-tools
certutil -A -n "Example Company CA" -t "TCu,Cu,Tu" -i example_ca.pem -d ~/.pki/nssdb

Note: Maybe this does not work correctly… Then, in Chromium, use Settings –> Privacy and Security –> Manage Certificates –> Import –> Select the CA –> Check all boxes.

Now we need the CA’s and the server’s certificate along with the server key for securing webserver traffic.

Install and Configure the Webserver

Welp, we will use nginx as our webserver. Feel free to use any other, it does not really matter. In fact, every further step (e.g. the let’s encrypt tutorial) will be based on nginx.

sudo apt install nginx
cd /etc/nginx
cp snippets/snakeoil.conf snippets/ssl_example.org.conf
mkdir -p ssl/pub
mkdir -p ssl/priv
sudo chown -R root:www-data ssl
sudo chmod -R 0755 ssl/pub
sudo chmod -R 0750 ssl/priv
cp ~/CA/ca.crt ~/CA/example.org.crt
cp ~/CA/example.org.key
# Edit the ssl config file to your needs
vi snippets/ssl_example.org.conf
# Now adjust the nginx configuration to use SSL
vi sites-enabled/default
# Ensure following lines are added and not commented out
# listen 443 ssl default_server
# listen [::]:443 ssl default_server
# include snippets/ssl_example.org.conf
service nginx restart

When everything is OK, use your desktop web browser and point it to the location https://example.org. Your should get the page without an error. Thos means, you have setup a CA you can use to sign server certificates and they get trusted.

If you plan to use the Debian package repository on many of your linux hosts, then you should add your CA certificate to the certificate store on all the machines.

Generating GnuPG Key-Pair

To sign a file, email, hash, debian package, repository,… you often need GnuPG. To be able to sign something, you need to first generate your own key, that get trusted from at least the receiving party. All this works again with asymmetric encraption, like the signing of certificates does. An in depth tutorial with links to even deeper knowledge can be found here.

First we should install a tool to gather some entropy, otherwise gnupg may be not able to generate a key on a headless system (no real user input,… –> very few entropy sources).

apt install rng-tools

IF it can not find a hw-rng, you can still try to get randomsound working (if you have a soundcard…)

apt install randomsound

Run this in a seperate window, when gpg is collecting entropy for too long. It will abort after some time, if it can not generate the key.

arecord -l # Do you have any soundcard?
randomsound -v

If all fails, you can still pipe some data into /dev/random to feed the entropy pool, e.g. with (also in a seperate window when gpg gen-key is running.

sudo dd if=/dev/sda of=/dev/random status=progress

You can watch the entropy-pool with:

watch -n 0.5 cat /proc/sys/kernel/random/entropy_avail

To finally generate a GPG-key, simply follow the instructions below:

apt install gnupg
# Create the .gnupg directory easily and add a secure configuration
gpg --list-keys --fingerprint
wget https://raw.githubusercontent.com/the78mole/scripts/master/templates/configs/gnupg/gpg.conf -O ~/.gnupg/gpg.conf
gpg --full-gen-key
# Select:
# Key type : RSA and RSA
# Keysize : 4096
# Expiration: 1y
# Then enter your name and email, but don't include a comment
# Skipping the password makes CI much easier, but less secure...
# It will take some time (maybe minutes) to generate the key

Creating the debian repository (reprepro)

make-debs already created a debian repository, but we will create one, that is more general, also serving well for other software packages. make-deps

…. to be continued …

Coming soon: To add some real NAS features, we could use just another embedded board with e.g. FreeNAS or NextCloud installed to mount the cluster file system and using the cluster as the storage backend. We already have the nginx SSL configured, so we easily can add reverse proxy targets… (for HTTPS-HTTPS-proxy, see here)

ESP32-EVB, PlatformIO And ESP-IDF – Yet Another ESP32 tutorial

I already posted a tutorial on using the ESP32-EVB from olimex with Arduino. This time, I will provide the same with ESP-IDF, the original SDK from Espressif. Why I decided to do this? Because I had a project using it 😛

To install PlatformIO, just follow the guide.

As shown in the below screenshots, create a new project VSCode -> The PIO-Icon -> New Project, define a project name, select Olimex ESP32-EVB with ESP-IDF framework and press Finish. After PIO downloaded all dependencies and configured your project, we are ready to go!

Now just wait until it finishes…

After PIO finished setting up the project, you will find the platformio.ini with its configuration. Add the line serial_speed = 115200 to the file.

After this, run an update on pio (especially useful if you had PlatformIO already installed).

pio update
pio upgrade --dev

We can also now build the empty application by entering pio run in the terminal.

If this succeeds, we will edit the main.c file and create some hello world application.

Hello World – Relay Toggle

Open src/main.c and edit it, so it will look like that:

#include <stdio.h>
#include "sdkconfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "driver/gpio.h"

#define RELAY_GPIO      32

void app_main() {

    gpio_pad_select_gpio(RELAY_GPIO); 
    /* Set the GPIO as a push/pull output */ 
    gpio_set_direction(RELAY_GPIO, GPIO_MODE_DEF_OUTPUT); 

    while(1) {
        printf("Test\r\n");
        gpio_set_level(RELAY_GPIO, 0);
        vTaskDelay(pdMS_TO_TICKS(500));
        gpio_set_level(RELAY_GPIO, 1);
        vTaskDelay(pdMS_TO_TICKS(500));
    }
}

…connect your Olimex ESP32-EVB and enter pio run -t upload in terminal.

Done! Your Relay should toogle twice every second.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code. We already prepared the test in our code above. It prints „Test“ every second on the serial console, just that we can not see it yet. For this to show up, you need to open the serial monitor:

From now on, you should see the printf output in your terminal every time you upload your code again:

Do not forget to close the serial monitor, when you upload your code next time. The serial is blocked by the monitor, so the uploader can not access this port.

After closing the serial monitor, you should drop back to your terminal, where you can start over with pio run -t upload.

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or bluetooth or whatever. You will find information on all peripherials here. Just remember to include the header(s) that are mentioned on top of every peripherial page, like we did with #include "driver/gpio.h" above in main.c.

Now, you have some great starting point for ESP-IDF 😛

Have fun coding!

ESP32-EVB, PlatformIO And Arduino – Yet Another ESP32 tutorial

What the heck? Aren’t there enough toturials out there about ESP32? I believe: Yes, too many. And there are too many that struggle with setting up the arduino environment for non-arduino hardware. But there is a straight-forward solution that works out of the box: platform.io or also known as PIO.

And there are enought tutorials about installing platform.io. It is as easy as you can think. Install Visual Studio Code from Microsoft and install the PIO plugin in VSCode.

Then create a new project VSCode -> New Project and select Olimex ESP32-EVB with Arduino Platform. After PIO downloaded all dependencies and configured your project, we are ready to go!

Hello World – Relay Toggle

Open main.cpp and edit it, so it will look like that:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

…connect your Olimex ESP32-EVB and hit the upload button:

Done! Your Relay should toogle every 10 seconds.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code.

Extend your code to look like this:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  Serial.begin(115200);
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  Serial.printf("Switch on\r\n");
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  Serial.printf("Switch off\r\n");
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

Add two lines to your platfromio.ini file. You will find the correct COM port in terminal when uploading the new code to your device.

In my case it is COM6 and platformio.ini looks like this:

[env:esp32-evb]
platform = espressif32
board = esp32-evb
framework = arduino
monitor_port = COM6
monitor_speed = 115200

Upload your code again after changing the ini file. If it does not print your output automatically to your terminal, start the monitor manually:

From now on, you should see the printf output in your terminal every time you upload your code again:

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or whatever. But now, you have some great arduino IDE without arduino IDE 😛

Have fun coding!

Sorting Your Digital Mess – How to Easily Set-Up a Private Search Engine

Motivation

In my vacations, I kicked-off some new projects. One of it is an ARM64 based SBC with integrated SATA mostly like the Odroid-HC1/2. The main difference is the ARM (64 vs. 32 Bit) to be able to run Ceph on it.

If you read some of my previous posts, you will notice, that I already put in some effort to make Ceph run on 32-bit controllers. But the effort was way to high, to make and keep it running. But the new SBC is another story, I will tell, when the time is ripe for it.

When I thought of putting all my data on a Ceph cluster, I just stumbeld over the probem, how to manage all these bits and bytes and how to keep the overview. The should be something like a Google-Search for your private data. When you dig through the net, you will find many sites, blogs and books about elastic search. But there is always a problem, how to get your data in there without a deep knowledge on data providers, ETL, graphs,… But you could also find OpenSemanticSearch (OSS) and I gave it a try.

A Private Search Engine – Open Semantic Search

To be honest, OpenSemanticSearch is not the most beatiful engine you could think of, but it gives you a very deep insight on your data, assuming you configured it correctly. Unfortunately, the documentation is quite sparse and there have not been many bloggers writing on the topic, at least not for some current version of it.

After you finished Installation, first do some deeper configuration. If you choose to use a VM, I suggest, to usa a RAM centric configuration. Better take 12+ GB of RAM and only 4 cores. Every core will lead to an etl-task that eats up a significant amount of RAM, depending on the data it is advised to index.

To avoid links of the WebUI to point you to some unreachable destination, do some configuration ahead of adding local crawler paths. If you mount your data using mount to a path in your filesystem on the OSS server, the link will be provided unchanged (e.g. /mnt/myData) and the URL will just be the location it is on your server without http, hostame, location-suffix,… Not really helpful, if you have mounted NFS shares and want to access it on a remote windows machine.

OSS provides already some document server, that proxies your requests to HTTP, but you need to configure it correctly. But if you do not configure it in the beginning, your whole indexing run will create wrong links and it will be hard to stop the indexing and do a rerun (see the troubleshooting section below). The task will be running, until it has done it’s job. For terabytes of data, this will last at least some days.

Installation and Configuration of OSS

Installation of OSS is quite straight forward, as described on their web page.

Configuration is a bit more complex due to the fact, that the documentation is a bit sparse. Some of it can be done with the web UI, but others are a bit more covered. But first things first.

Mounting Your Data

The first to do is, to make your data accessible to the indexer.

[TODO]

Configuring Apache to Proxy the Documents

The quick hack is, to add your path to proxy in /etc/apache2/sites-available/000-default.conf

Insecure Proxy for documents

This allows the documents to be accessed throuch http://<IP_OF_YOUR_OSS>/documents/mnt but also through http://<IP_OF_YOUR_OSS>/mnt. We will secure this later on. But for debugging, this is quite helpful.

To get the links in OSS right, you need also to adjust /etc/opensemanticsearch/connector-files to have a line with the following content

config['mappings'] = { "/": "http://172.22.2.108/documents/" }

You can also add more mappings, but this helps to get it right for the links in OSS Web UI.

(Re-)Starting the OSS server

I would advise to do a simple reboot of your server, to also check, if the shares get mounted correcly. For my debian buster, this is not the case. It is failing to mount the NFS shares for some reason, I have not yet digged down. I manually mount it, after it has startet. I assume, the systemd start dependencies are a bit buggy.

[Much more to write on… Coming soon]

Troubleshooting

So, what to do, if something goes wrong? E.g. if your decided to index a ton of data, it is time to purge the queue. But how?

Purging the Queue

OpenSemanticSearch uses RabbitMQ to organize the indexing tasks. If OSS decides to index a path, it will simply list all files in this path and put it in the open_semantic_etl_tasks-Queue of your RabbitMQ server. But the user interface of OSS does not provide a means of purging or deleting the content of the queue. But therefore, you neet to activate the rabbitmq management web ui.

sudo rabbitmq-plugins enable rabbitmq_management

After this, you can check, if your server listens on the apropriate interface (0.0.0.0 for acces from another host) and port (15672)

netstat -nlpt output of a typical OSS host with enabled RabbitMQ Web UI

This checked, we need to add a user with administrative rights to the OSS worker.

rabbitmqctl cluster_status   # Check if everything is fine
rabbitmqctl list_users       # Check if your webuser doesn't exist
rabbitmqctl add_user webuser <PASSWORD>
rabbitmqctl set_user_tags webuser administrator
rabbitmqctl set_permissions -p / webuser "." "." "."

This done, you can simply acces the user interface. Type in your browser http://<IP_OF_OSS_HOST>:15672/ and the login will be presented to you.

RabbitMQ Web Login

After loging in, head over to the queues and enter your open_semantic_etl_task-queue.

Queues view of RabbitMQ web UI

You should be presented with the queue details

Queue details of open_semantic_etl_tasks

In my example, you can see a very limited count of ready tasks. This can go up to a few thousand tasks when indexing a large directory with many files. Each file will add a task to this queue and RabbitMQ will hand this out to the etl workers of OSS.

To delete all messages in the queue, you can simply hit the Purge Messages button.

Purging the queue

After this, you can re-enqueue the index files on your OSS search site through http://<IP_OF_OSS_HOST>/search-apps/files/

FreeRTOS debugging on STM32 – CPU usage

Introduction

Since the information about FreeRTOS debugging with STM32CubeIDE is sparse and ST is not yet providing the task list view (that was part of the Atollic TrueStudio), here is, how you get it by installing a plugin from freescale and adding the approprite stuff to your code. I assume, you already have a project with FreeRTOS setup and running…

Adding the plugins

First start STM32CubeIDE and go to Help -> Install New Software…

Then add an Update Site by clicking the „Manage“-Button. Here you need to add the update site from freescale. And yes, NXP/Freescales plugin works with STM’s CubeIDE 🙂

http://freescale.com/lgfiles/updates/Eclipse/KDS

„Apply and Close“ and select the new site to „Work with“

Select the FreeRTOS Task Aware Debugger for GDB.

And click Next… Follow the Wizard until complete and after installation, restart your STM32CubeIDE.

Configuring the FreeRTOS project

Now add a timer and configure a reasonably a high tick rate (e.g. I used TIM13 of my STM32F469, running with 180 MHz HCLK, 90 MHz APB1 Timer clock and a timer counter period of 899 -> 100 kHz resolution).


Enable the interrupt

And in Middleware -> FreeRTOS, enable the run-time stats

If you like, you can also enable RECORD_STACK_HIGH_ADDRESS. Sometimes this is quite useful and avoids the little warning symbol in stack usage column of task list view.

Now regenerate your project…

Adjusting the code

Now it’s time to adjust your code for collecting the stats. Add a line for starting the timer in IT-mode by adding a function in some user code section in main.c.

volatile unsigned long ulHighFrequencyTimerTicks;

void configureTimerForRunTimeStats(void) {
  ulHighFrequencyTimerTicks = 0;
  HAL_TIM_Base_Start_IT(&htim13);
}

unsigned long getRunTimeCounterValue(void) {
  return ulHighFrequencyTimerTicks;
}

In stm32f4xx_it.c, add the following lines to the appropriate user sections

/* USER CODE BEGIN EV */
extern volatile unsigned long ulHighFrequencyTimerTicks;
/* USER CODE END EV */

[...]

void TIM8_UP_TIM13_IRQHandler(void)
{
  /* USER CODE BEGIN TIM8_UP_TIM13_IRQn 0 */
  ulHighFrequencyTimerTicks++;
  /* USER CODE END TIM8_UP_TIM13_IRQn 0 */
  HAL_TIM_IRQHandler(&htim13); 
  /* USER CODE BEGIN TIM8_UP_TIM13_IRQn 1 */
  /* USER CODE END TIM8_UP_TIM13_IRQn 1 */
}

If you are compiling with optimization levels above -O0, you also need to fix a bug (it is one in my opinion) in freertos tasks.c.

There are two possibilities:

  1. Switch of optimizations for tasks.c by right clicking on the file in project browser and changing the compiler optimization to -O0
  2. Change the line in tasks.c adding a volatile (see picture)

The problem with solution 2 is, that you need to do it after each STM32CubeMX code generation again. But there is a 3rd solution, that makes solution 2 persistend (until you update the MCU package).

Go to `%HOMEPATH%\STM32Cube\Repository\STM32Cube_FW_F4_V1.25.0\Middlewares\Third_Party\FreeRTOS\Source\` and edit the file like in solution 2, adding a volatile statement.

When you regenerate your project from CubeMX, it will include the correct line.

Profiling in action

Now after you put everything in place, it is time to run your code. Start the project in debugging mode, make the FreeRTOS/Task List view visible and let it run for some seconds. Then hit the pause button. The task list will collect the information from your target (from GDB) and show it nicely:

If the Task List view complains about FreeRTOS not have being detected, restart STM32CubeIDE and it should show up again.

Edit: During my last weeks of uing this Eclipse plugin, I had some problems seeing all tasks in Task Analyzer. In fact, the FreeRTOS functions to print the run-time statistics show them, while the plugin doesn’t. Also the data seems to be currupted sometimes within the plugin. So I would suggest, to better use the FreeRTOS internal stuff: https://www.freertos.org/rtos-run-time-stats.html

Citations

The information was collected from these links:

Just Another Hobby – 3D Printing

When it comes to leasure time, there are not many activities, that are more exciting than 3D printing. After you printed diverse things from Thingiverse or any other 3D printing gallery, there is a great wish, deeply buried in your mind, to design stuff yourself to get the real excitement — and what is more obvious than upgrading your 3D printer with parts printed by itself/ yourself 😛

If this was the right appetizer, look how my Ender3 got upgraded with a full metal frontend or take a look at my Thingiverse profile with some other designs. If you have any questions, especially the decisions, e.g. why I designed the new hotend exactly as it is (a wide monster hotend) or suggestions, how to improve it, do not hesitate to comment!

     

Copyright © 2018 Daniel Glaser. All rights reserved

Consent Management mit Real Cookie Banner