Compile Ceph (master) on ARM (32-Bit)

I gave up on getting Ceph run on ARM 32 bit. It was a huge effort to fix the types, that diverge when switching from 64 to 32 bit. The development of Ceph is simply to fast to cope with. Since the developers decided to drop all tests for 32 bit builds, the needed fixes are too many for a single person to hunt after it.

TODO: Test this all on a virgin armhf system (raspberry, odroid hc1/2/XU4,…) and complete the TODOs for openssl and phantomjs (and the sass-dependency). Maybe with the new master tree, it is not needed to build it outside the ceph repo.

First install prerequisites:

sudo apt install python-pip build-essential libgmp-dev \
libmpfr-dev libmpc-dev reprepro

Install nodejs from nodejs.org

curl -sL https://deb.nodesource.com/setup_11.x | sudo -E bash - sudo apt-get install -y nodejs
sudo npm install -g npm

Then prepare a swap partition (you will need it 😉 )

dd if=/dev/zero of=/<some-hdd-path>/swapfile \
bs=1M count=8192 progress=status
mkswap /<some-hdd-path>/swapfile
swapon /<some-hdd-path>/swapfile

Then we should install some dependencies

sudo apt install libgmp-dev libmpfr-dev libmpc-dev ruby

Now install a new GCC that supports C++17.

wget https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.xz
tar xfJ gcc-8.2.0.tar.xz
cd gcc-8.2.0
./configure # for armhf
# ./configure --disable-multilib # for x86_64/arm64
make

Building ceph with do_cmake, building a debian package with make-debs.sh or simply build packages using another compiler than the debian default one (6.3.0) requires you to change the default compiler e.g. to gcc-8.2.0 for the whole system:

sudo update-alternatives --install /usr/bin/cc cc /usr/local/gcc-8.2/bin/gcc-8.2 50
sudo update-alternatives --install /usr/bin/c++ c++ /usr/local/gcc-8.2/bin/g++-8.2 50

Checkout OpenSSL-1.0.2-stable (seems also necessary for armhf), PhantomJS, compile and install it:

cd /opt/GIT
git clone git@github.com:openssl/openssl.git
cd openssl
git checkout OpenSSL-1_0_2-stable
...TODO...
# Following seems only necessary on arm
# (or all platforms wihtout precompiled binary)
cd /opt/GIT
git clone git@github.com:ariya/phantomjs.git
cd phantomjs
...TODO...
sudo LD_LIBRARY_PATH=/opt/openssl_build_stable/lib/ \
deploy/package.sh --bundle-libs

Add the following to build.py (at L:244, just after PlatformOptions.extend)

phantom_openssl = os.getenv("PHANTOM_OPENSSL_PATH", "")
if phantom_openssl != "":
openssl = os.putenv("OPENSSL_LIBS", "-L" + phantom_openssl + "/lib -lssl -lcrypto")
openssl_include = "-I" + phantom_openssl + "/include"
openssl_lib = "-L" + phantom_openssl + "/lib"
platformOptions.extend([openssl_include, openssl_lib])
print("Using OpenSSL at %s" % phantom_openssl)

Then install it to /opt

Build and compile Ceph

git clone git@github.com:the78mole/ceph.git
cd ceph
git checkout wip-32-bit-arm-fixes
./install-deps.sh
./do_cmake_arm32.sh # for armhf
# ./do_cmake.sh # for x86_64/amd64 or arm64
cd build
make -j4
# if it gets really slow due to swapping, break an do make -j1
# or use the scheduler-script from link below

Here you can find a rudimentary (but working) script that suspends compilers processes based on total compilers memory consumption. Running it through ‚watch‘-tool you can start e.g. 8 tasks and when memory limit is reached, it will suspend (kill -TSPT) the youngest tasks in sense of user space runtime.
https://github.com/the78mole/scripts/blob/master/linux/bash/schedule_compile.sh

Now do…

cd ..   # Back to ceph base dir
./make-debs-arm32.sh # for armhf
# ./make-debs.sh # fox x86_64/amd64 or arm64

If you encounter problems with setuptools (Exception –> TypeError: unsupported operand type(s) for -= ‚Retry‘ and ‚int‘) try to get a more recent version of python pip with the following commands and rerun make-debs-arm32.sh.

apt-get remove python-pip python3-pip
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
python3 get-pip.py

If I forgot anything to make it work, feel free to write some comment…

ESP32-EVB, PlatformIO And ESP-IDF – Yet Another ESP32 tutorial

I already posted a tutorial on using the ESP32-EVB from olimex with Arduino. This time, I will provide the same with ESP-IDF, the original SDK from Espressif. Why I decided to do this? Because I had a project using it 😛

To install PlatformIO, just follow the guide.

As shown in the below screenshots, create a new project VSCode -> The PIO-Icon -> New Project, define a project name, select Olimex ESP32-EVB with ESP-IDF framework and press Finish. After PIO downloaded all dependencies and configured your project, we are ready to go!

Now just wait until it finishes…

After PIO finished setting up the project, you will find the platformio.ini with its configuration. Add the line serial_speed = 115200 to the file.

After this, run an update on pio (especially useful if you had PlatformIO already installed).

pio update
pio upgrade --dev

We can also now build the empty application by entering pio run in the terminal.

If this succeeds, we will edit the main.c file and create some hello world application.

Hello World – Relay Toggle

Open src/main.c and edit it, so it will look like that:

#include <stdio.h>
#include "sdkconfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "driver/gpio.h"

#define RELAY_GPIO      32

void app_main() {

    gpio_pad_select_gpio(RELAY_GPIO); 
    /* Set the GPIO as a push/pull output */ 
    gpio_set_direction(RELAY_GPIO, GPIO_MODE_DEF_OUTPUT); 

    while(1) {
        printf("Test\r\n");
        gpio_set_level(RELAY_GPIO, 0);
        vTaskDelay(pdMS_TO_TICKS(500));
        gpio_set_level(RELAY_GPIO, 1);
        vTaskDelay(pdMS_TO_TICKS(500));
    }
}

…connect your Olimex ESP32-EVB and enter pio run -t upload in terminal.

Done! Your Relay should toogle twice every second.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code. We already prepared the test in our code above. It prints „Test“ every second on the serial console, just that we can not see it yet. For this to show up, you need to open the serial monitor:

From now on, you should see the printf output in your terminal every time you upload your code again:

Do not forget to close the serial monitor, when you upload your code next time. The serial is blocked by the monitor, so the uploader can not access this port.

After closing the serial monitor, you should drop back to your terminal, where you can start over with pio run -t upload.

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or bluetooth or whatever. You will find information on all peripherials here. Just remember to include the header(s) that are mentioned on top of every peripherial page, like we did with #include "driver/gpio.h" above in main.c.

Now, you have some great starting point for ESP-IDF 😛

Have fun coding!

ESP32-EVB, PlatformIO And Arduino – Yet Another ESP32 tutorial

What the heck? Aren’t there enough toturials out there about ESP32? I believe: Yes, too many. And there are too many that struggle with setting up the arduino environment for non-arduino hardware. But there is a straight-forward solution that works out of the box: platform.io or also known as PIO.

And there are enought tutorials about installing platform.io. It is as easy as you can think. Install Visual Studio Code from Microsoft and install the PIO plugin in VSCode.

Then create a new project VSCode -> New Project and select Olimex ESP32-EVB with Arduino Platform. After PIO downloaded all dependencies and configured your project, we are ready to go!

Hello World – Relay Toggle

Open main.cpp and edit it, so it will look like that:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

…connect your Olimex ESP32-EVB and hit the upload button:

Done! Your Relay should toogle every 10 seconds.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code.

Extend your code to look like this:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  Serial.begin(115200);
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  Serial.printf("Switch on\r\n");
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  Serial.printf("Switch off\r\n");
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

Add two lines to your platfromio.ini file. You will find the correct COM port in terminal when uploading the new code to your device.

In my case it is COM6 and platformio.ini looks like this:

[env:esp32-evb]
platform = espressif32
board = esp32-evb
framework = arduino
monitor_port = COM6
monitor_speed = 115200

Upload your code again after changing the ini file. If it does not print your output automatically to your terminal, start the monitor manually:

From now on, you should see the printf output in your terminal every time you upload your code again:

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or whatever. But now, you have some great arduino IDE without arduino IDE 😛

Have fun coding!

Sorting Your Digital Mess – How to Easily Set-Up a Private Search Engine

Motivation

In my vacations, I kicked-off some new projects. One of it is an ARM64 based SBC with integrated SATA mostly like the Odroid-HC1/2. The main difference is the ARM (64 vs. 32 Bit) to be able to run Ceph on it.

If you read some of my previous posts, you will notice, that I already put in some effort to make Ceph run on 32-bit controllers. But the effort was way to high, to make and keep it running. But the new SBC is another story, I will tell, when the time is ripe for it.

When I thought of putting all my data on a Ceph cluster, I just stumbeld over the probem, how to manage all these bits and bytes and how to keep the overview. The should be something like a Google-Search for your private data. When you dig through the net, you will find many sites, blogs and books about elastic search. But there is always a problem, how to get your data in there without a deep knowledge on data providers, ETL, graphs,… But you could also find OpenSemanticSearch (OSS) and I gave it a try.

A Private Search Engine – Open Semantic Search

To be honest, OpenSemanticSearch is not the most beatiful engine you could think of, but it gives you a very deep insight on your data, assuming you configured it correctly. Unfortunately, the documentation is quite sparse and there have not been many bloggers writing on the topic, at least not for some current version of it.

After you finished Installation, first do some deeper configuration. If you choose to use a VM, I suggest, to usa a RAM centric configuration. Better take 12+ GB of RAM and only 4 cores. Every core will lead to an etl-task that eats up a significant amount of RAM, depending on the data it is advised to index.

To avoid links of the WebUI to point you to some unreachable destination, do some configuration ahead of adding local crawler paths. If you mount your data using mount to a path in your filesystem on the OSS server, the link will be provided unchanged (e.g. /mnt/myData) and the URL will just be the location it is on your server without http, hostame, location-suffix,… Not really helpful, if you have mounted NFS shares and want to access it on a remote windows machine.

OSS provides already some document server, that proxies your requests to HTTP, but you need to configure it correctly. But if you do not configure it in the beginning, your whole indexing run will create wrong links and it will be hard to stop the indexing and do a rerun (see the troubleshooting section below). The task will be running, until it has done it’s job. For terabytes of data, this will last at least some days.

Installation and Configuration of OSS

Installation of OSS is quite straight forward, as described on their web page.

Configuration is a bit more complex due to the fact, that the documentation is a bit sparse. Some of it can be done with the web UI, but others are a bit more covered. But first things first.

Mounting Your Data

The first to do is, to make your data accessible to the indexer.

[TODO]

Configuring Apache to Proxy the Documents

The quick hack is, to add your path to proxy in /etc/apache2/sites-available/000-default.conf

Insecure Proxy for documents

This allows the documents to be accessed throuch http://<IP_OF_YOUR_OSS>/documents/mnt but also through http://<IP_OF_YOUR_OSS>/mnt. We will secure this later on. But for debugging, this is quite helpful.

To get the links in OSS right, you need also to adjust /etc/opensemanticsearch/connector-files to have a line with the following content

config['mappings'] = { "/": "http://172.22.2.108/documents/" }

You can also add more mappings, but this helps to get it right for the links in OSS Web UI.

(Re-)Starting the OSS server

I would advise to do a simple reboot of your server, to also check, if the shares get mounted correcly. For my debian buster, this is not the case. It is failing to mount the NFS shares for some reason, I have not yet digged down. I manually mount it, after it has startet. I assume, the systemd start dependencies are a bit buggy.

[Much more to write on… Coming soon]

Troubleshooting

So, what to do, if something goes wrong? E.g. if your decided to index a ton of data, it is time to purge the queue. But how?

Purging the Queue

OpenSemanticSearch uses RabbitMQ to organize the indexing tasks. If OSS decides to index a path, it will simply list all files in this path and put it in the open_semantic_etl_tasks-Queue of your RabbitMQ server. But the user interface of OSS does not provide a means of purging or deleting the content of the queue. But therefore, you neet to activate the rabbitmq management web ui.

sudo rabbitmq-plugins enable rabbitmq_management

After this, you can check, if your server listens on the apropriate interface (0.0.0.0 for acces from another host) and port (15672)

netstat -nlpt output of a typical OSS host with enabled RabbitMQ Web UI

This checked, we need to add a user with administrative rights to the OSS worker.

rabbitmqctl cluster_status   # Check if everything is fine
rabbitmqctl list_users       # Check if your webuser doesn't exist
rabbitmqctl add_user webuser <PASSWORD>
rabbitmqctl set_user_tags webuser administrator
rabbitmqctl set_permissions -p / webuser "." "." "."

This done, you can simply acces the user interface. Type in your browser http://<IP_OF_OSS_HOST>:15672/ and the login will be presented to you.

RabbitMQ Web Login

After loging in, head over to the queues and enter your open_semantic_etl_task-queue.

Queues view of RabbitMQ web UI

You should be presented with the queue details

Queue details of open_semantic_etl_tasks

In my example, you can see a very limited count of ready tasks. This can go up to a few thousand tasks when indexing a large directory with many files. Each file will add a task to this queue and RabbitMQ will hand this out to the etl workers of OSS.

To delete all messages in the queue, you can simply hit the Purge Messages button.

Purging the queue

After this, you can re-enqueue the index files on your OSS search site through http://<IP_OF_OSS_HOST>/search-apps/files/

FreeRTOS debugging on STM32 – CPU usage

Introduction

Since the information about FreeRTOS debugging with STM32CubeIDE is sparse and ST is not yet providing the task list view (that was part of the Atollic TrueStudio), here is, how you get it by installing a plugin from freescale and adding the approprite stuff to your code. I assume, you already have a project with FreeRTOS setup and running…

Adding the plugins

First start STM32CubeIDE and go to Help -> Install New Software…

Then add an Update Site by clicking the „Manage“-Button. Here you need to add the update site from freescale. And yes, NXP/Freescales plugin works with STM’s CubeIDE 🙂

http://freescale.com/lgfiles/updates/Eclipse/KDS

„Apply and Close“ and select the new site to „Work with“

Select the FreeRTOS Task Aware Debugger for GDB.

And click Next… Follow the Wizard until complete and after installation, restart your STM32CubeIDE.

Configuring the FreeRTOS project

Now add a timer and configure a reasonably a high tick rate (e.g. I used TIM13 of my STM32F469, running with 180 MHz HCLK, 90 MHz APB1 Timer clock and a timer counter period of 899 -> 100 kHz resolution).


Enable the interrupt

And in Middleware -> FreeRTOS, enable the run-time stats

If you like, you can also enable RECORD_STACK_HIGH_ADDRESS. Sometimes this is quite useful and avoids the little warning symbol in stack usage column of task list view.

Now regenerate your project…

Adjusting the code

Now it’s time to adjust your code for collecting the stats. Add a line for starting the timer in IT-mode by adding a function in some user code section in main.c.

volatile unsigned long ulHighFrequencyTimerTicks;

void configureTimerForRunTimeStats(void) {
  ulHighFrequencyTimerTicks = 0;
  HAL_TIM_Base_Start_IT(&htim13);
}

unsigned long getRunTimeCounterValue(void) {
  return ulHighFrequencyTimerTicks;
}

In stm32f4xx_it.c, add the following lines to the appropriate user sections

/* USER CODE BEGIN EV */
extern volatile unsigned long ulHighFrequencyTimerTicks;
/* USER CODE END EV */

[...]

void TIM8_UP_TIM13_IRQHandler(void)
{
  /* USER CODE BEGIN TIM8_UP_TIM13_IRQn 0 */
  ulHighFrequencyTimerTicks++;
  /* USER CODE END TIM8_UP_TIM13_IRQn 0 */
  HAL_TIM_IRQHandler(&htim13); 
  /* USER CODE BEGIN TIM8_UP_TIM13_IRQn 1 */
  /* USER CODE END TIM8_UP_TIM13_IRQn 1 */
}

If you are compiling with optimization levels above -O0, you also need to fix a bug (it is one in my opinion) in freertos tasks.c.

There are two possibilities:

  1. Switch of optimizations for tasks.c by right clicking on the file in project browser and changing the compiler optimization to -O0
  2. Change the line in tasks.c adding a volatile (see picture)

The problem with solution 2 is, that you need to do it after each STM32CubeMX code generation again. But there is a 3rd solution, that makes solution 2 persistend (until you update the MCU package).

Go to `%HOMEPATH%\STM32Cube\Repository\STM32Cube_FW_F4_V1.25.0\Middlewares\Third_Party\FreeRTOS\Source\` and edit the file like in solution 2, adding a volatile statement.

When you regenerate your project from CubeMX, it will include the correct line.

Profiling in action

Now after you put everything in place, it is time to run your code. Start the project in debugging mode, make the FreeRTOS/Task List view visible and let it run for some seconds. Then hit the pause button. The task list will collect the information from your target (from GDB) and show it nicely:

If the Task List view complains about FreeRTOS not have being detected, restart STM32CubeIDE and it should show up again.

Edit: During my last weeks of uing this Eclipse plugin, I had some problems seeing all tasks in Task Analyzer. In fact, the FreeRTOS functions to print the run-time statistics show them, while the plugin doesn’t. Also the data seems to be currupted sometimes within the plugin. So I would suggest, to better use the FreeRTOS internal stuff: https://www.freertos.org/rtos-run-time-stats.html

Citations

The information was collected from these links:

Just Another Hobby – 3D Printing

When it comes to leasure time, there are not many activities, that are more exciting than 3D printing. After you printed diverse things from Thingiverse or any other 3D printing gallery, there is a great wish, deeply buried in your mind, to design stuff yourself to get the real excitement — and what is more obvious than upgrading your 3D printer with parts printed by itself/ yourself 😛

If this was the right appetizer, look how my Ender3 got upgraded with a full metal frontend or take a look at my Thingiverse profile with some other designs. If you have any questions, especially the decisions, e.g. why I designed the new hotend exactly as it is (a wide monster hotend) or suggestions, how to improve it, do not hesitate to comment!

     

STM32 FreeRTOS and printf

After some more coding, I found some more issues with FreeRTOS and printf, not being solved by my fix below. If you need to get it fixed completely, look at that forums post: ST Community
and the website of Dave Nadler: newlib and FreeRTOS
In my current project, I replaced the newlib-nano-printf implementation by adding github:mpaland/printf as a git submodule to my project and including the printf.h (it overwrites the printf-library function with macro defines) in my topmost header file.

This will be a very short post. If you experience hard faults when using printf (this happens mostly, when using floats) and you already ticked the appropriate settings in the project’s properties…

…don’t waste your time digging through assembler instructions with instruction stepping (like I did) just to realize, that memory management is broken when using FreeRTOS. It is simply a bug in CubeMX-generated source files. Locate your _sbrk-function (either in syscalls.c or in sysmem.c) and change it to the following:

caddr_t _sbrk(int incr)
{
  extern char end asm("end");
  static char *heap_end;
  char *prev_heap_end,*min_stack_ptr;
  if (heap_end == 0) 
    heap_end = &end; prev_heap_end = heap_end; 
  /* Use the NVIC offset register to locate the main stack pointer. */
  /* Locate the STACK bottom address */
  min_stack_ptr = (char*)(*(unsigned int *)*(unsigned int *)0xE000ED08);
  min_stack_ptr -= MAX_STACK_SIZE; 
  if (heap_end + incr > min_stack_ptr) {
    errno = ENOMEM;
    return (caddr_t) -1;
  }
  heap_end += incr; 
  return (caddr_t) prev_heap_end;
}

For what _sbrk does, have a look here.

If you want to digg a bit deeper, here are some websites dealing with this problem:

Doxygen – Tips and Tricks

LaTeX non-interactive

To make LaTeX skip some errors without user interaction, you can add the option --interaction=nonstopmode to the pdflatex call. Easiest way to do so, is changing the LATEX_COMMAND_NAME in your Doxyfile.

LATEX_CMD_NAME = „latex –interaction=nonstopmode“

Do not forget the double quotation marks. Otherwise doxygen will remove the space and the command in your make.bat will fail.

If you now want to generate the code, step into your doxygen-generated latex folder (designated by LATEX_OUTPUT option in Doxyfile) and execute make.bat (on Windows) or make all (on *nix).

Adding a favicon to html output

Adding a favicon to html output, you need to specify it in a custom header and include the original image in HTML, as described here. To extract the default header file:

doxygen -w html headerFile

Add the follwing line to in headerFile within the html header

<link rel="shortcut icon" href="favicon.png" type="image/png">

And add your headerFile and the image to the HTML_EXTRA_FILES in your Doxyfile. Its path is relative to your Doxyfile.

HTML_HEADER = headerFile
HTML_EXTRA_FILES = some_rel_path/favicon.png

Now you can generate your html documentation with some favicon in place.

PDF output destination

Did you ever search for the PDF file, doxygen (or better the Makefile in latex) generates? I just added an option to doxygen, copying the refman.pdf to a location of your choice. (Hopefully it soon get’s merged and released).

If you want to test it out? Compile doxygen from my doxygen fork and add the following option to the Doxyfile of your project.

PDF_DST_FILE = ../MyGenerated.pdf

The destination is relative to your Makefile in your doxygen latex folder. As soon as make finished it’s job, the PDF is just in the same folder, the latex folder resides in.

That’s all. Enjoy generating software documentation with doxygen

OrangePi 4G-IoT Complete Pack

Since Andorid 8.1 is quite slow on the OrangePi 4G-IoT, I decided to give Andorid 6 a try. OrangePi.org also provides mega.nz-links for these packages, which is quite inconvenient… Here is a torrent, that contains all stuff for the 4G-IoT. In many torrent clients, you can choose, which files should be downloaded…

https://downloads.the78mole.de/OrangePi-4G-IoT_Full.torrent

I’ve repackaged the tars, since xz is more dense than gz. It saves around 50% of space and traffic.

Enjoy the 4G-IoT 😉

Compiling Software on RAM-limited Multi-Core Systems

Since I often compile stuff on embedded ARM targets that are well equipped with processing power (Exinos Octa-Core), but are neglected regarding RAM (2G), I often facing the trade-off between running multiple or only a single/few compilation jobs (make -j8 vs. make -j1). If you start too many jobs and if you have large compilation units (e.g. with the Ceph Project sources), the system will feel like jam, as soon as it begins swapping.

I feel, that deciding the job count at the very beginning is (was) a trade-off, I was not willing to accept. Therefore, I decided to write a little script to suspend compile processes, that cross a certain memory limit. This way, the suspended processes get moved to swap and the still runnig processes get a comfortable amount of RAM. This way, the kernel is not forced to move pages around with every scheduling round. Instead, it will move it once on swap for suspended processes, when it needs RAM for the running ones and as soon as the processes with large memory footprint finish, the supended ones get back to live.

Welp, I decided to base the priority on the time the processes ate up user space processing time, so the older ones (often the most memory hungy ones) get processed first. This scheduling scheme proofed to be a optimal descision, that is also not hard to implement as a bash script.

Here you can find the little script, that needs to be run within a loop or simply with the watch-tool (maybe with sudo).

watch scripts/linux/bash/schedule_compile.sh

Happy compiling!

Cookie Banner von Real Cookie Banner