Think Sustainable, TH!NK City (reanimated) – Work in Progress

This is the first large project, I will document on this blog. Since it is a work in progress, there is no guarantee it will come to a happy end. Stay tuned and also watch the progress on YouTube on my favorite channel Steve&Julian#IGEMBB

What Is It All About?

Ever heard about the TH!NK City electric car? No? Before I got asked to help bringing it to life again, I also didn’t. In fact, if you do not count the first cars at all, which have been electrical, too, it was one of the first electric cars, at least in the 21st century, that entered serial production somehow. This happened a few years, before Tesla was born. Now many TH!NK Cities exist, that have been produced around 2010 and now living their dire existence in a corner of some garage. During their past few years without charging and mainteanance, their battery degraded to a state that they could not be brought back to live by a simple repair.

There are already some guys that reanimated the TH!NK City, but there needs to be some easier solution, which can be done by skilled handyman. Here are some of the public projects:

I’m part of the latter project 😛 and you will find design files and documentation in my GitLab project.

About TH!NK City’s Problem and The Energy Storage

The TH!NK City was produced with two types of battery. The first iteration was a ZEBRA battery, made of NaNiCl and has a working temperature of around 300°C. This batteries mostly survived and just need to be heated up. The second iteration was a LiPo-Stack with a 96s4p (mixed from 12p2s blocks) configuration. Many of the packs show degraded cells, going down to only a few mV, which means, the cells are damaged with lasting „effects“ and it will not be safe to reanimate them. Nevertheless, it can be done, but it needs special attention and maybe replacement of some of them or a total recombination with less power and energy. In our project, we will first reanimate the defect cells, measure the conditions, decide on what to do and then bring the TH!NK back to life.

To tackle the problem, we will first collect some data about the battery and the individual cells without damaging too much of it all. After disassembling the battery to inidividual modules, we need to get access to the single cells (in fact to the 2p-pairs). Measuring the connector pins and looking at some photos of Arndt’s blog, we identified the correct pins. A1-A11, B1-B10, A18-A21 and B18-B21 are potential candidates for the thermistors/NTCs. This arises from the pictures of the cell connection foils from Arndt’s blog.

Cell taps on the pack’s balancing connector

We (in fact it was all Eric) then measured the voltages of all cells in the TH!NK battery and found a bunch of low and zero volt cells. Cells below 2.5V are presumed to be pre-damaged, cells below 2.0V could be counted as dead. But as I told, we try to wake them up for some tests, before we replace them or re-organize the stack.

Cell voltages of all battery packs in our walk-in patient

First Step – Reanimate The Dead

We (again Eric did it) then charged the dead cells through their balancing connector with only a few hundred milli-amps to bring it up to around 3V. After that, we connected an iCharger X12 to charge the packs and pull out some data of the pack and it’s cells.

First measurements after precharging the defect cells of one pack showed promising results. Cell 12 was a zero volts candidate.

To have an easier access to the pins of the balancer connector (and to lower the risk of short circuits), I designed some adapter to directly connect the iCharger X12 balancer cable (find the design files in the mentiones GitLab project). By having multiple assembly options for the JST-PH (S13B-PH-K-S or S13B-PH-SM4-TB) balancing connector, we can adapt to different cables (1:1 = mirrored, cross = straight) to fit the polarities correctly. The following image shows the assembly for a crossed cable. The middle connector is mainly to more easily measure the cell voltages or to find the terminals for the temperature sensors. The pin names are the same as on the ELEC unit (the TH!NK BMS) shown in the nect section.

Update (2021-05-28) – Assembly Notes

When you assemble the 2X21 horizontal connectors, you should lift it a bit. The battery connector is a bit to thick, to accomplish with the 2X21 attached closely to the PCB. Best would be to solder the connector with attached cable and only fix it by the outer pins (A1, B1, A21, B21). Be careful not to short circuit the balancing pins.Then remove the cable and solder the remaining pins.

The JST-PH-Connector could be to narrow. It fits the Balancing port of the X12 exactly. But many Balancing cables have a XH type on the opposite end. This will not fit my adapter. I will soon optimize the adapter and provide an updated BOM.

Analysis Results of The Full Battery Stack

Cautious Charging

During our first, cautious charging attempts with one battery pack (12s2p), we observed the zero-cell (the one that had only a few millivolts in the beginning) loosing voltage during phases without charging, while all other cells kept their voltage tightly. Additionally, after having equal voltages across the pack, the balancer had most work on the 0-cell during charging and the capacity counter showed, that it got much less charge than other cells.

Getting to Speed – Cycle all Batteries

TO BE DONE…

Update (2021-05-28) – Zero-Volt Cells Behaviour

Cell 12 of the Pack #4 is discharging itself down to 0V within a few days. This means, we need to somehow get rid of these 0V-Cells. But first finish the inventory of all Cells…

The Annoying Thing – The ELEC (TH!NK BMS)

The most annoying thing with the TH!NK is it’s BMS. The PCB is coated with some plastics preventing you from measuring the signals, components and even make it hard to replace defect parts. If it would only fulfill it’s purpose…: Protection against humidity and water… But it seems, that humidity causes the BMS to blow up and discharge the lowest cell or even more of them over time.

If you look at some of the BMS, you will discover blown parts.

A detailed analysis of the circuit uncovers a discretely designed balancer circuit, a microcontroller (NXP), an isolated CAN driver and some isolated power supply powered by 12V rails (not from the battery pack).

image

Not yet finished analysing, but this is what could be revealed up to now from one of the balancing stages:

image

The circuit reveals, that the magician was some enthusiastic analog designer. You can see a MOSFET, connecting the battery to the resistive load and a level shifting cuircuit that drives the gate relative to it’s source but keeping the gate voltage within the allowed limits. Most interesting is the common-base amplifier circuit (Q3), I’ve not seen for a long time. It’s most favorable property it the low current and high voltage amplification, also called a current buffer with voltage amplification.

Thoughts about repairing or replacing (same or new) the TH!NK BMS

While a repair is the most economic way to bring the TH!NK back to life, it has some major drawbacks. The most important drawback also applies to a spare part: It has a bug that damages the cells (at least when not used/charged for some time). The other major drawback is the low balancing current and it’s passiveness. The large difference of the cells renders the good ones mostly useless, since a passive balancer is always limited to the worst cell in a serial string. This leads to a bad overall capacity, when only a few cells degenerate (worst case one 2p-cell-pair in each string).

Another problem is, that it is hard to find the appropriate semiconductors (transistors, diodes,…). You need to pull some good ones off the board, measure it, characterise it and find a good spare part.

An easier solution is to replace the whole BMS. But these are hard to source and since there exists a lot of documentation for the CAN communications (TH!NK A306 Remote Lithium Energy Controller (RLEC) CAN Programmers Guide), we will give it a try to design an own BMS for the TH!NK. We also will advance it’s features by optionally adding active cell and pack balancing to compensate for strong variations of the cells. Most TH!NKs nowadays will encounter it.

With passive Balancing, you will only get the performance of the worst cell-twin in a string (96s2p) and all better cells will used only partly. Assume, a single cell has only 17Ah of health left, while 34Ah are nominal. Even if all other cells in the string have 30Ah, they only can provide 17Ah. This means a loss of 50%@34Ah (43%@30Ah) of capacity, even if the string in total has 88% of capacity left.

Building our own BMS

Nowadays, you can source integrated circuits for LiPo battery monitors/supervison with balancing and various other features for a tiny amount of money. Together with some isolated CAN transceiver, a tiny microcontroller and some bird food, we will build our onw BMS. So our first iteration will be, to build a passive BMS with the option to upgrade to an active one…

TO BE CONTINUED

THANK YOU GUYS!!!!

I also want to thank the project team members that did most of the work getting the TH!NK to life again. Since I live in Erlangen, Bavaria, Germany and the TH!NK City is located near Berlin/Brandenburg, Germany the guys living there did most of the time consuming work with the car and it’s battery. Special thanks go to Eric (charging and analysing the battery packs), Klaus (for providing an appartment when I was in Berlin to support the team), Steve & Julian (which pulled me into this project and for their great YouTube channel) and all other guys for fertile discussions and ideas (DermitdemTiger, Thomas G. & B., Volker J. & S., Henning, Ronald, Reiner, Matthias, Hans, William)

How to Wire an EV Wall-Box (the economic perspective)

Have you ever planned to install some wall-box for your new electric car? You have some technical/electrical skills but you are not sure, which wire to run from your building connection to the place where your wall-box will be installed? Then you found the right place to get some deeper thoughts about your intent.

In germany, there is currently some promotional program to get 900 € when you install some private wall-box to charge electric cars. There are also some constraints to get the money from government:

  • location shall be in germany and at some residential building
  • the total price (wall-box + installation) needs to be at least 900 €
  • installation needs to be done by an expert (running a business and can hand out an official invoice)
  • the power shall be 11 kW (at least limited by firmware)
  • the wall-box neds to be smart (capable to be controlled by network operator)
  • the wall-box needs to be on the KFW440 list

I will assume, you already know, that you need an RCD Type B, if no DC fault current protection is integrated into your wall-box and all that other safety stuff. Nevertheless, in germany you will need an expert to get the sponsorship.

First of all, you should decide, if you want a software or hardware limited 11 kW equipment. In my opinion, even 3,7 kW will suffice, because you usually charge your car over night or during the day and for ultra-fast charging you will end up at some HPC station near some highway…

For further calculations, I’ll assume, the cable run length from your building-connection to your wall-box is 10 meters and the wall-boy is of 11 kW type (not more). This means, you have 3 current carrying wires (3-phase) and 16A of current per phase. If you now simply look at some tabluar overview of DIN VDE 0298-4:2013-06, it says, you need at least 2.5mm² for this current (regardless where you run the wires, since 2.5 mm² can carry 24 A at least if used inside a protective tube).

OK, we are done, run NYM-J 5G2.5, that’s all…

For sure, not! Let’s take a little time to think about losses. 11 kW and 3 x 16A is a huge amount of power that is running through your wires and the maximum current from the standards just takes the self heating into account. If you intend to keep your pavement free of ice in wintertime, this is what you are searching for. Run the 2.5mm² wire up and down your pavement and you are done with it.

But now lets calculate the losses:

  • Copper wire (0.0171 Ω*mm²/m) of 2.5 mm² has a resistance of 6.84 mΩ/m
  • 10m@16A => P = R * I² => 10 m * 0.00684 Ω * 16² A = 17,5 W per Phase => 52,5 W losses

Compared with 11 kW, 52W seem not much, but let’s continue. Assume, our car has 50 kWh of capacity and the battery has a lifetime of 2500 (0-100%) cycles. This means, we have a lifetime charge of 125 MWh. If we cycle it with 11 kW, it is equal to 11,364 hours of charging. Multiply this with 52,5 W and you get 593 kWh of losses. With a price tag of 0,30 €/kWh, this means 178 € of useless heat (for your first car…, second car adds the same price tag for heat only). We can normalize this back to 17,8 €/m of wire. I know, the losses will not go into your battery, and need to be supplied additionally, but it’s just a first order approximation. It will be a bit worse in reality. The losses also do not mean a lot, but you can easily find a sweet spot with common wire gauges. Let’s build up a little table for them:

Wire GaugeWire price
(NYM-J 5Gxxx €/m)
Losses
€/m
Total
€/m
2.5 mm²2.4017.7820.18
4 mm²3.2311.1114.34
6 mm²5.007.4112.41
10 mm²7.424.4411.86
16 mm²11.892.7814.67
25 mm²18.211.7819.99
35 mm²29.191.2730.46
Material and lifetime cost for different wire gauges per meter of length

Just for completeness, the NYY-J type (this one can be put into ground without protective tubes) has approximately the same price tag (2.20 €/m@2.5mm², 6.83 €/m@10mm²). Did you expect, that the 10 mm² cable has the best economic efficieny? OK, how long it will take to cycle the battery 2500 times? If you estimate 20 kWh/100km, we can run 625,000 km on this battery.

I did not yet dig the wire (yes, this is what moles do) below earth and also did not buy a wall-box (it’s hard to get some electrician around my hometown), but I will decide on the 10 mm². This also gives headroom for some 22 kW wall-box after the minimum run time at 11 kW, given by the sponsorship rules.

Now have fun to dig the wires and give me some feedback on this article.

Angular Web App on ESP32

In progress…

TL;DR (take me to the battle field)

Introduction and Motivation

In some recent project, we added some ESP32 to replace wired communication with a BLE solution. Additionally, the customer wants to have some administrative web application to configure the device more easily and to get more detailed information in service.

Creating a simple web ab is not a hard task for an embedded C developer. But to create a beatiful or at least not shitty looking web application is just another game. When you look more closely on the available space on some ESP32, you will realize, that there is at most 16 MB of space to live with.

If you create a simple WiFi AP on some ESP32 4MB version with ESP-IDF, you can easily see, that there is not much left for BLE and APP.

Build statistics for a simple WiFi AP with only a hello world index.html on some lolin32 board

Unfortunately, I only have the 4 MB version available, which complicates the task even more. And no, I can not simply change to 16MB, since a few hundred units are already in operation, waiting for some firmware update to provide WiFi service. You know how this works… Deliver the hardware as early as possible and hand features in later 😛

But this means, there is much headroom for optimization and not much space for resources and code. If you have a look at web application frameworks (e.g. react) and build some basic apps, they all come with megabytes of resources in the production build output folder. If you build a very basic hello-world angular application, you will find a few 100 kilobytes of resources after building it, to be deployed on a static web space. This is also just small enough, to be fitted in the flash.

Now, let’s talk about the solution…

Prerequisites

First of all, you will need to create an angular and an ESP-IDF project. This is easily done, if you already installed Visual Studio Code with PlatformIO.

To create an angular project, just follow the official guide. If you installed npm correctly, you can issue the commands in your Visual Studio Code Terminal, to fire up the project.

ng new esp-on-angular

After these steps, you have two separate projects. You can easily setup a script to copy the angular production build over to the ESP project, do it manually or link it in using git submodules. But this is not part of this tutorial. We simply copy over the website content manually from the angular project output folder.

In addition to npm, VSCode and PlatformIO, you could need some basic Linux system, when working on Windows, if you intend to use the automation script I’l provide later. On my machine, I use MSYS2/MinGW to get the CLI tools I need: bash, find, gzip, stat

Getting Started

Firmware Web Content Integration

While it is quite easy to simply handling incoming requests and responding to it, like you would do for some RESTful Web Service (which is the usual case for IoT stuff like the ESP32), integrating plain files is usually a more complex task. Thanks to the ESP-IDF, this is not the case.

In general, there are two options. Adding the files to a SPIFFS partition and reading it from there. Another solution is to embed the files in your firmware object. The latter one is what we will use in this tutorial, since it is more dense (in terms of size), althought it is less flexible and requires a bit more lumber in your config.

For it to work, simply add the files to board_build.embed_files environment in your platform.ini file of your ESP project. I usually put these ressource files into a separate folder called res.

board_build.embed_txtfiles = 
    res/index.html
board_build.embed_files =
    res/logo.png

When you did so, you can link the content easily with the following lines in your code, PlatformIO will care for it to be a valid symbol and to be linked in:

extern const uint8_t index_html_start[] asm("_binary_index_html_start");
extern const uint8_t index_html_end[]   asm("_binary_index_html_end");
extern const uint8_t logo_start[] asm("_binary_logo_png_start");
extern const uint8_t logo_end[]   asm("_binary_logo_png_end");

When you want to use that data in your http handler, just use the pointers declared above:

httpd_resp_set_type(httpRequest, "text/html");
httpd_resp_send(httpRequest, (const char*) 
    index_html_start, HTTPD_RESP_USE_STRLEN);

…or for your png image file…

httpd_resp_set_type(httpRequest, "image/png");
httpd_resp_send(httpRequest, (const char*) 
    logo_start, logo_start - logo_end);

I personally use only the binary version, since it can also process text files and makes the code more universal. You will later see, that there will not be much of a text left 🙂

The symbols of your file object (the part in the asm function) is built by concatenating _binary_ with the name of your file with all special chars replaced by underscores _ and the suffix _start and _end respectively. In short: _binary_<FILENAMEREPL>_start and _binary_<FILENAMEREPL>_end.

Welp, that is quite fine, but the files still take up quite some space and we do not have a lot. When using the 4M version of the ESP32, we simply can not allow it to do so. So, what can we do further to reduce the size of our resources?

You could ask: Why not push the images and fonts to a publically accessible location in the wolrd wide web? Because some of us want to make it self contained. The reason for me have been simply the security and safety requirements of the project: Keep the system closed to the outside world, provide some WiFi Access Point and serve all files that are needed yourself. Do not depend on internet in general. If you have other requirements, kick your ESP in your local WiFi and access it from there or use the AP/Client mode, where the ESP connects to a wifi as a client, provides a WiFi AP and acts as a router between these two networks. But be warned, don’t expect too much performance from the latter option.

Welp, back to the size problem… What to do to make it fit?

Some quirks to shrink it down

Optimizing the ESP-IDF libraries and build scripts to free up some flash, is maybe quite an intensive task and needs a deep understanding of the IDF internals. So, we will first try to find some other quirks, to reduce the size of our own big blobs.

Angular – Stripping the unneeded

While angular itself is really tiny, it links in some fonts from google (Remeber, we want to be self contained). This is usually not needed, because devices nowadays have some replacement fonts installed, we can easily delete the links from the generated index.html and rely on the local resources of the client.

<link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">

Zero Effort Zipping

While Angular already minimizes most of it’s output files, it would be really awesome to compress the resources somehow. But integrating a compression library and using it, stacks up again…

Then I had an flash of thought. Once upon a time, web speed analyzis tools started to raise warnings, when your website content was not delivered in a compressed format. Nowadays, webspace providers enable brotli and/or gzip compresson by default and there are plenty of tutorials available about WordPress and how to enable compression. All modern web browsers support gzip compression by default and offer it as a supported content encoding sheme.

Welp, but what this has to do with my web application content. Shall the ESP compress all content ahead of the transfer? Of course not! We simply compress all files ahead of linking the firmware and simply fake the content type to be the original while setting the content encoding to gzip. This way, the ESP is completely liberated from compression and we can reduce the size of our firmware as much as possible.

Do I need to compress all files manually and somehow save the original content type? Not really. I wrote a simple bash script, that does most of the work for you and creates some header with a look-up table (LUT) for the code to access it easily. A tiny URI-router will search it within the LUT and pick all information it needs from it.

Putting the pieces together

The Angular Project

The ESP Project

After we created the ESP project as described in my other post, we will start implementing a simple web service. This is done with

ESP32-EVB, PlatformIO And ESP-IDF – Yet Another ESP32 tutorial

I already posted a tutorial on using the ESP32-EVB from olimex with Arduino. This time, I will provide the same with ESP-IDF, the original SDK from Espressif. Why I decided to do this? Because I had a project using it 😛

To install PlatformIO, just follow the guide.

As shown in the below screenshots, create a new project VSCode -> The PIO-Icon -> New Project, define a project name, select Olimex ESP32-EVB with ESP-IDF framework and press Finish. After PIO downloaded all dependencies and configured your project, we are ready to go!

Now just wait until it finishes…

After PIO finished setting up the project, you will find the platformio.ini with its configuration. Add the line serial_speed = 115200 to the file.

After this, run an update on pio (especially useful if you had PlatformIO already installed).

pio update
pio upgrade --dev

We can also now build the empty application by entering pio run in the terminal.

If this succeeds, we will edit the main.c file and create some hello world application.

Hello World – Relay Toggle

Open src/main.c and edit it, so it will look like that:

#include <stdio.h>
#include "sdkconfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "driver/gpio.h"

#define RELAY_GPIO      32

void app_main() {

    gpio_pad_select_gpio(RELAY_GPIO); 
    /* Set the GPIO as a push/pull output */ 
    gpio_set_direction(RELAY_GPIO, GPIO_MODE_DEF_OUTPUT); 

    while(1) {
        printf("Test\r\n");
        gpio_set_level(RELAY_GPIO, 0);
        vTaskDelay(pdMS_TO_TICKS(500));
        gpio_set_level(RELAY_GPIO, 1);
        vTaskDelay(pdMS_TO_TICKS(500));
    }
}

…connect your Olimex ESP32-EVB and enter pio run -t upload in terminal.

Done! Your Relay should toogle twice every second.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code. We already prepared the test in our code above. It prints „Test“ every second on the serial console, just that we can not see it yet. For this to show up, you need to open the serial monitor:

From now on, you should see the printf output in your terminal every time you upload your code again:

Do not forget to close the serial monitor, when you upload your code next time. The serial is blocked by the monitor, so the uploader can not access this port.

After closing the serial monitor, you should drop back to your terminal, where you can start over with pio run -t upload.

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or bluetooth or whatever. You will find information on all peripherials here. Just remember to include the header(s) that are mentioned on top of every peripherial page, like we did with #include "driver/gpio.h" above in main.c.

Now, you have some great starting point for ESP-IDF 😛

Have fun coding!

Sorting Your Digital Mess – How to Easily Set-Up a Private Search Engine

Motivation

In my vacations, I kicked-off some new projects. One of it is an ARM64 based SBC with integrated SATA mostly like the Odroid-HC1/2. The main difference is the ARM (64 vs. 32 Bit) to be able to run Ceph on it.

If you read some of my previous posts, you will notice, that I already put in some effort to make Ceph run on 32-bit controllers. But the effort was way to high, to make and keep it running. But the new SBC is another story, I will tell, when the time is ripe for it.

When I thought of putting all my data on a Ceph cluster, I just stumbeld over the probem, how to manage all these bits and bytes and how to keep the overview. The should be something like a Google-Search for your private data. When you dig through the net, you will find many sites, blogs and books about elastic search. But there is always a problem, how to get your data in there without a deep knowledge on data providers, ETL, graphs,… But you could also find OpenSemanticSearch (OSS) and I gave it a try.

A Private Search Engine – Open Semantic Search

To be honest, OpenSemanticSearch is not the most beatiful engine you could think of, but it gives you a very deep insight on your data, assuming you configured it correctly. Unfortunately, the documentation is quite sparse and there have not been many bloggers writing on the topic, at least not for some current version of it.

After you finished Installation, first do some deeper configuration. If you choose to use a VM, I suggest, to usa a RAM centric configuration. Better take 12+ GB of RAM and only 4 cores. Every core will lead to an etl-task that eats up a significant amount of RAM, depending on the data it is advised to index.

To avoid links of the WebUI to point you to some unreachable destination, do some configuration ahead of adding local crawler paths. If you mount your data using mount to a path in your filesystem on the OSS server, the link will be provided unchanged (e.g. /mnt/myData) and the URL will just be the location it is on your server without http, hostame, location-suffix,… Not really helpful, if you have mounted NFS shares and want to access it on a remote windows machine.

OSS provides already some document server, that proxies your requests to HTTP, but you need to configure it correctly. But if you do not configure it in the beginning, your whole indexing run will create wrong links and it will be hard to stop the indexing and do a rerun (see the troubleshooting section below). The task will be running, until it has done it’s job. For terabytes of data, this will last at least some days.

Installation and Configuration of OSS

Installation of OSS is quite straight forward, as described on their web page.

Configuration is a bit more complex due to the fact, that the documentation is a bit sparse. Some of it can be done with the web UI, but others are a bit more covered. But first things first.

Mounting Your Data

The first to do is, to make your data accessible to the indexer.

[TODO]

Configuring Apache to Proxy the Documents

The quick hack is, to add your path to proxy in /etc/apache2/sites-available/000-default.conf

Insecure Proxy for documents

This allows the documents to be accessed throuch http://<IP_OF_YOUR_OSS>/documents/mnt but also through http://<IP_OF_YOUR_OSS>/mnt. We will secure this later on. But for debugging, this is quite helpful.

To get the links in OSS right, you need also to adjust /etc/opensemanticsearch/connector-files to have a line with the following content

config['mappings'] = { "/": "http://172.22.2.108/documents/" }

You can also add more mappings, but this helps to get it right for the links in OSS Web UI.

(Re-)Starting the OSS server

I would advise to do a simple reboot of your server, to also check, if the shares get mounted correcly. For my debian buster, this is not the case. It is failing to mount the NFS shares for some reason, I have not yet digged down. I manually mount it, after it has startet. I assume, the systemd start dependencies are a bit buggy.

[Much more to write on… Coming soon]

Troubleshooting

So, what to do, if something goes wrong? E.g. if your decided to index a ton of data, it is time to purge the queue. But how?

Purging the Queue

OpenSemanticSearch uses RabbitMQ to organize the indexing tasks. If OSS decides to index a path, it will simply list all files in this path and put it in the open_semantic_etl_tasks-Queue of your RabbitMQ server. But the user interface of OSS does not provide a means of purging or deleting the content of the queue. But therefore, you neet to activate the rabbitmq management web ui.

sudo rabbitmq-plugins enable rabbitmq_management

After this, you can check, if your server listens on the apropriate interface (0.0.0.0 for acces from another host) and port (15672)

netstat -nlpt output of a typical OSS host with enabled RabbitMQ Web UI

This checked, we need to add a user with administrative rights to the OSS worker.

rabbitmqctl cluster_status   # Check if everything is fine
rabbitmqctl list_users       # Check if your webuser doesn't exist
rabbitmqctl add_user webuser <PASSWORD>
rabbitmqctl set_user_tags webuser administrator
rabbitmqctl set_permissions -p / webuser "." "." "."

This done, you can simply acces the user interface. Type in your browser http://<IP_OF_OSS_HOST>:15672/ and the login will be presented to you.

RabbitMQ Web Login

After loging in, head over to the queues and enter your open_semantic_etl_task-queue.

Queues view of RabbitMQ web UI

You should be presented with the queue details

Queue details of open_semantic_etl_tasks

In my example, you can see a very limited count of ready tasks. This can go up to a few thousand tasks when indexing a large directory with many files. Each file will add a task to this queue and RabbitMQ will hand this out to the etl workers of OSS.

To delete all messages in the queue, you can simply hit the Purge Messages button.

Purging the queue

After this, you can re-enqueue the index files on your OSS search site through http://<IP_OF_OSS_HOST>/search-apps/files/

Just Another Hobby – 3D Printing

When it comes to leasure time, there are not many activities, that are more exciting than 3D printing. After you printed diverse things from Thingiverse or any other 3D printing gallery, there is a great wish, deeply buried in your mind, to design stuff yourself to get the real excitement — and what is more obvious than upgrading your 3D printer with parts printed by itself/ yourself 😛

If this was the right appetizer, look how my Ender3 got upgraded with a full metal frontend or take a look at my Thingiverse profile with some other designs. If you have any questions, especially the decisions, e.g. why I designed the new hotend exactly as it is (a wide monster hotend) or suggestions, how to improve it, do not hesitate to comment!

     

STM32 USB-DFU

I’m not sure, if I’m simply a problem magnet or why some stuff does not work as described… Here is another case. The tool I tend to use can be found here at STM’s site.

When I connected a piece of custom designed hardware with my laptops USB port with BOOT0 tied to VCC, it immediately showed up a new USB-Device called „STM32 Bootloader“ in group „USB Devices“. I was really happy about that and started the DFuSe Demo Software from STM. But hey, it could not find an appropriate device. What the heck???

After some digging through the web, I found different suggestions and problem solutions, but none of it worked.The simple solution was, to uninstall the device’s driver in „Device Manager“, unconnect and reconnet again. A different device showed up just a goup above the previous one: „USB Controller“.

Welp, as soon as the device showed up, also the DFuSe Demo Software recognized the device and stepped into life.

To create a DFU file is well described in all the other resources on the web. Just use the DFU file manager that was installed along with the DFuSe Demo Software to create a DFU file out of e.g. a hex file.

Then press the lower „Choose…“ button to select the generated DFU file and press „Upgrade“… That’s all. Your device has a new Firmware on it.

Have fun!

STM32 FreeRTOS and printf

After some more coding, I found some more issues with FreeRTOS and printf, not being solved by my fix below. If you need to get it fixed completely, look at that forums post: ST Community
and the website of Dave Nadler: newlib and FreeRTOS
In my current project, I replaced the newlib-nano-printf implementation by adding github:mpaland/printf as a git submodule to my project and including the printf.h (it overwrites the printf-library function with macro defines) in my topmost header file.

This will be a very short post. If you experience hard faults when using printf (this happens mostly, when using floats) and you already ticked the appropriate settings in the project’s properties…

…don’t waste your time digging through assembler instructions with instruction stepping (like I did) just to realize, that memory management is broken when using FreeRTOS. It is simply a bug in CubeMX-generated source files. Locate your _sbrk-function (either in syscalls.c or in sysmem.c) and change it to the following:

caddr_t _sbrk(int incr)
{
  extern char end asm("end");
  static char *heap_end;
  char *prev_heap_end,*min_stack_ptr;
  if (heap_end == 0) 
    heap_end = &end; prev_heap_end = heap_end; 
  /* Use the NVIC offset register to locate the main stack pointer. */
  /* Locate the STACK bottom address */
  min_stack_ptr = (char*)(*(unsigned int *)*(unsigned int *)0xE000ED08);
  min_stack_ptr -= MAX_STACK_SIZE; 
  if (heap_end + incr > min_stack_ptr) {
    errno = ENOMEM;
    return (caddr_t) -1;
  }
  heap_end += incr; 
  return (caddr_t) prev_heap_end;
}

For what _sbrk does, have a look here.

If you want to digg a bit deeper, here are some websites dealing with this problem:

ESP32-EVB, PlatformIO And Arduino – Yet Another ESP32 tutorial

What the heck? Aren’t there enough toturials out there about ESP32? I believe: Yes, too many. And there are too many that struggle with setting up the arduino environment for non-arduino hardware. But there is a straight-forward solution that works out of the box: platform.io or also known as PIO.

And there are enought tutorials about installing platform.io. It is as easy as you can think. Install Visual Studio Code from Microsoft and install the PIO plugin in VSCode.

Then create a new project VSCode -> New Project and select Olimex ESP32-EVB with Arduino Platform. After PIO downloaded all dependencies and configured your project, we are ready to go!

Hello World – Relay Toggle

Open main.cpp and edit it, so it will look like that:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

…connect your Olimex ESP32-EVB and hit the upload button:

Done! Your Relay should toogle every 10 seconds.

The pin numbers are simply the GPIO numbers you can find in the schematics of your board, in this case, it is 32.

printf-Debugging

Despite the fact, that printf debugging is somewhat frowned upon, it is a very practical first step. And with platform.io and evaluation boards like this one, it is as easy as pressing a button. No additional wiring, no hassle with pins, ports or blown up code.

Extend your code to look like this:

#include <Arduino.h>

const int relay1pin = 32;

void setup() {
  Serial.begin(115200);
  pinMode(relay1pin, OUTPUT);
}

void loop() {
  Serial.printf("Switch on\r\n");
  digitalWrite(relay1pin, HIGH);
  delay(10000);
  Serial.printf("Switch off\r\n");
  digitalWrite(relay1pin, LOW);
  delay(10000);
}

Add two lines to your platfromio.ini file. You will find the correct COM port in terminal when uploading the new code to your device.

In my case it is COM6 and platformio.ini looks like this:

[env:esp32-evb]
platform = espressif32
board = esp32-evb
framework = arduino
monitor_port = COM6
monitor_speed = 115200

Upload your code again after changing the ini file. If it does not print your output automatically to your terminal, start the monitor manually:

From now on, you should see the printf output in your terminal every time you upload your code again:

OK, that’s all. There are many tutorials out there, how to setup a webserver, controlling pins, using SPI or whatever. But now, you have some great arduino IDE without arduino IDE 😛

Have fun coding!

Doxygen – Tips and Tricks

LaTeX non-interactive

To make LaTeX skip some errors without user interaction, you can add the option --interaction=nonstopmode to the pdflatex call. Easiest way to do so, is changing the LATEX_COMMAND_NAME in your Doxyfile.

LATEX_CMD_NAME = „latex –interaction=nonstopmode“

Do not forget the double quotation marks. Otherwise doxygen will remove the space and the command in your make.bat will fail.

If you now want to generate the code, step into your doxygen-generated latex folder (designated by LATEX_OUTPUT option in Doxyfile) and execute make.bat (on Windows) or make all (on *nix).

Adding a favicon to html output

Adding a favicon to html output, you need to specify it in a custom header and include the original image in HTML, as described here. To extract the default header file:

doxygen -w html headerFile

Add the follwing line to in headerFile within the html header

<link rel="shortcut icon" href="favicon.png" type="image/png">

And add your headerFile and the image to the HTML_EXTRA_FILES in your Doxyfile. Its path is relative to your Doxyfile.

HTML_HEADER = headerFile
HTML_EXTRA_FILES = some_rel_path/favicon.png

Now you can generate your html documentation with some favicon in place.

PDF output destination

Did you ever search for the PDF file, doxygen (or better the Makefile in latex) generates? I just added an option to doxygen, copying the refman.pdf to a location of your choice. (Hopefully it soon get’s merged and released).

If you want to test it out? Compile doxygen from my doxygen fork and add the following option to the Doxyfile of your project.

PDF_DST_FILE = ../MyGenerated.pdf

The destination is relative to your Makefile in your doxygen latex folder. As soon as make finished it’s job, the PDF is just in the same folder, the latex folder resides in.

That’s all. Enjoy generating software documentation with doxygen

Cookie Banner von Real Cookie Banner