Compiling Software on RAM-limited Multi-Core Systems

Compiling Software on RAM-limited Multi-Core Systems

Since I often compile stuff on embedded ARM targets that are well equipped with processing power (Exinos Octa-Core), but are neglected regarding RAM (2G), I often facing the trade-off between running multiple or only a single/few compilation jobs (make -j8 vs. make -j1). If you start too many jobs and if you have large compilation units (e.g. with the Ceph Project sources), the system will feel like jam, as soon as it begins swapping.

I feel, that deciding the job count at the very beginning is (was) a trade-off, I was not willing to accept. Therefore, I decided to write a little script to suspend compile processes, that cross a certain memory limit. This way, the suspended processes get moved to swap and the still runnig processes get a comfortable amount of RAM. This way, the kernel is not forced to move pages around with every scheduling round. Instead, it will move it once on swap for suspended processes, when it needs RAM for the running ones and as soon as the processes with large memory footprint finish, the supended ones get back to live.

Welp, I decided to base the priority on the time the processes ate up user space processing time, so the older ones (often the most memory hungy ones) get processed first. This scheduling scheme proofed to be a optimal descision, that is also not hard to implement as a bash script.

Here you can find the little script, that needs to be run within a loop or simply with the watch-tool (maybe with sudo).

watch scripts/linux/bash/schedule_compile.sh

Happy compiling!

OrangePi 4G-IoT Android 8.1 SDK

Same as with the 2G-IOT, OrangePi also provides a quite inconvenient way (through mega.nz) to get the Android SDKs for the 4G-IOT… Here are the torrents (tar.gz and tar.xz have the same content but xz is much smaller)

Have fun…

OrangePi 2G-IOT Android 4.4 SDK

Getting the Android SDK

Since I just struggled getting the Android SDK for my little OrangePi 2G-IOT, I felt responsible to share it using bittorrent. The main problems have been, to download all that stuff from mega.nz (it raises limits for users not paying a monthly fee) and another one was, to concat all 7 files to a working tar.gz. I think, the wildcard they use in the OrangePi’s user manual does not expand the filesnames of the parts in the right order. But then, wired errors occur.

If you encounter problems with not being able to create symbolic links when unpacking, just use a real file system as the base for your unpacked archive (e.g. ext4, btrfs, …, but NOT NTFS or FAT).

Here you can find the torrent: 

 https://downloads.the78mole.de/OrangePi_2G-IOT.tar.gz.torrent

Getting the toolchain

Another dangling point of OrangePi’s SDK is the lack of a toolchain. They have a large download, but it does not contain a toolchain. So just download the correct one from linaro (take the appropriate one for your system, mine is x86_64), unpack it inside the folder where the empty folder toolchain is (here it’s ~/somewhere/OrangePi in the example) located, remove this toolchain folder (rmdir should succeed with a fresh unpack, when it is already a link, use rm) and set a symbolic link to the fresh linaro one.

$ cd ~/somewhere/OrangePi/
$ wget https://releases.linaro.org/components/toolchain/binaries/latest-5/arm-linux-gnueabi/gcc-linaro-5.5.0-2017.10-x86_64_arm-linux-gnueabi.tar.xz
$ rmdir toolchain
$ ln -s
gcc-linaro-5.5.0-2017.10-x86_64_arm-linux-gnueabi toolchain

Running the kernel build

Just stick to your User’s guide again and do:

$ ./build.sh

Google ChromeCast’s (Smartphone) Independance Day

Have you ever been upset about your chromecast heavy depency on a smarthphone. So, nor do I, but my wife complained, since she always drops her phone somewhere else around the house… What could you do about it? Chromecast hardly has other input options and writing against it’s API is not an easy task.

I’d not be an embedded electronics engineer, if I would not have a solution to that… My house currently runs an EQ-3 (former part of ELV, now two somehow independent companies) homematic smart home and around 80 nodes connected to it, anything from simple wireless switches to RGBW-Led-Stripes. If you put an Odroid HC-x beside with iobroker running on it, connecting the world is not enough :-). But EQ-3 also offers nice modules for makers like the HM-MOD-EM-8 for just a few bucks. Equipped with that module and some Bread-Board, there is nothing as easy as building a 8-Button HMI. Since it is not a hard task (only a few wires and some buttons), I decided to give ALLPCB a try, sketched a shematic and a PCB within KiCAD (for my first time) and ordered it from ALLPCB. Since I plan to distribute some more of these gadgets around my house, I ordered 30 for just $1.50/piece.

Here is the result:

Now, just install the iobroker.scenes and iobroker.chromecast plugin and feel free to connect your buttons to scenes. Your Chomecast is now free like Willi 😉

And here are the KiCad-Design files:

Atlassian And New SSL Certificates

Or How CAs Drain Your Lifetime

When you check my activity, you simply can see, that the past weeks my time for writing was quite limited. Since I had a few very urgent projects, I had no time to care about my blog. But when it can get no worse, your website provider, that previously relied on the Symantec CA, decides to switch it’s root CA to some better and quite new CA that 1. is not listed in some JRA distributions‘ keystore and 2. uses trust chains with intermediate certificates. And this leads to an ugly situation when you run e.g. some Atlassian tool environment where the tools in turn use SSL to connect to each other. I would have not realized the problem so early, nor it would not have been so urgent, when I would not have decided to do all authentication (Bitbucket and Confluence) through JIRA’s Crowd-API.

But let’s start from the beginning… My provider, where I get my SSL certificate from (namely the german 1&1) used the Symantec CA for years. This CA somehow attracted Google’s anger, so Google decided to remove Symantec from the trusted CAs list of it’s chrome browser and announced this around beginning of 2018. 1&1 did not find a reason to get in hurry, so they kept their CA until mid of 2018. Then they started to remind their customers to update their SSL certificates, forcing their customers to hurry a lot. I realized, that there is some neccessity to follow their appeal, but I also felt to still have some time and do it, when the certificate expires…

Then the time came and my Chrome browser refused to show my Atlassian pages. So I logged in to my 1&1 Account and my Linux machine where JIRA & Co. runs, ordered a new SSL certificate, copy-pasted it into my Nginx configururation (which I use as an SSL proxy) and everything went fine at the first glance. I could log in to JIRA without a hassle, did not see any JIRA warnings or hovers and my browser also was not shouting at me about the SSL connection. Everything was fine. But then came the surprise… I tried to log into Confluence (no I currently have no SSO 🙁 ). I let the browser enter my credentials and… got refused. I tried some more times manually with different combinations of users and passwords, checked my password store, checked CAPS LOCK,… but I did not get in. Since I disabled the „local admin“ of Confluence (due to being tight on the 10 user limit), I could also not check from inside Confluence.

What happened? After digging (this is what I often do 😉 ) through the settings, I stumbled over the Application Links section, which stated, that the connection could not be established due to SSL errors. Ah, OK, nice that Atlassian is recommending to install the JIRA SSL Add-on that helps with all that suff. Really all? For sure, not! Especially not with problems I encounter 🙁 . After digging deeper and deeper, I found, that the new chain of trust, 1&1 uses, is not setup in the JIRA keystore, the atlassian tools ship with their JRE. In contrast Chrome’s certificate store is updated tightly.

How to fix? First you need to find the Java keystore to add the CA certificate to be trusted. This is quite a bit difficult, if you do not know about the tool’s shipped JRE. So don’t hassle around with your distributions keystore, step into the folder where your tools are install (/opt/atlassian/ in my case) and do a

find . -type f -name cacerts

In my case, the list shows as follows:

./jira/jre/lib/security/cacerts
./confluence/jre/lib/security/cacerts
./bitbucket/5.2.2/jre/lib/security/cacerts

I know, Bitbucket 5.2.2 is quite old (and the other tools, too), but keeping it up-to-date is quite hard for a spare-time setup. I believe, with an up-to-date installation, the problem would not have encountered. Let’s see, when I update… With this knowledge, it is quite easy (and some manual work) to add your cert to the keystore.

To get the certificate from your server, just do a

bash$ openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ${HOME}/www.example.com.crt

Then you can import that certificate into your Java keystore with the following command set (e.g. for confluence):

bash$ cd /opt/atlassian/jira/jre
bash$ bin/keytool -delete -alias www.example.com \
-keystore lib/securety/cacerts -storepass changeit
bash$ bin/keytool -import -alias www.example.com \
-keystore lib/securety/cacerts -storepass changeit \
-noprompt -file ${HOME}/www.example.com.crt

After doing that (I didn’t even need a restart of the tools), the application connections resurrect and you can log in to your confluence again.

The more convenient way is to write a little bash script that does the job. You can find mine here on Github. Feel free to improve it and issue some pull request if you think it’s worth to be shared.

Lessons learned

  1. Alsways keep a local admin account active in each and every Atlassian tool 😉
  2. Better use an automated SSL framework like Let’s encrypt. With it, you need to make the key rolling-update working from the beginning, not when it is too late (OK, this would not have helped me in my situation, nevertheless it is a good idea to do so)
  3. Document your problem solutions (which I do with scripts and this blog 🙂 )
  4. Don’t document the tool in the tool (e.g. this howto in Confluence), you will shoot both your feet 🙂
  5. Keep your tools up to date!

Windows (was) just a pain

Since I use Linux at home and love to develop embedded, backend and (web)-fronteds within a real operating system, I sometimes get crazy at work, when I just search for an alternative to a simple command line… So, what’s the alternative in Windows? The Command Prompt, then PowerShell or some specialized, magic, woodoo,… GUI application with the worst design ever seen in the universe and beyond?

OK, I see, you need an example 😉 Here it is one of my favorite: Syncronize a 100-GB-folder from one machine to another when one is at the end of the world, connected by avian carriers (see also RFC 2549 – IP over avian carriers) with a perceived rate of 2 bit per hour. 22 years ago, rsync was invented and serves every (unidirectional or pseude-bidirectional) syncronization desire with a sheer infinite amount of options… But, it is not available (directly) for windows…

Welp, some time ago, there was cygwin, which was driven by Red Hat. OK, it is still being developed, but somehow, I feel it is not serving my desires very well. At least not the desires a developer has. I also found MinGW and MSYS some years ago, but as I ran into trouble with wget and rsync when handling large files, I tested MSYS2 (it already includes MinGW-32 or MinGW-64, whichever you prefer). That was the starting point to test MSYS2 and to my surprise, it is exactly what makes my heart pound faster. All previous POSIX/Linux/Universe/Multiverse/… compatibility layers for Windows had some GUI to select packages, run updates,… But not MSYS2! It uses a pacman port. OK, I didn’t use pacman before when I was not forced to (I prefer APT), but it is COMMAND LINE and it runs on Windows (64-bit).

And even better, it did the ssh configuration well (a problem in old MSYS), so that you could generate and use an SSH key, which makes rsync even more powerful…

Hey guys of MSYS2, whenever you pass by in Germany (Erlangen), I will spend you some beer at Steinbach Bräu. This brewery is just to beer, what you are to Windows. Simply the greatest enrichtment 😉

TL;DR

So, if you would like to use rsync and wget in windows, just install MSYS2, do the obligatory update described on their page and execute the following:

pacman -S wget openssh rsync

Have fun with Bash, SSH, Rsync and all that other cool Linux tools on Windows!

Getting Started Embedded – Part II – The Embedded Project

Start embedded projects in an ordered way using the right tools

For many people, „project“ is a mysterious word and everybody understands something different about it. Sales department has a totally different understanding than engineers. Engineers see it quite similar to software developers, but there are facets their opinion differs. Just take a minute to think about what a project means to you.

So, after you got your own opinion, let’s first define, what a project means to me. At first it means nothing more than a vision, e.g. a customer’s product idea, to talk about. And with every vision, everybody has an individual understanding of it in the beginning.

The Information Collection Project

Welp, what is that? It has at first nothing to do with software, hardware or an embedded project. We start to collect information. Since we are living in the 21st century, we try to avoid using paper and put everything into „the cloud“. One way to organize such information is putting everything into a MS Word document… OK, this was a joke 🙂

I tend to use „the Atlassian tools“ (Jira, Confluence, Bitbucket) for that purpose. Not that I receive any revenue for that, but I did not find anything that can compete with it at the moment… Most important in my opinion at this stage is Confluence. But to start a „Space“ in Confluence, the more effective workflow is to start with a Jira project (again this word…).

If you create a Jira project, I suggest you use a template that contains user stories and bugs (e.g. Scrum or Kanban), and then add issue types (Tutorial) for requirements and impediments. Don’t try to optimize the workflow for every issue type, during the project is the perfect time to do it.

Now create a new Confluence space (Software Project Space template) and link it to your just created Jira project. This space is the container for all relevant project information. Neither is Jira only suited for software projects, nor is Confluence (with this software template). In fact, you can drive anything with it efficiently. I love it to steer my renovation projects around my house with it. (Yes, I’m a nerd 🙂 !) This is not classical project management with a tight schedule, it is simply collecting issues, linking them and pick them according to their priority and dependencies.

For those old stagers around us and for those new to agile principles or just interested people, I would recommend to have a look into Design Thinking. It gives you some ideas on how to collect the relevant information, define stakeholders,…

The Vision

One of the most important things, before we can talk about „the project“, we need to get a common understanding, what the inventor of that vision had in mind. If, and only if, the vision can be realized with hard- and software, we have an embedded project starting and many developers already have an idea, how it can be „solved“ as a collection of electronic parts and code. But this is the „solve a problem“ approach (engineers like that) to start a project, but very often not leading to customer’s satisfaction. We will start differently…

To get a common understanding, it is necessary to talk about the system’s requirements, making the vision more clear to all involved parties. At this point, it is the right time, to list possible stakeholders and the users of the potential product. Don’t forget to create meeting notes and other pages about your findings in Confluence and put every requirement into your Jira project. In Confluence you can create a nice page with a Jira-Table-Macro collecting requirements and also grouping them with filters for different stakeholders. You can use tags for that purpose or, even better, create Jira Service Desk accounts for your stakeholders (saving the costly developer accounts). For sure that requires the Jira Service Desk add-on (the smalles 3-Agent-Version should serve very well even for projects).

During this time of user and stakeholder analysis, the vision becomes more clear and a common understanding settles. Still it is not the time to carve the solution in stone, like many engineers/developers tend to (again, this is what they love). The vision will still evolve during „the project“ and also system requirements will do. Now it’s time for personas and use cases, to get an even better common understanding.

Use Cases

When we talk about use cases, we often only talk about users or customers of the end product. But you should never forget the other stakeholders. Even the development team can be a stakeholder, because they (in an agile environment) should be the responsible for quality and they for sure have specific requrements regarding QA (Quality Assurance, e.g. testing). BTW: Don’t forget a UART port in your hardware requirements if you go with headless embedded system. Otherwise the developers will kill you instantly when they move from eval board to prototype hardware.

If you have some uses cases collected, hopefully a few for every stakeholder, it’s time to move on. Don’t try to finalize the list, you will fail! It is much better to keep in mind, that the list will still evolve and steer your project to be agile. Try to prioritize them together with all stakehoders. Invite some avatars, if stakeholders can not be included (e.g. real end customers). Here personas come get handy.

The Project’s Disciplines

The list of requirement typed issues in Jira should be already extend across some pages and you identified the ones with most impact on your system design. Now the paths will split, depending on the resulting product. If it has some GUI, like many HMI projects tend to have nowadays, you should get some experienced Design Thinking engineer that possibly starts with paper prototypes or even more wired stuff. But this is not the focus of this post…

We know should define a few components. At first HW, SW and possible ME (Mechanical Engineering) will serve well. We can put more granularity in later on. Now try to assign your requirements to these components. If you did this, let the engineers of these disciplines have a look on every requirement that belongs to their component and let them comment on their understanding. Pull in the stakeholders (or their avatars) again to dicuss the requirements, should something be unclear.

Thougts about the Software Project

Now it is time to start the software project, isn’t it? Well, if you can tell, what a software project is! No, I don’t talk about the „Create project…“-button of your IDE. First we need to step back quite some distance to get a better overview on it.

We already defined some requirements above and assigned them components. The ones tagged with the SW-Component are the base for our user stories. Maybe we already defined some of them when we talked with the stakeholders, but now it’s time to check, if we have important requirements left, that have not been addresses by a story. But collecting these stories is another topic, we will not address in this post…

So, why I’m talking about that…? Because there should be some requirements and stories hanging around, that could conflict with the tools (and IDEs) we tend to use. If you are using e.g. Keil µVision and your have the requirement to do intensive testing (e.g. because there is some SIL-level required), then you should definitely rethink your decision using this IDE. Or if a customer requires a close feedback loop and high delivery rates (daily in extreme situations), then you need some continuous integration pipeline, which is hard to establich using GUI-driven IDEs.

Let me give an example. You have a project with many parties involved (maybe different companies in extreme). Think of a smart home system. Different companies design the sensor nodes, another the central (headless) unit, you design the HMI-unit and another company designs the GUI using a propritary tool for GUI design, sending you some pieces of API code and some static libraries to be linked in. You get a delivery every week and need to integrate it. The customer wants to test the new functionality of the GUI’s HMI (e.g. this week switch on light) as soon as possible. Many of the new functionalities and bug fixes from the GUI do not need your intervention, because they do not affect the API between embedded system and GUI. But not having an automated CI pipeline requires you to integrate the new GUI in your IDE and test it with every week’s delivery MANUALLY! I would hate your job from the bottom of my heart!

But you are an engineer and you love to solve problems immediately. Don’t make a project out of it, just raise an impediment (use this issue type in Jira) and find people to help solving it (Scrum-Master for communications and mental support, your boss for budget, IT employee for Jenkins,…).

The essence of this example is: If there are large parts of the development that could be automated using other tools (ot toolchains), start today changing your tools. Engineering should not consist of repetitive tasks, it should challenge your creativity with new tasks every week, every day,… every second. Switch over to CI and TDD, there is nothing giving you more satisfaction in your daily job as a software engineer. When mentioning TDD, already a few years old, but still very valueable is James W. Grenning’s book Test-Driven Development for Embedded C.

So before you start, choose the right toolchain. I highly recommend using GCC together with Eclipse and/or Makefiles. Why? Because it is platform independend, can be automated (even headless) and does not require a license. The licensing problem is not about the price, it is about the problem that you need one with anything or anybody involved in the development process. Even your Jenkins host needs a license, when you use commercial tools. And requiring a license makes it nearly impossible to spin up a fresh docker container for running a build. So, don’t shut the door on amazing technologies you don’t know yet but could make your life easier, like the wheel did six thousand years ago.

What’s next…?

In this post, I talked about starting an embedded project. Next post will be about starting the (non-GUI) embedded software of it.

Getting Started Embedded – Part I – The Toolchain

Introduction

For getting started with any embedded development, the most important piece is the toolchain.

Many people suggest to use Keil, IAR, or some other fancy, professional, rocket-sience (and very expensive) IDE. In my opinion, that is just rubbish.

In former days, when tiny embedded controllers just have not been designed with compilers in mind, it was good to have some highly optimized compilers that could transfer a piece of C code to the ASM of these devices.

Nowadays, processors are designed with compilers in mind. Therefore, I would highly recommend to use GCC not only because of its low price tag (0 $), but because of its stunning community and active development. Since the community and many companies still putting so much effort into this piece of software, no single company can compete with an own closed source product.

Installing the ARM toolchain

There are many possible sources, where you can get GCC and all the tools you need to start. You can compile ARM-GCC yourself using your platform GCC, download the one from ARM directly, take a release from GNU MCU Eclipse and many many more…

Don’t know what to do? Just get kickstarted, and give XPM (a node/npm module) a try. Download and install node.js. The version does not matter too much. If you are not developing with node.js itself, better stick to the stable version.

When node.js is installed, install xpm and the ARM GCC toolchain (same for Windows, Linux & macOS):

me@diggerVM:~$ npm install xpm
me@diggerVM:~$ xpm install --global @gnu-mcu-eclipse/arm-none-eabi-gcc
me@diggerVM:~$ arm-none-eabi-gcc -v

OK, that was easy… If this worked, you maybe need some supporting tools. Best practice differs a bit, depending on your platform. For Linux, just install the build-essentials package (Debian dn ubuntu call it like this).

me@diggerVM:~$ sudo apt-get install build-essentials

For Windows (would also work for Linux, but I prefer the OS provided package), you can use xpm again:

C:\Users\me\>xpm install --global @gnu-mcu-eclipse/windows-build-tools

After this has finished, you possibly need to add the build tools to your PATH environment. You will find it in %APPDATA%\xPacks\@gnu-mcu-eclipse\windows-build-tools\2.11.1-1\.content\bin.

Testing your Toolchain

To test your setup, just clone, download,… the STM32 example project. Open a command line prompt, cd to the project directory and fire make:

me@diggerVM:~/GIT/stm32-example$ make

If this worked without errors, you have your toolchain up and running. Congratulations!

What’s next…

The next post will explain, how to setup STM32CubeMX and the Eclipse IDE to start developing own embedded applications effectively. Until now, there is not much difference to commercial IDEs and Toolchains from a workflow point of view. But don’t worry, we still have automation in mind and the goal is to have a CI-Pipeline running soon.

Just digged up

Welcome to MolesBlog.

In my first post, I want to introduce to you, who am I and why I started this blog.

I’m a professional elelctronics engineer, never getting enough from hardware, software, IT, … everything that has to do with creativity and technology is a good candidate to attract me 😉

I started my experience around 30 years ago, when I was a child. 1987 I got a Commodore C64 and a few electronics construction kits from my uncle, including a soldering iron. This made me locking myself in the garage for weeks. I soon blew away the CIA of the so called User-Port (max output current is 2mA) of the C64 with my first clavilux LED (drawing 20mA) experiments and soon, beside going to school, I spent most of my time in my garage, sometimes together with friends having the same interrests. This was the starting point of my career, leading to a diploma degree in electronics engineering with a great emphasis on VHDL, software in general and open source in particular.

During the last years, I realised that engineering has a lot of best practices, but hardly anybody is doing it with heart and soul. Many projects, independent from the company’s industry, either rule their developers with strict processes or give them plenty of rope. Neither plenty rope, nor strictness leads to high quality software. This I encountered many times with brownfield projects I need to take as a base for my developments.

But even when I got a greenfield project (not yet happened in my business career, only with private projects), it is hard to find some guideline, how quality assurance works best and makes fun. There are too many pitfalls and time wasting hurdles to not being forced to take some shortcut, always guiding you to a messy product, lacking real software quality. This always leads to dissatisfaction. Sometimes you just skip the testing, sometimes the toolchain effectively resist to play well or at all in a continuous integration environment. This mostly applies to very expensive commercial toolchains in my experience.

The most important question seems: Who else should back the quality of software, hardware and embedded systems, if not professional developers, engineers or technology addicted people like me and possibly you? Your boss, your project manager? Most probably some (so called) quality engineer, if he/she has time or exists at all. Regarding to SCRUM (which does not define a Project Manager and a quality engineer), the responsibility for quality belongs to the development team, not anybody else. This means, YOU, the developer is totally responsible and always needs to fight against commercial demands from the product management.

Another question simply is: Does it give you satisfaction to write bad code, hunt silly bugs or getting an „enema“ from your boss because of bad quality?

Welp, would you like to be satisfied about your work? Could quality lead to this satisfaction?

From my experience in various projects, a good starting point to get satisfaction (and quality) are the following rules:

  • Try to be the best in your class – You are a professional, then be one. If you order a mason, you expect he uses a mason’s level and delivers extraordinary quality. With software customers usually don’t see „your work“, but your colleauges and yourself do. Apart from this, it is simply awkward if customers experience a lot of bugs.
  • Keep yourself up-to-date – Read blogs, magazines, books and watch videos about your profession. Especially with technology, you can not expect the world keeps as it is until you retire.
  • Review your work and yourself – One of the most important things is review and retrospection. Look back on your work and yourself and always ask, what could have been done better.
  • Keep human 🙂 – Take advantage of not working on your own. Your teammates are also professionals and have deeper knowledge in some parts than you. Trust them, talk to them and work together to get the most out of your project and life.

These are only a few thoughs to prepare the base for being a responsible and accountable development team. BTW, SCRUM already introduces some of these concepts as a part of it’s process.

I’m happy to receive responses about my statements above and I’m also interested in topic suggestions for future posts.

In my following posts, I plan to write about the STM32 test driven development (TDD) process, I currently try to establish, using unit tests and a CI to ensure a high quality and about how it is to (try to) be a Clean Code Developer (CCD).

Best Regards,
Daniel /themole