2015-12-22 - cross-compiling alternatives

when doing cross-compilation, typical approach is to have separate toolchain that runs on your native (host) platform (x86 most likely) and outputs binaries for another (target) platform (avr/arm/mips – you name it). this model works great for small µC, where typically there is not too much code aside from your actual projects. when target platform is “something bigger”, has an operating system, etc. proper sysroot with all the libs for linking and header for compilation is needed as well… and so we have…

… and this this is where it gets bumpy! you can easily make a mistake and accidentally use include files from your host platform, and just link with target libraries, thus breaking ODR and causing non trivial issues. even is you make it right, it still it usually requires hell a lot of work to prepare all dependent libraries cross-compiled for your target.

using customized builds for your platform, like the one shipped with OpenWRT, may become handy here. yocto can become very handy as well. yet still it is far from zero-effort. this is all a must-have for any medium/bigger project. however when doing small project, it is not unlikely to take more time than the actual development.

what can we do with it?

as usually when there is a difficult problem, you can try avoiding it in a first place. if you happen to develop for a platform with an operating system and enough space on flash/ram, you can actually install native toolchain on the device itself, and then just compile final code there! yes – this will most likely take significantly longer than n your local PC, yet if you'll do it rarely (most of the development/automated-testing should be done on PC any way), this can actually be a huge time saver!

i've did few project on using RasPi with this model. most of the time it works like a charm!

i did however once bumped into a piece of code, that required more RAM for compilation, than my RapsPi had. whet then? i'm lazy, so i just bought newer RasPi with more RAM. :P another option would be to add a sshfs-mounted swap on RasPi, that would be a ramdisk on PC. this would be even slower, but again - if done just a few times, might give a better net time result.

there are quite a lot of cases where this approach will not suffice. if your code base is big/complex enough, it might still be an issue to do compilation on device, due to time/space limits. what then?

then docker kicks in! “wait – what?” you might ask? well… it's an interesting subject. :) first you need an image of a working system. target system! ok - but how to run it on a PC? let's see an example how to do this.

first we need an image. to make things simple i'll just use of the shelf docker image for RasPi – resin/rpi-raspbian. and so:

docker pull resin/rpi-raspbian

ok – now how to run it? with qemu! :)

mkdir /tmp/test
cd $_
cp /usr/bin/qemu-arm-static .
cat > Dockerfile <<EOF
FROM resin/rpi-raspbian
COPY qemu-arm-static /usr/bin/qemu-arm-static
EOF
docker build -t test-raspi .
docker run -it --rm test-raspi
# voila - you now have shell on "target" system! :)

how does this work? thanks to kernel feature foreign binaries can be run, using proper emulator. quemu does exactly that. that's it! you now have a working environment, where you can compile for target platform, while being on more powerful PC.

all of this happens at a cost of emulation… but with a nice feature for free – you are now preparing docker image for your project. if you organize it like this:

1. raw platform
2. based on “1” add all runtime dependencies (libs, tools, etc…).
3. based on “2” add all development tools you like.
4. based on “3” create an image with qemu, that you can just run from your PC.
5. based on “2”, with addition of binaries compiled on “4”, you have a target-runnable image!

using “4” you can develop locally. using “3” you can develop on target, if still needed. using “2” you have a base, where you can add precompiled binaries, this making a runnable image of the final system! now you have an easy deployment for free! after pushing images to a docker registry, it is enough to run:

docker run -it --rm my/package

on the device and you're done! it's up and running! :)

of course solution is not perfect. you still need to run inside an emulator. i'll not get into performance details here. as a quick test one can run a simple test app, using cpp-netlib. the lib is nice, but it takes A LOT OF TIME to compile when using it.

results? on my PC it tool little under 12s to build, while inside the emulated docker container build lasted for a little over 3 minutes… ;) compiling simple hello-world takes 180ms on a PC and 2600ms inside the emulated environment. so the factor is ~15x. a lot…

on RasPi2 the same programs' compilation lasted for little over 140s and 1700ms consecutively. this means that RasPi was roughly 30-40% faster than the emulated environment.

the good news is that inside qemu you have all the resources of your PC – all the cores, RAM and disk space (ssd?). it is also more convenient as actual hardware is needed to work.

as usually – no golden hammers. different tools will be better for different tasks. you have been presented with 2 alternatives to classical cross-compilation. choose the best one, suiting your needs and have fun developing! :)