Felix' Ramblings Felix' RamblingsFor my Linux desktop, I used to use Waybar to display my current desktop, the time & date, as well as some minimal CPU/RAM metrics. I got it all looking like I wanted to, while being a bit shocked that customization is done via CSS. Then one day, I updated my system, and I was greeted with the bar filled with "■■■■■" symbols.
What happened? Well, I was using a bitmap font in the bar. And apparently, at some point, a library used for font rendering decided to drop support for bitmap fonts. The official workaround was either to
This annoyed me enough to write my own little waybar application (pain), including writing just enough of a parser read and draw the pcf-file of the bitmap font. This worked (apart from various small bugs in my wayland code).
So after I discovered the joy of creating small tools for my own system, I continued on in creating a few (very basic) tools for my own use, even if they are just shitty clones of existing software. Great learning opportunity, and I only have myself to blame for any issues rising up.
I write the software, and I stop working on it once it's good enough for my needs. When I move machines, ideally I can just yoink the binary and be good to go, but alas I learned about (not-)portable Linux binaries the hard way:
Okay, at least when I'm talking about my own software, I have the source code available, so I guess I just stick to re-compiling it for the machine. That's certainly better, but it sure as fuck does not save you from complications along the way.
Libraries, if managed by the system, are constantly updated, leading to interfaces
becoming deprecated/obsolete/removed, breaking your compilation.
On Void Linux, I had this happen with them moving (back) from LibreSSL to OpenSSL.
The toolchain itself, meaning your compiler, also isn't safe from this.
I compile with -Wall -Wextra -Werror, a compiler update rolls around,
and suddenly the old code refuses to compile because there are new warnings
preventing compilation.
This can also quickly happen if you use different machines for development,
e.g. if you use a slowly updating distro (for example Debian) with a
cutting edge one (for example Arch).
The more third-party libraries / systems you use, the more complicated this whole process becomes. I had python projects fail on me because my system used a newer python version, causing issues with library A, and trying to resolve said issue by updating the libraries caused further incompatibilities along the way.
Using the C++ package manager Conan, I had the most cryptic linking errors because a compiler update from Visual Studio caused a third-party-library to compile slightly differently, resulting in linking errors because some of the third-party-libraries were used via (incompatible) pre-compiled blobs. But because this is not funny enough, the package manager also comes with its own per-system cache so you don't have to recompile all packages every single time, meaning depending on which system you are own and depending on which packages you have compiled at certain points in time with certain toolchain versions, you get different linking errors.
All that to say: There have been numerous discussions ([1] | [2] | [3]) talking about whether binaries and/or third-party-libraries belong in version control, and this is something I want to persue. I want to be able to checkout old version, use old binaries without issue, and also create old binaries (in other words: use the old development environment) without issues.
I think Windows, for all of its flaws, gets this part right.
The binary compatibility of Windows is nothing short of insane (in a good sense). At work, I develop in projects which are older than myself. In some parts equally fascinating as horrifying, not only does the source code use some old ass Windows API calls, but there's also really fucking old binaries laying around. Libraries we link against, but also executables generating data, where trying to create a new binary is a whole adventure of its own as we only have a handful of source files, including some "Turbo C" project files - but the executable still works!
So for the system API, AFAIK it's well documented when a function was introduced. You link against the system libraries statically, and you're good to go. Outside of the system API, statically linking third-party dependencies keeps the amount of files to deploy at a minimum, while also ensuring pretty good portability. The only downside being that compiling third-party-libraries, in contrast to e.g. Linux, can be a bit of a pain, but other than that, the deployment itself works pretty well. But I don't want Windows on my main machines; using Windows at work is enough pain as is.
The VPS I rent (which is hosting this website) is running OpenBSD, and they use the complete opposite approach. As stated in their FAQ:
So from what I can tell, they do not guarantee binary ABI stability; you are expected to maintain software for each major release (which are published roughly twice a year). I can respect the philosophy / code idea, but this is does not align well with my goals/preferences, and the official download locations only offer the latest few OpenBSD versions. You can probably get older versions from some mirror, but at that point you are strictly locked into that older version, including the old package repository (if the mirrors even offer them, I honestly haven't checked).
During some of my research, I came across two very good articles in regards to binary compatibility:
Generally, the strategy seems to be:
Especially the JangaFX article goes into great detail why you cannot link glibc statically, which I found hella confusing when I found out about that. I mean for windows, it's one flag to statically link the C runtime into your application. Even though both articles explicitly cover that the glibc-alternative musl is not a suitable drop-in replacement, I tinkered around with it a bit.
To make this clear:
dlopen" does not work
either. The source code is simple, take a look,
it's a dead-end.Still, could be interesting enough to use for simple tools, background services, and so on. How can I get my hands on such a compiler? The, by far, easiest solution is to use musl-cross-make, as stated by the musl website. The default configuration uses a pretty old GCC version, but you can simply instruct musl-cross-make to use a newer version like:
# I just looked in ./hashes/ for the most recent available versions
> make TARGET=x86_64-linux-musl BINUTILS_VER=2.44 GCC_VER=15.1.0 install
If you want to do these things yourself - I'm honestly not sure how exactly this all works.
For reasons I have yet to understand in its entirety, it's not
as straight forward as "just build GCC with musl".
The explanations I found on the internet involved
compiling GCC twice(?), by first compiling binutils, musl, then a
"minimal" GCC, and then the "final" musl GCC.
At the same time, musl-cross-make explicitly lists
"Single-stage GCC build" as a feature, so I don't even know.
Afterwards, you can theoretically build and use C & C++ applications, or start compiling and using third-party libraries as well. Depending on the project, for example OpenSSL, I would classify this as non-trivial as googling "OpenSSL use musl" yields >8 year github repositories with a bunch of instructions. Given that, as stated above, musl doesn't cover the usecases I require, I decided to stop exploring this path for the time being.
I tried to pin down what exactly I want from my developer environment. After trying a bunch of different tools, it became painfully clear that my vision of an "ideal" environment clearly differs from most people, as I wasn't able to find a tool which "just works". Here's the list of stuff I want:
Having this all typed out, this is quite a list, and I certainly don't expect any tool to just magically solve this via two commands, but one can dream. The windows world of "just unzip this file and run the software" has spoiled me, and if this can be somehow combined with the awesomeness of package managers, I can live happily ever after.
Spoiler: I haven't found this magical environment yet. Heck, I still have "setup a linux vm" on my TODO-list to actually test the deployment of compiled binaries.
But nonetheless, I tried out a bunch of tools and leave notes for future me of what I have tried so far. I will go through the (very opinionated) list of tools in the order I tried them out on.
The JangaFX article I mentioned above also briefly describe their deployment strategy: They use debootstrap, a script to create a minimal Debian chroot. They install their toolchain within that environment, and by compiling their project in there, the system libraries they link against are not cutting edge, so there shouldn't e.g. be any weird glibc conflicts.
This is, most likely, the easiest way of creating portable binaries. The dependencies you compile against are reasonably old, and I think that within the chroot, you can just install third-party libraries via the package manager. So for the final "create the binaries which I can send to someone else", this seems like a good solution.
I haven't tested this myself because I wanted to avoid a container like that, if possible. From my understanding, you would do most of the development outside of said container, but for my goals, this seems like I would need to move my entire development setup into this chroot. Old depedencies/versions also means that the development tools themselves are older as well. This feels like a convenient version of having a dedicated old development machine, which is nicely separated from the rest of the system without much hassle, but does not seem suitable to move the entire development into.
To start off simple, I looked for "basically musl-cross-make but for glibc", and crosstool-NG seems to fill that gap. Honestly, this is the one tool which "just worked" and was pretty nice to use. You can get a self-contained, modern GCC in a single output directory by just downloading the tool, doing some minimal configaration in the TUI and then kicking off the build process.
Here are my notes for setting this up, it's reasonably short:
# Determine versions I want to use / target
# https://en.wikipedia.org/wiki/Linux_kernel_version_history
# https://en.wikipedia.org/wiki/Glibc
# Download tool
> git clone https://github.com/crosstool-ng/crosstool-ng
> cd ./crosstool-ng/
# Setup
> ./bootstrap
> ./configure --enable-local
> make
> ./ct-ng x86_64-unknown-linux-gnu
# Config with interactive menu
> ./ct-ng menuconfig
# To configure:
# Paths: Set "Prefix directory" to my target output directory
# Toolchain: activate static toolchain binaries
# Operating system: Linux Version
# C library: glibc version
# Compiler option: GCC version
# Save & exit
# Actually create toolchain (in output directory)
> ./ct-ng buildSupposedly, you can re-use the options from a previous build if you need to re-create the toolchain, but I haven't tested this myself.
Then in the output directory, you can just call the compiler
located in [output]/bin/x86_64-unknown-linux-gnu-gcc.
For simple programs, I was able to build stuff using the
header files of my system by passing the correct include directories,
but you'll quickly run into compatibily problems if you try to target
dynamic library .so files which were most likely built using a
newer glibc.
This is also the largest "downside" of crosstool-ng: It only does the toolchain creation. You are on your own when it comes to using that toolchain to compile third-party libraries.
For the occasional one-off third-party library, this is okay. But for e.g. a graphical application which uses the internet, the list is already annoyingly long:
I haven't actually tried doing this manually, but given the fun instructions for OpenSSL, I would prefer if there are solutions which solve this problem for me.
Buildroot tries to make the creation of a Linux environment as easy as possible. It seems like it's primarily targeting embedded Linux systems developement, so I might be misusing this software completely, as you can also create entire images and bootloaders for a variety of cpu architectuers.
However, Buildroot supports configuring and building a bunch of third-party packages. Not as extensive as a large distro package manager, but all of the libraries I listed above are covered, and the entire output is contained in one directory. It's especially nice that you can usually configure whether you want third-party libraries built for dynamical or static linkage, and you can hack some config ontop of existing packages if some configs are missing.
While you can set the GCC / glibc version, it also comes with a few limitations:
The output of Buildroot is movable/relocatable only with the use of a (provided) relocation script.
As far as I know, the GCC & glibc version selection is not as flexible as crosstool-NG,
but I haven't actually tested this, because Buildroot explicitly supports using
toolchains provided by crosstool-NG.
This solves the relocation issues for the binaries (as crosstool-NG does not
require relocation scripts). If I understand correctly, the libraries
only have relocation issues if you plan on using the .la / .pc files
(so if you have more complicated build systems on-top and/or use pkg-config), which I don't.
The most annoying part so far was convincing Buildroot to generate the files I wanted.
Specifically getting the OpenGL-files to be generated took a bunch of trial and error,
but the tool also generates a config file, which I should be able to re-use in the future.
The config (defconfig) looks pretty readable:
BR2_x86_64=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_PATH="/home/user/toolchain_crossng/"
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="x86_64-unknown-linux-gnu"
BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_4=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_SHARED_STATIC_LIBS=y
BR2_PACKAGE_ALSA_UTILS=y
BR2_PACKAGE_PULSEAUDIO=y
BR2_PACKAGE_FUSE_OVERLAYFS=y
BR2_PACKAGE_LIBGLVND=y
BR2_PACKAGE_MESA3D=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SOFTPIPE=y
BR2_PACKAGE_MESA3D_OPENGL_GLX=y
BR2_PACKAGE_MESA3D_OPENGL_EGL=y
BR2_PACKAGE_MESA3D_OPENGL_ES=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_DBUS=y
BR2_PACKAGE_OPENSSL=y
BR2_PACKAGE_LIBFUSE=y
BR2_PACKAGE_LIBGLU=y
BR2_PACKAGE_WAYLAND=y
BR2_PACKAGE_WAYLAND_UTILS=y
BR2_PACKAGE_LIBXKBCOMMON=y
BR2_PACKAGE_LIBXKBCOMMON_TOOLS=yThis is my favorite setup I have found, and I will test the results of this setup more extensively in the future, as I have yet to test the actual portability of the files. I was able to use the crosstool-NG compiler, combined with the header files & library binaries of Buildroot to compile my code, and run the result directly on my host machine.
I have heard of NixOS, which puts a strong emphasis on reproducible setups/configurations. I remember people configuring everything within nix (like e.g. Vim-configuration, not just what packages are installed), so I know that the tool is pretty powerful and covers a lot of packages. The package manager itself, Nix, can also be used outside of NixOS, so I gave it a go.
Before spending too much time on trying to understand the package manager language, I found enough discrepency to think that this tool, although very interesting, is likely not a good fit for my ideal setup.
Nix, by default, uses the /nix/ directory, which goes against my whole relocatable / self-contained preference.
There is a --store argument you can use to make it point to another directory.
However:
/nix directory is still created (okay I guess)/nix/ directory, so they were broken when I pointed to a different --store-location
> ./nix/store/zs1br5qmyfgkfi4wk090li399c8a2gpm-gcc-10.2.0/bin/gcc --version
exec: Failed to execute process './nix/store/zs1br5qmyfgkfi4wk090li399c8a2gpm-gcc-10.2.0/bin/gcc':
The file exists and is executable. Check the interpreter or linker?
If I understand correctly, this is because nix expects the usage of nix-user-chroot.
So it's some sort of container I'd have to work in.
There also seem to be some third-party solutions to my portability requirements:
Nix might very well be capable of fulfilling my requirements, but this was a bit too much of an up-hill battle for me to get into.
If I understand correctly, there is also the GNU equivalent Guix,
but from a quick glance online, it seems like it also puts everything into /gnu/store/.
I also saw mentions of a "portable deployment", but this seems to be about exporting
everything into a tarball, and less of a "this work environment is self-contained" kind of thing.
If I get more into this declarative package manager design, I will probably take another closer look at these
two.
Initially, spack seemed really promising:
When I started writing a configuration, I could list various GCC & glibc versions and third-party libraries, but the actual compilation always failed when glibc was involved. It turns out that, currently, Spack always relies on the glibc version provided by the host system ([1] | [2]), so me targeting an older glibc than my system provides automatically fails the build.
If it weren't for that, it would look very interesting, but this is a dealbreaker for me.
The Yocto Project seems to go in the direction of Buildroot. I haven't heard of it before, but Yocto advertises itself as
But given the tools I covered before, I quickly lost interest in spending too much time on understanding Yocto.
First, I found an official git repo, which was completely empty with a note telling me to switch to a different repo. Probably just a dogshit search engine result, unlucky me.
Trying out bitbake-setup, the initial setup script (after loading some shell environment script)
gives you a bunch of options, none of which I was able to understand intuitively:
OpenEmbedded and poky qemux86-64 and genericx86-64Then came some more struggling getitng any configuration with GCC to work: issues with fish shell, which seemingly got resolved by switching to bash. Me not understanding the differences between config, layers and environments. I'm not even sure Yocto supports the thing I'm trying to do; it just sounded like Buildroot so I assumed it kind of does.
And then when searching up some stuff, I came across a reddit post, with subtle hints that I should probably stick to something simpler:
Ya okay I'll take the hint on this one.
There are two more tools on my list which I haven't gotten to:
There's also a kind-of adjacent design philosopy, which I might as well mention here: Permacomputing. It's not quite a development environment I would use, but the core idea seems to be to vastly simplify the computing environment itself. This is often achieved by using bytecode: Instead of targeting x64, target some sort of fantasy console / some simple bytecode spec which can be emulated.
If the bytecode/fantasy console is simple, you can easily write an emulator. If your code can be converted to bytecode, it can basically run everywhere. This kind of side-steps the entire problem, as you get a lot of control over the entire toolchain / the computing environment itself. Bytecode seems to be a very effective way of preserving code/software; or at least I first heard about it from the old game Another World. PICO-8 is a popular example of a fantasy console, while uxn is probably a good illustration of how the entire computing environment can be simplified.
After all, it's kind of funny how we do have the x86/x64 instruction set, which somehow most desktop hardware just supports so we can even run software across thousands of different CPUs - only to then introduce additional layers on-top which introduce incompatibility.
As mentioned above, crosstool-NG w/ Buildroot seems to cover my usecase well enough so far, so I will most likely proceed with some testing and see if there's anything broken / anything I'm missing.
It still feels like I'm using Buildroot outside of its intended purpose, and given the experiences I had with the other tools, my requirements/preferences don't seem to match with the rest of the software development world. Perhaps other people are prioritizing getting shit done, but I am too stubborn for that.
Felix' Ramblings