I have mentioned before that I really, really, really do not like systemd. Whilst it remained a simple replacement for init (albeit with some peculiarites) I could put up with it so long as it didn’t get in my way, or my way of working. After all, if all the major distro developers were convinced it was the way to go, then who am I to disagree? But it hasn’t just remained an init replacement, like Topsy, it “just growed”, and “growed”, and “growed”, until now it is involved in process management way beyond what seems reasonable to me (and to others).
Here is a process listing from my Mint desktop:
mick@shed ~ $ ps -ef | grep systemd
root 379 1 0 12:44 ? 00:00:00 /lib/systemd/systemd-journald
root 408 1 0 12:44 ? 00:00:01 /lib/systemd/systemd-udevd
systemd+ 665 1 0 12:44 ? 00:00:00 /lib/systemd/systemd-networkd
message+ 687 1 0 12:44 ? 00:00:00 /usr/bin/dbus-daemon –system –address=systemd: –nofork –nopidfile –systemd-activation –syslog-only
root 723 1 0 12:44 ? 00:00:00 /lib/systemd/systemd-logind
root 726 1 0 12:44 ? 00:00:00 /lib/systemd/systemd-machined
root 729 1 0 12:44 ? 00:00:01 /usr/sbin/thermald –systemd –dbus-enable –adaptive
systemd+ 747 1 0 12:44 ? 00:00:00 /lib/systemd/systemd-resolved
mick 2406 1 0 12:44 ? 00:00:00 /lib/systemd/systemd –user
mick 2433 2406 0 12:44 ? 00:00:00 /usr/bin/dbus-daemon –session –address=systemd: –nofork –nopidfile –systemd-activation –syslog-only”
I have been a unix/linux user for well over 35 years. My backround is in systems which used the System V init (sysvinit) approach (as an astute reader would note from my use of the “ps” command above. System V guys routinely and automatically use “ps -ef”, BSD guys use “ps aux” even though on modern linuxes either formulation will work). I am used to the “/etc/init.d/daemond stop|start|restart” etc. approach for starting/stopping daemons and object when systemd interferes with that. I also much prefer classic text logfiles so that I can do something like “/etc/init.d/daemond restart ; tail -f /var/log/daemond.log” (Yes, OK, I could do it the systemd way and use “systemctl restart daemon.service ; journalctl -f” but I don’t see why I should.)
Systemd uses a silly binary format and stores all log information in a “journal” controlled by the journald daemon. Systemd fans say that because most of the boot process and service management is now handled by the systemd process, it makes sense to standardise the way that logs are collected and accessed. So the systemd-journald daemon collects data from all available sources and stores them in a binary format for easy and dynamic manipulation. Well, yes, if you must use one monolithic system to do everything then of course it makes sense. But the unix system was originally designed around the principle of one tool doing one thing and doing it well. Systemd ignores that concept and tries to do everything (well, more and more and more than its original function of a parallelised init process). It now gets involved in network management, user logins, DNS resolution, disk and device management, and so on. For a good example of how systemd makes what was a relatively simple process unnecessarily complicated take a look at the file /etc/resolv.conf from one of my VMs.
This what a systemd resolv.conf file looks like (I love the “do not edit” bit):
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run “resolvectl status” to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.nameserver 127.0.0.53
options edns0 trust-ad
and this is what my preferred file looks like:
# Generated by NetworkManager
domain baldric.net
search baldric.net
nameserver 192.168.200.1
And I actually prefer, and often employ, manually edited resolv.conf files on my servers where I do not even use Network Manager. Why the hell should I even need a systemd.resolvd daemon?
Furthermore, systemd “timers” replace the venerable cron which can have unfortunate effects. However I must concede that systemd timers can be more useful, and certainly more flexible and accurate than cron in some circumstances.
But first, before I continue to discuss why I don’t like systemd. let us take a step back and look at what it is, and how and why it came into being.
The demon (deliberate misspelling) systemd was first released in 2010 and was used in Fedora in 2011. It was developed by Lennart Poettering and Kay Sievers, two software engineers at RedHat (which is why it first appeared in the Fedora desktop system). Since 2010, it’s been adopted by many, in fact most, other distributions.
Debian agreed to change to systemd for init in 2014 after a vote within the Technical Committee. This vote was not taken lightly, and there is a note on the Debian wiki about the debate. Debian has not committed irrevocably to systemd and could, in theory, back it out. But that would be problematic.
Arch changed in 2012 and this post explains why.
Ubuntu (which is based on Debian) changed in 2015 (in the 15.04 release) after first trialling their own SysV init replacement “upstart”. This post on their wiki explains how to move to systemd from upstart. Derivatives and re-spins of Ubuntu (such as Xubuntu, Kubuntu etc) as well as distros such as Mint (my choice since late 2011) will, of necessity, also use systemd.
Here is what Wikipedia has to say about systemd:
Since 2015, the majority of Linux distributions have adopted systemd, having replaced other systems such as the UNIX System V (also known as SysVinit). Systemd may be one of the most, if not the most polarizing piece of software in the Linux community.
Most arguments against systemd are that it suffers from mission creep and bloat. Subsequent criticism also affects other software (such as the GNOME desktop) adding dependencies on systemd complicating compatibility with other Unix-like operating systems, and making it hard to move away from systemd. Concerns have also been raised about Red Hat and its parent company IBM controlling the scene of init systems on Linux. Some even doubt the integrity of systemd against attackers, claiming that the complexity of systemd results in a greatly enlarged attack surface, reducing the overall security of the platform.
And this is what the project website itself has to say:
systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system.
systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, maintains mount and automount points, and implements an elaborate transactional dependency-based service control logic. systemd supports SysV and LSB init scripts and works as a replacement for sysvinit.
Other parts include a logging daemon, utilities to control basic system configuration like the hostname, date, locale, maintain a list of logged-in users and running containers and virtual machines, system accounts, runtime directories and settings, and daemons to manage simple network configuration, network time synchronization, log forwarding, and name resolution.
It is the “other parts” bit I find most problematic. And I particularly dislike the (increasingly deep) integration of systemd components into desktop UIs (such as in Gnome) which should be completely agnostic about the underlying system architecture. That integration forces “non-systemd” distros to either ignore the desktop or application or fudge the issue by introducing some sort of api “shim” to capture the expected interaction with systemd components and enable the application to work.
I’m far from alone in worrying about systemd and where it is going. A search for “systemd problems” or “systemd controversies” or similar will get you plenty of examples. Back in May 2016, E. M Smith over at chiefio quoted Tolkien’s maxim, “One ring to rule them all, one ring to bind them…” where he said:
“So by design this thing is supposed to isolate you and install itself between all other code and the kernel. To take control of all outside the kernel. And to force “uniformity”. That is not The Unix Way, nor the FOSS way, nor even “playing well with others”. Not nice.”
Systemd developers also have an unfortunate, and rather cavalier, attitude to both the security problems with their code and the problems they cause other developers. Linus Torvalds famously ranted at Kay Sievers on the Linx Kernel mailing list saying, inter alia:
“I’m not willing to merge something where the maintainer is known to not care about bugs and regressions and then forces people in other projects to fix their project. Because I am *not* willing to take patches from people who don’t clean up after their problems, and don’t
admit that it’s their problem to fix.Kay – one more time: you caused the problem, you need to fix it. None of this “I can do whatever I want, others have to clean up after me” crap.”
Of late, not content with systemd taking over more and more of the management of the OS, its developers now intend messing with the user’s home directory system. And they seem to want to do this for some odd ideological reasons rather than any demands from the users themselves that this is either wanted or necessary.
As all linux users know well, the home directory (usually mounted at /home/username on a separate partition to that used for system files) stores all the user’s data such as documents, spreadsheets, videos, photographs, email etc. together with some (hidden) files which store the settings for the applications which the user actually uses. The home directory also stores critical digital identity information (such as ssh or pgp keys) This means that, for example, as soon as the user logs in, his or her desktop settings are applied and when that user fires up his or her email client, that client will know where to fetch mail from, and what the credentials are which will allow that transfer. Similarly, when the user opens his or her browser, that browser will have access to all the previously bookmarked webpages together with (possibly) any login credentials for those pages.
The advantage of having a separate user home directory system is fairly obvious in that it allows for the data to be kept safe during system upgrades, or even between moves to completely different distributions. My own current home directory has been around for quite some time now and has moved through various versions of various different distros with no loss of data.
But not /all/ the information about users is stored within their home directories. In particular, the user’s UID and password are stored in system files (/etc/passwd and /etc/shadow). And it is here that systemd developers seem to want to change things. Under the new systemd-homed.service user’s complete identity data (including UID, password and default shell) will be stored in a JSON formatted record in a (hidden) file called .identity. Now given that this new home directory system contains not just sensitive personal data, but also critical authentication data, the new structure will be encrypted by default. Systemd envisages this encryption mechanism being fscrypt.
In the wierd parallel universe occupied by systemd developers (OK, I’m prejudiced) this complete encapsulation of all a user’s data, including metadata describing the user and his/her authentication credentials, in one place, means that the entire home directory could be completely portable (and stored on, say, a USB stick). The systemd documentation even suggests it could be stored on a separate CIFS server!
Why would I want that? What happens if my UID is already in use by another user on the system I try to log in to? Worse, given that the entire directory structure is encrypted. what happens if I want to login remotely over ssh? (I have loads of different home directories on different remote VMs). Since the keys will not be accessible until the directory is decrypted and since the directory cannot be decrypted until I login, how do I do this?
Who asked for this nonsense? Seriously. Who wanted this?
When I had problems re-flashing my new Motorola G7 Plus back in May of last year I found that the problems disappeared when I moved the process to a systemd free system I was testing. At the time I said:
“Now I cannot be sure what was wrong, or why the change of PC should have made any difference, there are too many variables at play here. The second PC runs an AMD processor, not Intel so it has a completely different architecture from the motherboard up. It runs a version of linux without systemd (which I may move to entirely when I have finished my testing), and it runs a completely different kernel (version 5.10.0-5mx-amd64 rather than 5.4.0-73-generic) so it is probably unfair to blame systemd – but I’m going to anyway. It gives me one more reason to hate it.”
That may sound irrational, but it sums up how I was feeling about systemd at the time.
Given that Ubuntu is tied closely to systemd and will be implementing the ridiculous systemd-homed.service shortly, and that Mint is based on Ubuntu and will perforce probably follow, I have now given up on Mint and moved to another distro which will not do so. In my next post I will discuss which distro, and why I chose it.
Footnote:
(For a lengthy, fairly comprehensive history of the development of systemd which places it into context with other developments at the time, I recommend you read VR’s post on darknedgy. The same author on that blog also wrote a fascinating history of modern init systems.)