I use Debian Stable on almost all the systems I use (one is stuck on 10/Buster due to MoinMoin). I installed Trixie in a container last week, using an LXC container downloaded from linuxcontainers.org [1].
Three things I noted on the basic install :
1) Ping didn't work due to changed security settings (iputils-ping) [2]
2) OpenSSH server was installed as systemd socket
activated and so ignored /etc/ssh/sshd_config*. Maybe this is something specific to the container downloaded.
3) Systemd-resolved uses LLMNR as an name lookup alternative to DNS and pinging a firewalled host failed because the lookup seemed to be LLMNR accessing TCP port 5355. I disabled LLMNR.
Generally, Debian version updates have been succesful with me for a few years now, but I always have a backup, and always read the release notes.
> 2) OpenSSH server was installed as systemd socket activated and so ignored /etc/ssh/sshd_config.
sshd still reads /etc/ssh/sshd_config at startup. As far as I know, this is hard-coded in the executable.
What Debian has changed happens before the daemon is launched: the service is socket activated.
So, _if you change the default port of sshd_ in its config, then you have to change the activation:
- either enable the sshd@service without socket activation,
- or modify the sshd.socket file (`systemctl edit sshd.socket`) which has the port 22 by default.
Since Debian already have a environment file (/etc/default/ssh), which is loaded by this service, the port could be set in a variable there and loaded by the socket activation. But then it would conflict with OpenSSH's own files. This is why I've always disliked /etc/default/ as a second level of configuration in Debian.
I suspect that systemd people are looking at this thread in perplexity, and probably doing their thing (that I've seen over the years) of regarding the world of Debian as being amazingly behind the times in places.
The SSH server being a socket unit with systemd doing all of the socket parallelism-limiting and accepting was one of the earliest examples of socket activation ever given in systemd. It was in one of Lennart Poettering's earliest writings on the subject back in 2011.
And even that wasn't the earliest discussion of this way of running SSH by a long shot, as this was old news even before systemd was invented. One can go back years earlier than even Barrett's, Silverman's, and Byrnes's SSH: The Secure Shell: The Definitive Guide published in 2005, which is one of many places explaining that lots of options in sshd_config get ignored when SSH is started by something else that does all of the socket stuff.
Like inetd.
This has been the case ever since it has been possible to put an SSH server together with inetd in "nowait" mode. Some enterprising computer historian might one day find out when the earliest mention of this was. It might even be in the original 1990s INSTALL file for SSH.
The original SSH server was daemonized by default, but inetd operation was always supported. The INSTALL file said this in 1995:
The server is not started using inetd, because it needs to generate
the RSA key before serving the connection, and this can take about a
minute on slower machines. On a fast machine, and small (breakable)
key size (< 512 bits) it may be feasible to start the server from
inetd on every connection. The server must be given "-i" flag if
started from inetd.
And, of course, SSH was designed to replace rlogin and rsh - both of which ran from inetd as standard. As you say, socket-activation of sshd is simply following very long-standing Unix practice: it's not at all novel, and there's no reason to believe that it's any more risky than any other method of running it.
As you say, network programs activating from a master network management is, of course, the history of Unix. It's ironic to see knee-jerk complaints about it.
systemd-resolved is an effing nightmare when combined with network-manager. these two packages consistently manage to stomp all over DNS resolution in their haste to be the one true source of name resolution. i tried enabling systemd-resolved as part of an effort to do dns over https and i end up with zero dns. i swear that /etc/resolv.conf plus helper scripts is more consistent and easy.
It’s why I always say in the typical “systemd bad” threads that systemd the init system is great, it’s the systemd-* everything else’s that give it a bad name.
I want systemd nowhere fucking near my NTP or DNS config.
I've not had any problems with swapping out systemd-* with other packages, including -coredump, -cron, -oomd, -resolved or -timesyncd. Even journald was fairly painless to swap out. Unlike systemd itself, the distros' approach them not so much as a core part of the userland, but as lightweight basic implementations that meet many user's needs, but which are in no way a replacement for more fully-functional implementations.
(used it before, mostly to learn. went to debian for new laptop. gave up after fighting systemd.
I'm aware of devuan and artix, but gentoo just worked (after all the time spent))
Debian, at least until bookworm works perfectly without systemd. The easiest way to make this transition, is to installed Debian with nothing but 'standard system utilities' and 'SSH server' (if you want) during install:
It's important to remember that it has to be this aggressive in excluding systemd, too. There is quite a lot of strong coupling amongst the parts of systemd, so there are very few half-measure scenarios, where one can have only some systemd stuff, that will actually function correctly in toto.
Moreover, there are the odd one or two unrelated packages that just happen to have the string "systemd" in their names. (-:
MX Linux for the win. Debian based, but defaults to 'init'. Booting with systemd is an option. Just enough systemd-* running to make things easy and seamless.
$ ps agxf|grep 'systemd'
607 ? S 0:00 /lib/systemd/systemd-udevd
2201 ? S 0:00 /sbin/cgmanager --daemon -m name=systemd
2726 ? S 0:00 /lib/systemd/systemd-logind
I’ve been dropping systemd-timesyncd and using chrome since forever, and it works well. I’m sure some systemd-* things are harder to replace, but not every replacement is a fight against your distro.
Yeah, pretty incredible distro. Some rough edges here and there but man, even maintenance is a dream. It's just plain simpler to diagnose, and actually control what the system does. Instead of having to go down abstraction hell land.
Don't get me wrong, systemd is cool. But my god people really abuse of it. Especially distros. Some really make it hard to understand what service is in actual control. Why wrap daemons in daemons in daemons? With the worst possible names and descriptions to boot.
It's the only long-term solution to that problem that I endorse. Every attempt of working with the system, whether via systemd.network, resolved.conf or resolvconf, has always eventually bit me one way or another.
The problem is that SSH is usually the admin interface to the system. You don't want unnecessary moving parts between you and your remote shell access when you need to troubleshoot a half-running system.
> What else is lurking that you and I aren’t aware of?
All of systemd... and network manager/modem manager/...
I grew up with static configuration files, init scripts, inetd, etc... then grew into Solaris smf, dtrace, zones ... and then Linux implemented systemd. I really miss smf and related tools but systemd is still just meh to me. The implementation feels like a somewhat half-in-each-world brainchild.
What is the rationale for changing OpenSSH into a socket activated service? Given that it comes with issues, I assume the benefits outweigh the downsides.
> Given that it comes with issues, I assume the benefits outweigh the downsides.
Any change can introduce regressions or break habits. The move toward socket activation for sshd is part of a larger change in Debian. I don't think the Debian maintainers changed that just for the fun of it. I can think of two benefits:
+ A service can restart without interruption, since the socket will buffer the requests during the restart.
+ Dependencies are simpler and faster (waiting for a service to start and accept requests is costly).
My experience is that these points largely outweigh the downsides (the only one I can think of is that the socket could be written in two places).
I also have a hunch that socket activation allows for more predictability around what a not-running/listening-for-activation service's port is doing at the syscall/kernel level, which in turn makes power saving and/or sleep states a little more predictably efficient.
> Given that it comes with issues, I assume the benefits outweigh the downsides.
I think it doesn't outweight the downside. Let's not forget this:
"OpenSSH normally does not load liblzma, but a common third-party patch used by several Linux distributions causes it to load libsystemd, which in turn loads lzma."
The "XZ utils backdoor" nearly backdoored every single distro running systemd.
People (including those who tried to plant this backdoor) are going to say: "systemd has nothing to do with the backdoor" but I respectfully disagree.
systemd is one heck of a Rube-Goldberg piece of machinery: the attack surface is gigantic seen that systemd's tentacles reaches everywhere.
With a tinfoil hat on one could think the goal of systemd was, precisely, to make sure the most complicated backdoors could be inserted here and there: "Let's have openssh use a lib it doesn't need at all because somehow we'll call libsystemd from openssh".
Genius idea if you ask me.
What could possibly go wrong with systemd "now just opening a port for openssh" uh? Nothing I'm sure.
Now that said I'm very happy that we've now got stuff like the Talos Linux distribution (ultra minimal, immutable, distro meant to run Kubernetes with as few executables as possible and of course no systemd) and then containers using Alpine Linux or, even if Debian based, minimal system with (supposedly) only one process running (and, once again, no systemd).
Containerization is one way out of systemd.
I can't wait for a good systemd-less hypervisor: then I can kiss Microsoft goodbye (systemd is a Microsoft technology, based on Microsoftism, by a now Microsoft employee).
Thanks but no thanks.
Talos distro, systemd-less containers: I want more of this kind of mindset.
The future looks very nice.
systemd lovers should just throw the towel in and switch to Windows: that's what they actually really want and it's probably no better than they deserve.
> 2) OpenSSH server was installed as systemd socket activated and so ignored /etc/ssh/sshd_config*. Maybe this is something specific to the container downloaded.
Doesn't the .socket unit point to a .service unit? Why would using a socket be connected to which config sshd reads?
I use Debian Stable on almost all the systems I use (one is stuck on 10/Buster due to MoinMoin). I installed Trixie in a container last week, using an LXC container downloaded from linuxcontainers.org [1].
Three things I noted on the basic install :
1) Ping didn't work due to changed security settings (iputils-ping) [2]
2) OpenSSH server was installed as systemd socket activated and so ignored /etc/ssh/sshd_config*. Maybe this is something specific to the container downloaded.
3) Systemd-resolved uses LLMNR as an name lookup alternative to DNS and pinging a firewalled host failed because the lookup seemed to be LLMNR accessing TCP port 5355. I disabled LLMNR.
Generally, Debian version updates have been succesful with me for a few years now, but I always have a backup, and always read the release notes.
[1] https://linuxcontainers.org
[2] https://www.debian.org/releases/trixie/release-notes/issues....