12 KiB

Raw Blame History

systemd services in NixOS and hardening of them

introduction

When it comes to security, we care about limiting access of each entity of a system to as few other entities as possible. Network input, executables and users must be able to reach only those resources, which are necessary to perform the defined server tasks.

Generally, it's better to implement as many layers of security as possible. Although, there is no way to make a server 100% bullet proof - it's a huge endless topic, this article covers some feasible essential systemd tunables that give us a layer of protection.

Systemd is the standard software suite for organizing and running services/daemons in a modern GNU/Linux distribution, including NixOS. Systemd provides means to secure services. And in many ways, the isolation level of a systemd service can be similar to that of containers (by the means of sandboxing, namespaces, cgroups, etc; interestingly, systemd even allows running multiple instances of the same service). However, systemd hardening defaults are quite loose (perhaps, not to disturb the operation of newly written services and their administrators in any way).

What NixOS does - it generates systemd configuration files in accordance to NixOS configuration given, written in Nix language. To some extent, Nix acts as a macro language and NixOS configuration module system acts as a unified control center, so that you don't bother about location of systemd files, their syntax and common stuff, which NixOS generates for you. Also, NixOS manages runtime switching between systemd configurations, conducting services restarts when required and whole system rollbacks from GRUB/systemd-boot/extlinux.

overview of systemd services integration within NixOS configuration

NixOS features lots of systemd services, which are ready to use (without even knowing what systemd is) just by setting appropriate options in configuration.nix. For example, write services.netdata.enable = true; to enable Netdata monitoring service. Documentation for all related options can be found on the website or in man configuration.nix (also in man home-configuration.nix for managing desktop user services). Often many useful high-level tunables are available as services.<name>.* options.

When services, provided by NixOS, are insufficient or additional tuning is demanded, systemd.services.<name>.* set of options comes into play. They allow to define custom systemd services or modify existing ones. Regardless of the origin of a systemd service (provided by NixOS or written by yourself), systemd native options for sections such as [Unit] and [Service] can be specified accordingly in the following nix attribute sets: systemd.services.<name>.unitConfig and systemd.services.<name>.serviceConfig. [Install] section options such as Alias, WantedBy and RequiredBy can be specified as nix lists in systemd.services.<name>.aliases, systemd.services.<name>.wantedBy, systemd.services.<name>.requiredBy. You can find more information about such options online or in man configuration.nix as usual.

In a nutshell, configuring systemd options for services on NixOS typically boils down to these steps:

edit systemd.services.* options in configuration.nix or in other imported nix files;
run sudo nixos-rebuild test to apply new configuration just for now or sudo nixos-rebuild switch to apply changes permanently;
evaluate systemd service operation (we will elaborate on this further);
return to step 1 or finish.

Alternatively, new configurations can be tested inside a QEMU VM clone of your system without affecting your running system configuration. nixos-build build-vm leaves a symlink ./result in the current directory that contains the built VM. To run it, use result/bin/run-<hostname>-vm.

Be aware that systemd directives (options) are case sensitive! But NixOS doesn't know whether systemd recognizes any directives or not, whereas systemd does not complain neither! So, once new configuration is applied, analyze output of these commands and compare with intended objectives:

systemd cat <name> - contents of a systemd unit file, generated by NixOS
systemd show <name> - actual properties of a systemd unit in use

Also, keep in mind that mutable operations like systemd SERVICE enable are useless, because they would deviate the system from declarative reproducible configuration and NixOS won't let or will stubbornly resist you doing so at the design level. And there is no need, since each permanent setting is in the hands of NixOS.

hardening

NixOS provides many services, available as services.<name>.*, which already have more or less hardening implemented by the means of systemd. For example, services.nginx, services.gitea, services.jitsi-meet, services.redis. At least, these services run under specific system non-root users without access to spawn a shell.

There are, however, services like services.dovecot2, services.postfix and services.nextcloud, which use their own means to spawn sub-processes under a specific user by a master process. Such master process is run under root. For example, nextcloud uses php:fpm (PHP FastCGI Process Manager). Obviously, shell can be spawned by such processes and a lot more, but they do not have network connections outside world and intended specifically for process/workers management and logging. Ideally, we would want them to be run under non-root user regardless, but usually it's not easy to do and upstream might not expect such usage.

blocking outgoing internet connections

The idea is to keep responding to incoming requests to some service, but forbid any outgoing connections, initiated by itself.

When it comes to a more sophisticated firewall, unfortunatelly systemd is not capable of such granular control. So, iptables configuration will be:

networking.firewall = {
  extraCommands = ''
    iptables -t filter -I OUTPUT 1 -m owner --uid-owner ${user} -m state --state NEW -j REJECT
  '';
  extraStopCommands = ''
    iptables -t filter -D OUTPUT 1 -m owner --uid-owner ${user} -m state --state NEW
  '';
};

resources limits strategy

Systemd resource control settings allow you to limit the resources provided to a service. For example, if MemoryMax limit is exceeded, OOM killer gets invoked.

systemd.service = {
  nginx = {
    serviceConfig = {
      CpuAccounting = true;
      CpuQuota = "70%";
      MemoryAccounting = true;
      MemoryMax = "768M";
      BlockIOWeigth = 10;
    };
  };
}

MemoryMax is the absolute limit. It is recommended to use MemoryHigh as the main control mechanism, because it allows to go above the limit, but the processses are heavily slowed down and memory is taken away aggressively.

Btw, if your systemd service code gets large and you want to wrap it into something more esthetic, you can write your own NixOS service module.

cgroups

cgroup - control group. Docker's isolation implementation is also based on cgroups.

Enabling netdata service in NixOS enables systemd.enableCgroupAccounting, which in turn enables these options in systemd.conf:

DefaultCPUAccounting=yes
DefaultIOAccounting=yes
DefaultBlockIOAccounting=yes
DefaultIPAccounting=yes

list of systemd options and their implications

testing

basic systemd commands for diagnostics

systemd status, systemd restart, systemd cat, htop tree

systemd-analyze's words "SAFE", "EXPOSED" and "UNSAFE" do not mean the factual situation, rather whether various systemd hardedning features are in use or not.

when trying systemd options alone

You can manually test various systemd options without writing service files with the help of systemd-run, for example:

$ ls -l /home
total 0
drwx------ 1 alex users 1126 2023-06-21 19:26 alex

sudo systemd-run -p ProtectHome=yes --shell
Running as unit: run-u2544.service
Press ^] three times within 1s to disconnect TTY.

# ls -l /home
total 0

# exit
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 2.749s
CPU time consumed: 50ms
IP traffic received: 0B
IP traffic sent: 0B
IO bytes read: 0B
IO bytes written: 0B

using tmux shell via socket inside a systemd unit

With the help of tmux you can run a shell inside a hardened systemd unit in order to test our isolation in practice. Here is example-systemd-service.nix nix file, the path to which you can add to the imports list in configuration.nix and then execute nixos-rebuild switch or nixos-rebuild test (if you don't want new configuration to be permanent; however, it leaves ./result symbolic link in current directory).

# nix-shell -p tmux --run "tmux -S /run/example-service/tmux.socket attach"

existing practices and solutions within NixOS

There is no universal way in configuring systemd services options sandboxing/hardening for all services.

unsolved problems

systemd.services.<name>.confinement.enable NixOS option is not compatible with systemd's ProtectSystem.

final notes

Systemd hardening is just a part of measures to be taken to narrow the potential threat landscape and risks for a server. Ideally, vulnerabilities scanning, penetration testing, unauthorized access prevention and security audits should be involved. Take advantage of monitoring tools and respond quickly, according to a rescue plan to mitigate the impact of incidents. This might include restoring system from backups, keys and passwords reset, etc. Keep running software up to date and respond to CVEs (deploying software with patches is easy in NixOS in case it hasn't been already patched). Have a business continuity plan. Many measures must not be ad-hoc, but rather systematic to stay vigilant against emerging threats.

As for NixOS, it also features security.apparmor, security.audit and even programs.firejail options which might help in building a more secure system.

12 KiB Raw Blame History