Support systemd in containers with podman-style `--systemd` flag #2785

sazzy4o · 2024-02-01T02:49:05Z

Adds support for systemd to nerdctl with a --systemd based on the flag used in podman

Usage:

$ sudo nerdctl run --systemd=always --rm -it registry.hub.docker.com/sazzy4o/build
:systemd
systemd v246.15-1.fc33 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Fedora 33 (Container Image)!

Set hostname to <9eff88d04af6>.
Queued start job for default target Graphical Interface.
[  OK  ] Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/modprobe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on User Database Manager Socket.
         Starting Rebuild Dynamic Linker Cache...
         Starting Journal Service...
         Starting Create System Users...
[  OK  ] Finished Rebuild Dynamic Linker Cache.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Create System Users.
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Rebuild Journal Catalog...
         Starting Network Name Resolution...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Finished Rebuild Journal Catalog.
         Starting Update is Completed...
[  OK  ] Finished Update is Completed.
[  OK  ] Reached target System Initialization.
[  OK  ] Started dnf makecache --timer.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting MariaDB 10.4 database server...
         Starting The PHP FastCGI Process Manager...
         Starting Home Area Manager...
         Starting User Login Management...
         Starting Permit User Sessions...
         Starting D-Bus System Message Bus...
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
         Starting The nginx HTTP and reverse proxy server...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Home Area Manager.
[  OK  ] Started User Login Management.
[  OK  ] Started The PHP FastCGI Process Manager.
[  OK  ] Started The nginx HTTP and reverse proxy server.
[  OK  ] Started MariaDB 10.4 database server.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

Fedora 33 (Container Image)
Kernel 6.5.0-14-generic on an x86_64 (console)

9eff88d04af6 login:

sazzy4o · 2024-02-01T02:50:34Z

cmd/nerdctl/container_run.go

@@ -184,6 +184,7 @@ func setCreateFlags(cmd *cobra.Command) {
 	cmd.Flags().StringSlice("cap-drop", []string{}, "Drop Linux capabilities")
 	cmd.RegisterFlagCompletionFunc("cap-drop", capShellComplete)
 	cmd.Flags().Bool("privileged", false, "Give extended privileges to this container")
+	cmd.Flags().String("systemd", "false", "Enable systemd integration (default: false)")


This differs from the podman flag, which has a default value of true

sazzy4o · 2024-02-01T02:51:24Z

pkg/cmd/container/create.go

@@ -173,7 +175,6 @@ func Create(ctx context.Context, client *containerd.Client, args []string, netMa
 		return nil, nil, err
 	}
 	cOpts = append(cOpts, restartOpts...)
-	cOpts = append(cOpts, withStop(options.StopSignal, options.StopTimeout, ensuredImage))


Moved. See: https://github.com/containerd/nerdctl/pull/2785/files#diff-2ab6676c895e1ac2caa11b1f39c2b22d51b4c8c8219119492ab124d3de0ff2cdR409

AkihiroSuda · 2024-02-01T02:53:53Z

pkg/cmd/container/create.go

+	systemdPaths := []string{
+		"/usr/sbin/init",
+		"/sbin/init",
+		"/usr/local/sbin/init",


Is there any image that has /usr/local/sbin/init?

I set these to be the same as podman:

true enables systemd mode only when the command executed inside the container is systemd, /usr/sbin/init, /sbin/init or /usr/local/sbin/init.

(But, most images I have seen use /sbin/init)

https://docs.podman.io/en/latest/markdown/podman-run.1.html#systemd-true-false-always

AkihiroSuda · 2024-02-01T02:54:09Z

pkg/cmd/container/create.go

+
+	systemdPaths := []string{
+		"/usr/sbin/init",
+		"/sbin/init",


Wondering if /sbin should have higher priority over /usr/sbin

Yeah, /sbin/init is more common. I'll move higher

sazzy4o · 2024-02-01T02:54:39Z

pkg/cmd/container/create.go

+
+			// See: https://github.com/containers/podman/issues/15878
+			if !privilegedWithoutHostDevices {
+				return nil, nil, errors.New("If --privileged is used with systemd `--security-opt privileged-without-host-devices` must also be used")


In the podman implementation all /dev/tty* devices are unmounted to prevent causing host to crash

I could not find an easy to achieve this, so instead I return an error (also prevents causing host to crash). If that functionality is required, maybe it can be added in a future PR

AkihiroSuda · 2024-02-01T02:54:44Z

pkg/cmd/container/create.go

@@ -27,6 +27,7 @@ import (
 	"path"
 	"path/filepath"
 	"runtime"
+	"slices"


This doesn't work with Go 1.20.
I guess we can drop the support for Go 1.20 though.

Switch to using or statement to maintain compatibility with 1.20

AkihiroSuda · 2024-02-01T02:55:18Z

docs/command-reference.md

@@ -232,6 +232,8 @@ Security flags:
 - :whale: `--cap-add=<CAP>`: Add Linux capabilities
 - :whale: `--cap-drop=<CAP>`: Drop Linux capabilities
 - :whale: `--privileged`: Give extended privileges to this container
+- :nerd_face: `--systemd=(true|false|always)`: Enable systemd compatibility (default: false).


How does always differ from true?

You may also want to update https://github.com/containerd/nerdctl?tab=readme-ov-file#features-present-in-nerdctl-but-not-present-in-docker

Added some more to docs about the options and added a note to nerdctl specific features in README:
https://github.com/containerd/nerdctl/pull/2785/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R206

cmd/nerdctl/container_run.go

sazzy4o · 2024-02-01T02:57:05Z

pkg/cmd/container/create.go

+				{Type: "tmpfs", Source: "tmpfs", Destination: "/var/lib/journal"},
+			}),
+		)
+		stopSignal = "SIGRTMIN+3"


SIGTERM causes restart in systemd (This functionality is the same as podman)

See:
https://www.freedesktop.org/software/systemd/man/latest/systemd.html#Signals

AkihiroSuda · 2024-02-01T02:57:37Z

Thanks,

please add tests
please squash commits

sazzy4o · 2024-02-01T02:59:05Z

pkg/cmd/container/create.go

+	} else if len(ensured.ImageConfig.Cmd) > 0 {
+		entrypointPath = ensured.ImageConfig.Cmd[0]
+	}
+


There might be an easier way to determine the entrypoint executable path, if there is I am open to updating

.github/workflows/test.yml

sazzy4o · 2024-02-04T03:25:20Z

cmd/nerdctl/container_stop_linux_test.go

@@ -84,5 +84,6 @@ while [ $quit -ne 1 ]; do
 done
 echo "signal quit"`).AssertOK()
 	base.Cmd("stop", testContainerName).AssertOK()
+	base.Cmd("inspect", "--format", "{{json .Config.Labels}}", testContainerName).AssertOutContains("SIGQUIT")


Saw that this test was not checking container labels, so added assert

sazzy4o · 2024-02-04T03:26:23Z

pkg/testutil/testutil_linux.go

@@ -41,6 +41,7 @@ var (
 	DockerAuthImage             = mirrorOf("cesanta/docker_auth:1.7")
 	FluentdImage                = mirrorOf("fluent/fluentd:v1.14-1")
 	KuboImage                   = mirrorOf("ipfs/kubo:v0.16.0")
+	SystemdImage                = "ghcr.io/containerd/stargz-snapshotter:0.15.1-kind"


Those this image since it has a working systemd and is controlled by containerd

(with --systemd flag) Signed-off-by: Spencer von der Ohe <[email protected]>

sazzy4o · 2024-02-04T15:54:10Z

@AkihiroSuda I have added some tests and squashed the commits. Could you please have another look?

AkihiroSuda

Thanks

jfernandez · 2024-11-10T18:58:32Z

pkg/cmd/container/create.go

+		}
+
+		opts = append(opts,
+			oci.WithoutMounts("/sys/fs/cgroup"),


@sazzy4o I found your change while looking into supporting containers with systemd using k8s + containerd. I did the tmpfs mounts for /run, /tmp/, etc., but I was mounting the host's /sys/fs/cgroup as ready-only, which didn't work.

Here, you are removing the mount, which caught my attention. Is this so that systemd creates /sys/fs/cgroup when it initializes?

@jfernandez Yes, this allow systemd to run inside the container and create /sys/fs/cgroup

This was based on the podman --systemd flag

sazzy4o commented Feb 1, 2024

View reviewed changes

AkihiroSuda reviewed Feb 1, 2024

View reviewed changes

sazzy4o commented Feb 1, 2024

View reviewed changes

AkihiroSuda reviewed Feb 1, 2024

View reviewed changes

cmd/nerdctl/container_run.go Outdated Show resolved Hide resolved

sazzy4o commented Feb 1, 2024

View reviewed changes

AkihiroSuda added this to the v2.0.0 milestone Feb 1, 2024

AkihiroSuda added the enhancement New feature or request label Feb 1, 2024

sazzy4o commented Feb 1, 2024

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

sazzy4o commented Feb 4, 2024

View reviewed changes

sazzy4o force-pushed the main branch 5 times, most recently from 5cd85bf to ed090fb Compare February 4, 2024 06:18

Add support to run systemd in containers

741d06e

(with --systemd flag) Signed-off-by: Spencer von der Ohe <[email protected]>

sazzy4o force-pushed the main branch from ed090fb to 741d06e Compare February 4, 2024 06:30

sazzy4o requested a review from AkihiroSuda February 4, 2024 15:54

AkihiroSuda approved these changes Feb 5, 2024

View reviewed changes

AkihiroSuda merged commit 9b76bcc into containerd:main Feb 5, 2024
22 checks passed

github-actions bot mentioned this pull request Nov 10, 2024

Bump nerdctl from 1.7.6 to 2.0.0 kokyhm/kubespray#73

Open

jfernandez reviewed Nov 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support systemd in containers with podman-style `--systemd` flag #2785

Support systemd in containers with podman-style `--systemd` flag #2785

sazzy4o commented Feb 1, 2024

sazzy4o Feb 1, 2024 •

edited

Loading

sazzy4o Feb 1, 2024

AkihiroSuda Feb 1, 2024

sazzy4o Feb 1, 2024 •

edited

Loading

AkihiroSuda Feb 1, 2024

sazzy4o Feb 1, 2024

sazzy4o Feb 1, 2024 •

edited

Loading

AkihiroSuda Feb 1, 2024

sazzy4o Feb 1, 2024

AkihiroSuda Feb 1, 2024

AkihiroSuda Feb 1, 2024

sazzy4o Feb 1, 2024

sazzy4o Feb 1, 2024

AkihiroSuda commented Feb 1, 2024

sazzy4o Feb 1, 2024

sazzy4o Feb 4, 2024

sazzy4o Feb 4, 2024

sazzy4o commented Feb 4, 2024

AkihiroSuda left a comment

jfernandez Nov 10, 2024 •

edited

Loading

sazzy4o Nov 10, 2024

Support systemd in containers with podman-style --systemd flag #2785

Support systemd in containers with podman-style --systemd flag #2785

Conversation

sazzy4o commented Feb 1, 2024

sazzy4o Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sazzy4o Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sazzy4o Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Feb 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sazzy4o commented Feb 4, 2024

AkihiroSuda left a comment

Choose a reason for hiding this comment

jfernandez Nov 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Support systemd in containers with podman-style `--systemd` flag #2785

Support systemd in containers with podman-style `--systemd` flag #2785

sazzy4o Feb 1, 2024 •

edited

Loading

sazzy4o Feb 1, 2024 •

edited

Loading

sazzy4o Feb 1, 2024 •

edited

Loading

jfernandez Nov 10, 2024 •

edited

Loading