The Great Divide: Running Systemd Inside Docker Versus Managing Docker with Systemd

The intersection of system initialization and containerization has long been a subject of intense debate, technical experimentation, and architectural restructuring within the infrastructure engineering community. For years, the prevailing wisdom dictated a strict separation of concerns: containers should run a single process, and the host operating system should handle service management via init systems like systemd. However, as infrastructure requirements grow in complexity, engineers have found themselves navigating two distinct, yet often conflated, paradigms. The first paradigm involves running the systemd init system inside a Docker container to manage multiple services within a single isolated environment. The second paradigm involves using the host’s systemd to manage the lifecycle of Docker containers or Docker Compose stacks as if they were traditional system services. These approaches serve fundamentally different architectural goals, present unique technical challenges, and require specific configurations to function reliably. Understanding the nuances of each approach is critical for DevOps engineers, system administrators, and developers who must balance isolation, persistence, security, and ease of management.

The debate is not merely academic; it impacts how services are deployed, how logs are managed, how security boundaries are defined, and how systems recover from failures. Running systemd inside a container challenges the "one process per container" philosophy that has been central to Docker's design since its inception. It introduces complexity related to cgroup management, namespace isolation, and filesystem overlays. Conversely, managing Docker containers via systemd on the host offers a familiar interface for service management, leveraging the robust dependency tracking, logging, and restart policies inherent to systemd, while still benefiting from the isolation provided by containers. This article explores both methodologies in exhaustive detail, drawing from practical implementations, community discussions, and technical documentation to provide a comprehensive guide for navigating this complex landscape.

Running Systemd Inside a Docker Container

The concept of running systemd inside a Docker container emerged from the need to run legacy applications or complex service stacks that were not designed for containerization. These applications often rely on systemd for service discovery, dependency management, or initialization sequences that cannot be easily replicated in a single-process container entrypoint. While Docker’s official documentation and community leaders have historically advised against this practice, citing the principle that a container should represent a single atomic unit of execution, the reality of enterprise infrastructure often demands more flexibility. The community has responded by creating specialized base images and configuration techniques that allow systemd to function within the constrained environment of a container.

One of the most prominent solutions for this challenge is the jrei/systemd-ubuntu image, available on Docker Hub. This image provides pre-configured Ubuntu environments with systemd enabled, supporting a wide range of Ubuntu releases including 16.04, 18.04, 20.04, 22.04, 24.04, 25.04, and the development release 25.10. The image serves as a foundation for users who need to run systemd services inside a container without manually configuring the init system for each new deployment. The image size is approximately 44.3 MB, which is relatively compact for a full operating system base, making it suitable for environments where storage space is a consideration. The last update to the image was recorded 21 days ago, indicating active maintenance and responsiveness to upstream Ubuntu changes.

To run a systemd-enabled container using this image, specific Docker run commands are required to ensure that systemd can interact correctly with the host’s kernel and cgroup filesystem. The standard command involves mounting the host’s cgroup filesystem into the container and creating temporary filesystem mounts for essential directories such as /tmp, /run, and /run/lock. The command structure is as follows:

docker docker run -d --name systemd-ubuntu --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:ro jrei/systemd-ubuntu

This command creates a detached container named systemd-ubuntu. The --tmpfs flags mount temporary filesystems for /tmp, /run, and /run/lock, which are necessary because these directories are typically used by systemd for runtime data, sockets, and locks. Mounting them as tmpfs ensures that they are writable and ephemeral, preventing data persistence issues that could arise from overlay filesystems. The -v /sys/fs/cgroup:/sys/fs/cgroup:ro flag mounts the host’s cgroup filesystem in read-only mode, allowing systemd to monitor and manage resources within the container’s namespace. The ro (read-only) flag is crucial for security, as it prevents the container from modifying the host’s cgroup structure.

In some cases, the standard cgroup mounting approach may fail due to kernel version differences, namespace restrictions, or security policies. In such scenarios, a more permissive approach is required. This involves running the container with the --privileged flag, which grants the container almost all capabilities of the host, including access to devices and the ability to modify system settings. The alternative command is:

docker docker run -d --name systemd-ubuntu --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro jrei/systemd-ubuntu

While this command resolves many compatibility issues, it significantly reduces the security isolation of the container. Privileged containers can access the host’s devices, modify kernel parameters, and potentially compromise the host system if the container is exploited. Therefore, this approach should be used with caution and only in trusted environments where the risks are understood and mitigated.

Once the container is running, users can interact with it using the docker exec command to enter a shell and manage systemd services. For example:

docker docker exec -it systemd-ubuntu sh

This command opens an interactive shell inside the container, allowing the user to run systemctl commands to enable, start, stop, and manage services as they would on a standard Ubuntu system. This capability is particularly useful for debugging, testing, or deploying complex service stacks that require multiple interdependent processes.

However, running systemd inside a container is not without its pitfalls. One of the most common issues encountered by users is the failure of services to start automatically when the container restarts. This is because systemd’s enable command creates symlinks in the /etc/systemd/system directory to ensure that services start at boot. In a container, the concept of "boot" is ambiguous. When a container restarts, it does not go through a full initialization sequence like a physical machine or a virtual machine. Instead, it simply resumes execution from the entrypoint. If the entrypoint is not configured to trigger systemd’s startup sequence, enabled services will not start.

A user on the Docker forums highlighted this issue, noting that after creating a systemd service inside a Docker container, the service did not start automatically upon container restart, even after running systemctl enable ServiceName. The user was attempting to run PostgreSQL replication and failover components using Bitnami images, specifically bitnami/postgresql-repmgr:14 and bitnami/pgpool:4. They wanted to install the pgagent process inside the PostgreSQL container and decided to use systemd to manage it. However, when they tried to start pgagent via the entrypoint script, it failed, leading them to rely on systemd. The community response was clear: systemd is not designed for containers, and the recommended approach is to manage the application lifecycle directly in the container’s entrypoint script.

The technical reason behind this limitation lies in how Docker containers are designed. A container is essentially a process wrapper. When a container starts, it executes a single process defined in the ENTRYPOINT or CMD instruction of the Dockerfile. If this process exits, the container stops. Systemd, on the other hand, is an init system designed to manage multiple processes and handle system startup. When systemd runs as the PID 1 process inside a container, it acts as the main process, and other services are managed as child processes. However, if the container restarts, systemd may not re-initialize properly, or it may not re-read the enabled service configurations, leading to services failing to start.

To address this, users who insist on running systemd inside a container must ensure that the container’s entrypoint is configured to properly initialize systemd and that the container is configured to restart automatically using Docker’s --restart policies. For example, adding --restart=always to the docker run command ensures that the container restarts if it crashes, but it does not guarantee that systemd will re-enable services. Advanced configurations may involve using custom entrypoint scripts that mimic the behavior of a full system boot, including reloading systemd configurations and starting enabled services.

Managing Docker Containers with Systemd on the Host

While running systemd inside a container is a niche solution for specific legacy or complex use cases, a more common and widely accepted practice is to use the host’s systemd to manage Docker containers. This approach treats Docker containers as first-class citizens in the system’s service management framework. By creating systemd service units that invoke Docker commands, administrators can leverage systemd’s powerful features such as dependency management, automatic restarts, logging integration, and status monitoring. This method is particularly useful in environments where Docker is used to run long-running services that need to be managed alongside traditional system services.

The basic principle behind this approach is to create a systemd service file that specifies the Docker commands required to start, stop, and manage the container. The service file defines the lifecycle of the container in terms that systemd understands. For example, a service file for running a Redis container might look like this:

```ini
[Unit]
Description=Redis Key Value store
Requires=docker.service

[Service]
ExecStartPre=/bin/sleep 1
ExecStartPre=/bin/docker pull redis:latest
ExecStart=/bin/docker run --restart=always --name=systemdredis -p=6379:6379 redis:latest
ExecStop=/bin/docker stop systemdredis
ExecStopPost=/bin/docker rm -f systemdredis
ExecReload=/bin/docker restart systemdredis

[Install]
WantedBy=multi-user.target
```

This service file is saved as /etc/systemd/system/redis.service. The [Unit] section provides metadata about the service, including a description and dependencies. The Requires=docker.service directive ensures that the Docker daemon is running before the Redis service is started. The [Service] section defines the commands to be executed at various stages of the service lifecycle. The ExecStartPre commands are executed before the main start command. In this case, a one-second sleep is used to ensure that Docker is fully ready, followed by a docker pull to ensure the latest version of the Redis image is available. The ExecStart command runs the container with the --restart=always flag, which instructs Docker to restart the container if it crashes. The -p 6379:6379 flag maps the container’s port 6379 to the host’s port 6379, allowing external applications to connect to Redis via 127.0.0.1:6379. The ExecStop command stops the container gracefully, and the ExecStopPost command removes the container to ensure a clean state for the next start. The ExecReload command restarts the container, which can be useful for applying configuration changes. The [Install] section specifies that the service should be started in the multi-user target, which is the standard runlevel for most Linux systems.

After creating the service file, the systemd daemon must be reloaded to recognize the new service. This is done by running:

docker systemctl daemon-reload

Once reloaded, the service can be started, stopped, and enabled using standard systemd commands:

docker systemctl start redis systemctl stop redis systemctl enable redis

Enabling the service ensures that it starts automatically on boot, integrating the Docker container into the host’s standard service management workflow. This approach provides several advantages. First, it allows administrators to use familiar tools and workflows to manage containers. Second, it leverages systemd’s robust logging capabilities, integrating container logs into the system journal for centralized monitoring and troubleshooting. Third, it provides a clear way to manage dependencies between containers and other system services. For example, if a web application container depends on a database container, the systemd service files can be configured to reflect this dependency, ensuring that the database is started before the web application.

However, this approach also has limitations. One of the primary challenges is that systemd service files are static configuration files. If the Docker image is updated, the service file must be modified to reflect the new image tag. This can lead to configuration drift and maintenance overhead. Additionally, systemd does not natively understand Docker-specific concepts such as networks, volumes, or compose files. While simple containers can be managed effectively, complex multi-container applications defined by Docker Compose files require a more sophisticated approach.

Integrating Docker Compose with Systemd Templates

For applications that consist of multiple interconnected containers, Docker Compose is the standard tool for defining and running multi-container Docker applications. Compose files allow users to define services, networks, and volumes in a single YAML file, simplifying the management of complex applications. However, integrating Docker Compose with systemd requires a different strategy than managing individual containers. Instead of creating a separate service file for each container, a template service file can be created to manage any Docker Compose project.

A template service file allows systemd to instantiate multiple services from a single configuration file by using a placeholder for the service name. For example, a template service file for managing Docker Compose projects might be placed at /etc/systemd/system/[email protected]. The @ symbol indicates that this is a template, and the %i specifier is used to insert the instance name (the part after the @) into the configuration. The service file might look like this:

```ini
[Unit]
Description=%i Docker service
After=docker.service docker-traefik-network.service
Requires=docker.service docker-traefik-network.service
AssertPathExists=/usr/local/etc/containers/%i/docker-compose.yml

[Service]
TimeoutStartSec=0
Restart=always
WorkingDirectory=/usr/local/etc/containers/%i
ExecStart=/usr/bin/docker-compose --file /usr/local/etc/containers/%i/docker-compose.yml --project-name traefik up --force-recreate
ExecStop=/usr/bin/docker-compose --file /usr/local/etc/containers/%i/docker-compose.yml --project-name traefik down

[Install]
WantedBy=multi-user.target
```

This template service expects a Docker Compose file to exist in the /usr/local/etc/containers/<instance_name>/ directory. The AssertPathExists directive ensures that the service will not start if the Compose file is missing. The ExecStart command runs docker-compose up with the --force-recreate flag, which ensures that containers are recreated if the Compose file has changed. The ExecStop command runs docker-compose down to stop and remove the containers. The --project-name flag is used to set the project name, which can be useful for organizing containers and networks.

This approach allows administrators to manage multiple Docker Compose projects using a single template service file. Each project is represented by a separate instance of the service, such as [email protected] or [email protected]. To enable a service, the administrator simply enables the specific instance:

docker systemctl enable [email protected]

This method provides a scalable and maintainable way to manage multi-container applications with systemd. It also allows for the integration of other systemd features, such as dependency management. For example, the service file above requires the docker-traefik-network.service, which ensures that a Traefik network exists before the application containers are started. This is particularly useful for applications that rely on external networks or shared resources.

Systemd as a Docker Replacement

While systemd is commonly used to manage Docker containers, some engineers have explored using systemd as a direct replacement for Docker, particularly in environments where the overhead and complexity of Docker are seen as unnecessary. This approach, often referred to as "systemd-nspawn," involves using systemd’s systemd-nspawn command to run containers. systemd-nspawn is a lightweight container runtime that uses Linux namespaces and cgroups to isolate processes, similar to Docker, but without the need for a separate daemon or complex tooling.

The motivation for using systemd as a Docker replacement often stems from frustrations with Docker’s architecture and behavior. Some users have cited several annoyances with Docker and Docker Compose:

Compose acts like an alien on the target system. It has its own commands, configuration files, service management, and even managed networks, which can be confusing and inconsistent with the host operating system’s tools.
Docker creates its own firewall rules, which can conflict with existing firewall configurations such as ufw or nftables, leading to potential security gaps or connectivity issues.
Log management in Docker can be challenging, as logs are stored in JSON files in the /var/lib/docker/containers/ directory, making it difficult to integrate with standard system logging tools.
Docker can easily fill up disk space with unused images or run out of inodes, requiring regular maintenance and cleanup.
The always-running Docker daemon is a potential security issue, especially when running root containers, as it provides a persistent entry point for attackers.
The commercialization of Docker and related projects, such as the shift to the Business Source License (BSL) by companies like Elastic, HashiCorp, and Redis, has raised concerns about the future of open-source containerization. Systemd, being part of the Linux kernel ecosystem, is less likely to be affected by such licensing changes.

In a recent project, an engineer chose to replace Docker with systemd for running services on a single VM in an on-premise datacenter with limited internet access. The project requirements favored a lightweight, stable, and maintainable solution that did not rely on external dependencies or complex tooling. Systemd-nspawn provided a suitable alternative, allowing the engineer to run containers using standard systemd service files and commands.

However, running containers with systemd-nspawn is not without its challenges. One of the primary differences between Docker and systemd-nspawn is the handling of filesystem overlays. Docker adds a writable overlay filesystem by default, allowing containers to make changes to their filesystem without affecting the base image. Systemd-nspawn does not provide this feature by default, which can result in namespace mount errors when starting service units. This is because mountpoints must exist in the container filesystem before they can be used, and it is not possible to modify the parent directory to create one.

To fix this problem, empty directories or files must be created at the end of the Dockerfile or the container image configuration. This creates a direct dependency between the container image and the deployment method, which may not be desirable in all cases. However, for environments where the developer controls both the development and deployment processes, this trade-off is acceptable.

Another consideration is the ability to run ad-hoc commands inside the container. With Docker, this is done using docker exec. With systemd-nspawn, the equivalent command is systemd-nspawn. However, to make it work in all cases, a variety of flags must be used. For example, to spawn an interactive admin shell inside a container, the following command might be used:

docker systemd-nspawn --boot --machine=mycontainer --register=no

The --register=no flag is particularly important, as it prevents errors when the image is already mounted by a service. This flag ensures that the container is not registered with systemd’s machine database, avoiding conflicts with existing service units.

Practical Implementation and Best Practices

When deciding between running systemd inside Docker, managing Docker with systemd, or replacing Docker with systemd, several factors must be considered. The choice depends on the specific requirements of the project, the level of control needed over the container lifecycle, and the trade-offs between simplicity and flexibility.

For legacy applications or complex service stacks that rely heavily on systemd, running systemd inside a Docker container may be the only viable option. In such cases, it is crucial to use specialized base images like jrei/systemd-ubuntu and to configure the container with the appropriate cgroup mounts and tmpfs directories. Users must also be aware of the limitations of this approach, particularly the difficulty of ensuring that services start automatically on container restart. Custom entrypoint scripts and careful testing are essential to mitigate these issues.

For most modern applications, managing Docker containers with systemd on the host is the recommended approach. This method provides a familiar interface for service management, leverages systemd’s robust features, and maintains the isolation and portability benefits of Docker. When managing individual containers, simple systemd service files can be used to define the container’s lifecycle. For multi-container applications, template service files can be used to manage Docker Compose projects, providing a scalable and maintainable solution.

In environments where the overhead of Docker is seen as unnecessary, systemd-nspawn offers a lightweight alternative. This approach is particularly suitable for on-premise deployments, edge computing, or other scenarios where minimal dependencies and maximum control are desired. However, users must be prepared to handle filesystem overlay issues and ad-hoc command execution manually, as systemd-nspawn does not provide the same level of automation as Docker.

Regardless of the approach chosen, security should always be a top priority. Running containers with privileged mode or without proper isolation can expose the host system to significant risks. Similarly, relying on external networks or shared resources without proper dependency management can lead to complex and hard-to-debug issues. By understanding the strengths and limitations of each method, engineers can make informed decisions that align with their project requirements and organizational standards.

Conclusion

The relationship between systemd and Docker is complex and multifaceted, reflecting the evolving landscape of containerization and system management. Running systemd inside a Docker container is a specialized solution for specific use cases, requiring careful configuration and an understanding of its limitations. Managing Docker containers with systemd on the host is a widely adopted practice that leverages the strengths of both technologies, providing a familiar and robust interface for service management. Replacing Docker with systemd-nspawn is a niche but viable option for environments that prioritize simplicity and control over convenience and automation.

Each approach has its own set of trade-offs, and the choice depends on the specific requirements of the project. By exhaustively exploring these options, engineers can navigate the complexities of modern infrastructure and build systems that are reliable, secure, and maintainable. The key is to understand the underlying mechanisms, to anticipate potential pitfalls, and to choose the tool that best fits the job. Whether managing a single container or a complex multi-service application, the integration of systemd and Docker remains a critical skill for any DevOps professional.