systemd: What to Know and

How to Transition

Tom Sorensen
Solutions Architect, Red Hat

2 RED HAT | Tom Sorensen

 Agenda

What is systemd?
What does systemd offer?
How does this affect me?
How do I transition to systemd?
Whats the next step?

4 RED HAT | Tom Sorensen

 What is systemd?

5 RED HAT | Tom Sorensen

 What is systemd?

systemd is a suite of system management daemons,

libraries, and utilities designed as a central management
and configuration platform for the Linux computer
operating system.
- From http://en.wikipedia.org/wiki/Systemd

6 RED HAT | Tom Sorensen

 *YAWN*

7 RED HAT | Tom Sorensen

 What is systemd?

Replaces init literally

init is now a symlink to systemd
First process to start and last to stop
Parent process of all other processes
Manages services and other resources

9 RED HAT | Tom Sorensen

 But I liked init...
init is really, really old
systemd adopted across all major Linux versions

Default in Fedora 15 as
of May 2011
Slackware in the
enterprise anyone?

10 RED HAT | Tom Sorensen

 What does systemd offer?

11 RED HAT | Tom Sorensen

 What does systemd offer?
Controls More than Services
Dependency Control
Service Activation
Faster Start Up and Shutdown
Improved Resource Management
Tracks and Restarts Services
Improved Logging, Debugging and Profiling
Plenty More...

12 RED HAT | Tom Sorensen

 Units
Controls more than services, it controls all resources on
the system - referred to as units.

Examples of Units:

Services Sockets

Targets and more..

Units are defined using Unit Files

Naming convention is name.unit_type

13 RED HAT | Tom Sorensen

 Dependencies
Since systemd controls more than services, it can also
more intelligently handle dependencies.

Define order and requirements for each unit

Example: nfs-lock.service
Requires=rpcbind.service network.target
After=network.target named.service rpcbind.service
Before=remote-fs-pre.target
No more semi-arbitrary 00-99 ASCII order loading

14 RED HAT | Tom Sorensen

 Common Unit File Options
Description=Unit description
Documentation=Documentation links
Requires=Additional units required
Before/After=Unit must start Before/After
Wants=Weaker Requires
Conflicts=Units cannot co-exist
WantedBy/RequiredBy=Set other units requirement

Lots of great detail in the RHEL 7 System Administrator's Guide

15 RED HAT | Tom Sorensen

 Service Activation
With dependency control for all resources, systemd can
also activate services on demand.

Start up services when needed

Save resources
Increased reliability
Transparent to client
Activation by Socket, Device, Path, Bus, and Timer
Recommended to convert xinetd services to units

16 RED HAT | Tom Sorensen

 Parallel, Not Serial
With defined dependencies we dont need to wait for
other services, sockets, file systems, etc that arent
needed for that specific unit.

Allows for Faster Start Up and Shutdown

Why does this matter?
Downtime matters Five 9s = 5.26 minutes per year
Cloud - Charges by runtime
Capacity on demand

17 RED HAT | Tom Sorensen

Improved Resource Management
Services are labeled with individual cgroups, including all
components and children. Resources are also divided
into Slices.

Services labeled and isolated with Cgroups

More control than nice alone
Can configure multiple instances for a single service
Can balance by shares or by hard limits

18 RED HAT | Tom Sorensen

 Kill/Restart Cleanly
With each service being tracked by individual cgroup,
systemd has an understanding of all related processes.

Tracked in the kernel

Knows all children
Dont need to rely on a potentially misbehaving
process to hopefully kill its children

19 RED HAT | Tom Sorensen

 Auto-Restarting
With more control over the whole service chain, systemd
can track the state of all services and make corrections
with less impact.

Its paying attention!

Reality: software does crash occasionally
Reduces need for manual intervention
Socket stays open, only lose that single transaction

20 RED HAT | Tom Sorensen

 Improved Logging
Every service logs, from when the initial RAM disk is
loaded to start the kernel to final shutdown of the system,
from all channels.

Dont need to wait for syslog to start

No More Losing STDERR and STDOUT
More detail than classic syslog alone
Improved debugging and profiling

21 RED HAT | Tom Sorensen

 (This is the end of the overview, well now get to the practical.)

22 RED HAT | Tom Sorensen

 How does this affect me?

23 RED HAT | Tom Sorensen

 How does this affect me?
File Locations
Targets vs Runlevels
Booting with systemd
Analyzing the Boot Process
Resetting the Root Password
Systemctl and Other Commands
Logging and journalctl

24 RED HAT | Tom Sorensen

 File Locations

Maintainer files: /usr/lib/systemd/system/

Administrator files: /etc/systemd/system/
Drop-in configs: /etc/systemd/system/<name.unit>.d/
Non-persistent, runtime data: /run/systemd/
Systemd journal, non-persistent: /run/log/journal

Note: unit files under /etc will take precedence over /usr

25 RED HAT | Tom Sorensen

 Targets are the new Runlevels
Targets are logical groupings of services and other units
that are started and stopped together.
Targets != Runlevels some equivalency
Traditional Runlevel Equivalent Target Symlink Target
Runlevel 0 poweroff.target runlevel0.target
Runlevel 1 rescue.target runlevel1.target
Runlevel 2 multi-user.target runlevel2.target
Runlevel 3 multi-user.target runlevel3.target
Runlevel 4 multi-user.target runlevel4.target
Runlevel 5 graphical.target runlevel5.target
Runlevel 6 reboot.target runlevel6.target

Targets can and will contain other targets

26 RED HAT | Tom Sorensen
 Common Targets
Common targets are similar to traditional runlevels.
Target Purpose
graphical.target Supports multiple users, graphical and text-based logins
multi-user.target Supports multiple users, text-based logins only
rescue.target Single user, local file systems mounted and basic system
initialization completed, networking is not activated
emergency.target Single user, root file system is mounted read-only, only a few
essential services are started, networking is not activated

Rescue and Emergency require root password!

27 RED HAT | Tom Sorensen

 Working with Targets
Viewing the default target:

Setting default target:

Default target is just a symlink:

28 RED HAT | Tom Sorensen

 Working with Targets
Changing currently loaded target:

Changing to rescue mode:

Changing to emergency mode without sending message:

29 RED HAT | Tom Sorensen

 Working with Targets
View list of currently loaded targets:

Results pipe to less by default: (can use no-pager)

Not a single runlevel, but a a collection of targets

30 RED HAT | Tom Sorensen
Shutting Down, Suspending, Etc.
Old Command New Command Description
halt systemctl halt Halts the system
poweroff systemctl poweroff Powers off the system
reboot systemctl reboot Restarts the system
pm-suspend systemctl suspend Suspends the system
pm-hibernate systemctl hibernate Hibernates the system
pm-suspend-hybrid systemctl hybrid-sleep Hibernates and
suspends the system

31 RED HAT | Tom Sorensen

 Boot Process Quick Review

BIOS/UEFI

Bootloader

Kernel and Initrd

System Initialization

32 RED HAT | Tom Sorensen

 Boot Process
Boot path determined by default.target

Lets track it backwards!

33 RED HAT | Tom Sorensen

 Boot Process
We see graphical.target requires multi-user.target and
should start after it. Lets look at multi-user.target...

Next, we see multi-user.target requires basic.target...

34 RED HAT | Tom Sorensen

 Boot Process
Which requires basic.target...

Which requires sysinit.target...

No more requires, but still some wants...

35 RED HAT | Tom Sorensen
 Boot Process

Which wants local-fs-pre.target and swap.target...

Once we look in these we see weve reached the end of

the line!

36 RED HAT | Tom Sorensen

 Boot Process
Once systemd has the beginning of the chain, it starts
processing at the earliest step, starting all of the
associated units in the specified order.

But, how does this work for starting individual services?

37 RED HAT | Tom Sorensen

 Boot Process Services/Units
Each target may also have a wants directory in addition
to the specific Wants and Requires directives in the
unit files.

Target Wants Directories:

/usr/lib/systemd/system/<name>.target.wants/
/etc/systemd/system/<name>.target.wants/

Files are symlinks to actual unit files

38 RED HAT | Tom Sorensen

 Boot Process - Services/Units
Example for multi-user.target.wants:

/usr/lib are static units for the OS, /etc are system specific

39 RED HAT | Tom Sorensen

 Analyzing Boot Process
Each unit is tracked during start up

Can see what is taking the longest to start at boot

40 RED HAT | Tom Sorensen
 Analyzing Boot Process
Analysis data can also be exported as SVG

41 RED HAT | Tom Sorensen

 Resetting Root Password
With the change from runlevels to targets, we also have a
change to the traditional single user mode.

Rescue and emergency targets require root password

Lets not learn this at 3am during a fire!

Resetting the Root Password of RHEL-7 / systemd

https://access.redhat.com/solutions/918283

42 RED HAT | Tom Sorensen

 Resetting Root Password
Step 1: Reboot your system
Step 2: When the GRUB2 boot loader menu appears,
press any key to interrupt, and press e to edit selection

43 RED HAT | Tom Sorensen

 Resetting Root Password
Step 3: Using the cursor keys, navigate to the end of the
line that starts with linux16
Step 4: Append rd.break to the end of the line and press
Ctrl-X to boot modified config

44 RED HAT | Tom Sorensen

 Resetting Root Password
Step 6: At the switch_root prompt, remount the /sysroot
file system read-write, then use chroot to enter the jail
Step 7: Reset the root password with passwd

45 RED HAT | Tom Sorensen

 Resetting Root Password
IMPORTANT! (Because youre using SELinux, right?)
Step 8: Instruct SELinux to relabel all files upon reboot.
System may not boot properly if skipped!
Step 9: Exit twice to initiate relabel and reboot

NOTE: The relabel may take quite a while to run before

the system becomes available.

46 RED HAT | Tom Sorensen

 systemctl vs service
Start, status, stop a service:

47 RED HAT | Tom Sorensen

 systemctl vs service

Note the different order of command option service

instead of command service option. This allows you to
run systemctl against multiple services at once.

Systemsctl start/stop will be silent unless there is an

error, in which case it will print to screen and log.

48 RED HAT | Tom Sorensen

 systemctl vs service
List services:

49 RED HAT | Tom Sorensen

 systemctl vs chkconfig
Enable, check, disable a service:

--list is not as important because you can see this in status

50 RED HAT | Tom Sorensen

 systemctl vs chkconfig
List all services:

51 RED HAT | Tom Sorensen

 systemctl
List all services by target as a dependency tree:

52 RED HAT | Tom Sorensen

 systemctl
Lots of options...

53 RED HAT | Tom Sorensen

 systemd-cgtop
Show top control groups by their resource usage:

May need to enable accounting perfect drop-in!

54 RED HAT | Tom Sorensen

 systemd-cgls
Recursively show control group contents:

55 RED HAT | Tom Sorensen

 systemd-*
Lots of new options...

56 RED HAT | Tom Sorensen

 journalctl
View basic logs:

Time stamps converted to system local time zone

All logged data is shown, including rotated logs
Non-persistent by default, can be preserved

57 RED HAT | Tom Sorensen

 journalctl
View most recent logs: (-f to follow)

Can force stdout/stderr to write to journal with

systemd-cat if wanted

59 RED HAT | Tom Sorensen

 journalctl
Specify form:

Other options:

60 RED HAT | Tom Sorensen

 journalctl
Filter by priority:

Filter by time and priority:

Advanced filtering by field, UID, unit, etc..

61 RED HAT | Tom Sorensen

 Systemd Journal
How to enable persistent logging for the systemd journal
https://access.redhat.com/solutions/696893

System Administrator's Guide

https://access.redhat.com/documentation/en-US/Red_Hat_Enter
prise_Linux/7/html/System_Administrators_Guide/s1-Using_the
_Journal.html

Lennart Poettering - The systemd Journal

https://www.youtube.com/watch?v=i4CACB7paLc

62 RED HAT | Tom Sorensen

 The future is not so future any more...
(RHEL 7 is here and systemd is part of every major Linux distro. Its time to start using it.)

63 RED HAT | Tom Sorensen

 How do I transition to
systemd?

64 RED HAT | Tom Sorensen

 How do I transition?

Start using the new commands

Learn more about cgroups
Convert your init scripts
Drop-in config overrides
Benefit from auto-restart
Address rc.local
Convert xinetd

65 RED HAT | Tom Sorensen

 Start using the new commands

Bash Completion is your friend!

service
# yum install bash-completion
chkconfig

systemd Cheat Sheet for Red Hat Enterprise Linux 7

https://access.redhat.com/articles/systemd-cheat-sheet
Common Administrative Commands in RHEL 5, 6, & 7

https://access.redhat.com/articles/1189123

66 RED HAT | Tom Sorensen

 Embrace cgroups

Already available in RHEL 6

More easily controlled in RHEL 7

Resource Management and Linux Containers Guide

https://access.redhat.com/documentation/en-US/Red_Hat_Enterpris
e_Linux/7/html/Resource_Management_and_Linux_Containers_Guide/i
ndex.html

67 RED HAT | Tom Sorensen

 Convert your init scripts

How to convert init scripts to systemd units files

https://access.redhat.com/solutions/912263

Step 1: Check out your simple init script

68 RED HAT | Tom Sorensen

 Convert your init scripts
Step 1, continued...

69 RED HAT | Tom Sorensen

 Convert your init scripts
Step 1, continued a bit more...

70 RED HAT | Tom Sorensen

 Convert your init scripts
Step 2: Modify/Install your RHEL 6 package on your
RHEL 7 system and ensure it works with a manual start
Step 3: Create a unit file on the RHEL 7 system

71 RED HAT | Tom Sorensen

 Convert your init scripts
Step 4: Notify systemd of the new unit file
Step 5: Start the service and verify it is running

72 RED HAT | Tom Sorensen

 Convert your init scripts
Step 6: Enabled the service to start on boot once youve
verified its functioning as expected

Step 7: Profit!

73 RED HAT | Tom Sorensen

 Drop-in Config Overrides

Drop-in configs allows you to override only specific

directives without overriding the entire unit file.

If you just copy the maintainer file from /usr/lib into /etc
and make changes your system will use only the
directives in that /etc unit file. This means that if the
maintainers improve or change something in the default
unit file your system will not ever see the new changes.

74 RED HAT | Tom Sorensen

 Drop-in Config Overrides
Step 1: Create /etc/systemd/system/<name.unit>.d/
directory

Step 2: Create a file containing only the directives youd

like to modify

Step 3: Reload systemd (systemd will warn you in status)

75 RED HAT | Tom Sorensen

 Drop-in Config Overrides
Step 4: Restart your service and check the new status

76 RED HAT | Tom Sorensen

 Auto-Restarting Processes
SSHD configured to auto-restart by default

If main process dies, it will restart after 42 seconds

77 RED HAT | Tom Sorensen

 Using rc.local
rc.local is supported, but no longer runs last
chmod +x /etc/rc.d/rc.local

Just a service - /usr/lib/systemd/system/rc-local.service

78 RED HAT | Tom Sorensen

 Convert xinetd
Example: Converting SSHD to Run On-Demand
Stolen from: http://0pointer.de/blog/projects/inetd.html
Step 1: Create an /etc/systemd/system/[email protected]
file like the following example

Note the @ after the sshd, this marks it as an instance

79 RED HAT | Tom Sorensen

 Convert xinetd
Step 2: Disable sshd.service
Step 3: Enable sshd.socket

Probably a good idea to make sure you have non-SSH

access to this machine as well. ;)

80 RED HAT | Tom Sorensen

 Convert xinetd
Step 4: Stop sshd.service and start sshd.socket

Step 5: Reconnect with SSH and check the new

connection

81 RED HAT | Tom Sorensen

 Reaching Maximum Capacity!

82 RED HAT | Tom Sorensen

 Review: systemd

Replaces init and does much more

It is here and its powerful
New boot and root password reset process
New commands and functionality
Plenty of great information and resources available

83 RED HAT | Tom Sorensen

 Whats the next step?

84 RED HAT | Tom Sorensen

Continue planning your transition

Define your goals Is it time to embrace SELinux?

Continue your research
RHEL 7 Migration Planning Guide
https://access.redhat.com/documentation/en-US/Red_Hat_Ente
rprise_Linux/7/html/Migration_Planning_Guide/
Navigating Red Hat Enterprise Linux 7
https://access.redhat.com/articles/887923

Talk to your TAM and SA!

85 RED HAT | Tom Sorensen

 Learn More
Lots of great info at https://access.redhat.com and
http://www.freedesktop.org/wiki/Software/systemd/

Overview of systemd for RHEL 7

https://access.redhat.com/articles/754933
Red Hat Summit 2013 - Getting Ready for systemd
https://access.redhat.com/videos/403833
RHEL 7 - systemd Service & Resource Management
https://access.redhat.com/videos/898503
Rethinking PID 1 by Lennart Poettering
http://0pointer.de/blog/projects/systemd.html

86 RED HAT | Tom Sorensen

 Learn More not just Red Hat!
Arch Wiki
https://wiki.archlinux.org/index.php/Systemd
Arch Why We Switched
https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530
Gentoo Documentation
https://wiki.gentoo.org/wiki/Systemd
OpenSUSE, Debian, Ubuntu also on or transitioning to systemd

87 RED HAT | Tom Sorensen

 Thank you!

88 RED HAT | Tom Sorensen