<h1>systemd For Learner Drivers</h1>
 <p>A graphic to help out: <a href="https://hackerpublicradio.org/eps/hpr1672.svg">https://hackerpublicradio.org/eps/hpr1672.svg</a></p>
 <p>This is a subject that attracts controversy, but I am not today going to be controversial, I hope. Many Linux systems are moving away from SysV Init and adopting systemd instead; both Linuxes that I use, Fedora and Mint have adopted systemd, and I understand that Debian has now forked to allow both sides of the argument to have their way. I am not going to get into the debate here. My personal stance is that I see both sides of the argument and I will continue to perch on top of the fence until systemd either proves itself or fails to do so.</p>
 <p>In this HPR I am going to try to fill a gap that I have seen in the systemd discussion; that is - how to operate it. I am not an expert on systemd, I have just tried to work it, and in doing so I have fished around in my file system and in the documentation. If you want to know what I found, then keep on listening. By way of opening I will remind myself, and you also, what systemd is replacing.</p>
 <p>SysV initd works with runlevels, the most common being</p>
 <ul>
  <li>5 for graphical multiuser networked</li>
  <li>3 for cli multiuser networked</li>
  <li>1 for single user</li>
  <li>6 for reboot</li>
  <li>0 for halt</li>
 </ul>
 <p>In moving to a runlevel, unwanted services are shut down and wanted services are started up. For most users on most systems the most appropriate default runlevel is 5 giving multiuser, GUI &amp; networking. Services can be started and stopped on demand by inetd.</p>
 <p>systemd works differently. It has target units. For most users on most systems the most appropriate default target is the graphical.target, which does a similar thing to runlevel 5 . Units are configured by unit configuration files. These files may start other units and stop other units. They can impose sequence and dependancies. There is a lot of cascading going on, with unit launching unit launching unit. Units also can be started and stopped on demand by systemd.</p>
 <h2>Units</h2>
 <p>The term Unit refers to a resource that systemd is taking under its control. There are 12 different types of Unit.</p>
 <dl>
  <dt><strong>systemd.service</strong></dt>
  <dd>that starts/stops daemons</dd>
  <dt><strong>systemd.socket</strong></dt>
  <dd>activates network connections</dd>
  <dt><strong>systemd.device</strong></dt>
  <dd>activates kernel devices</dd>
  <dt><strong>systemd.mount</strong></dt>
  <dd>controls mount points</dd>
  <dt><strong>systemd.automount</strong></dt>
  <dd>provides on-demand mounting of file systems</dd>
  <dt><strong>systemd.swap</strong></dt>
  <dd>does for swap what systemd.mount does for filesystems</dd>
  <dt><strong>systemd.scope</strong></dt>
  <dd>starts/stops external processes</dd>
  <dt><strong>systemd.target</strong></dt>
  <dd>groups of services akin to init level 3, init level 5</dd>
  <dt><strong>systemd.snapshot</strong></dt>
  <dd>saves/restores the momentary state of other units</dd>
  <dt><strong>systemd.timer</strong></dt>
  <dd>triggers units based on date/time</dd>
  <dt><strong>systemd.path</strong></dt>
  <dd>trigger units based on changes in file system objects</dd>
  <dd>organises units in a hierarchical tree of cgroups, for resource management purposes</dd>
 </dl>
 <p>Units files called by systemd live in <code>/etc/systemd/system</code>. But these are symbolic links to the real ones stored in <code>/usr/lib/systemd/system</code></p>
 <p>There is a parallel <code>/etc/systemd/user</code> structure which does not seem to do anything on my computers, so I work for now like its not there.</p>
 <p>There is also a <code>/run/systemd/system</code> structure which appears to contain runtime configuration files with names like <code>session-xxxx.scope</code>. These are the unit type for external processes.</p>
 <table border="1" cellpadding="5" cellspacing="5" width="500">
  <caption>Table 1. Directory structure for systemd</caption>
  <tr>
   <th>Path</th>
   <th>Description</th>
  </tr>
  <tr>
   <td><code>/etc/systemd/system</code></td>
   <td>Local configuration</td>
  </tr>
  <tr>
   <td><code>/etc/systemd/user</code></td>
   <td>User configuration</td>
  </tr>
  <tr>
   <td><code>/run/systemd/system</code></td>
   <td>Runtime units</td>
  </tr>
  <tr>
   <td><code>/usr/lib/systemd/system</code></td>
   <td>Units of installed packages</td>
  </tr>
 </table>
 <h2>Directives</h2>
 <p>The next thing we need is Directives.</p>
 <p>The unit configuration files contain directives to start/stop a unit, and directives that cascade to other unit configuration files that start/stop dependant units. Directives may impose conditions on whether or when to call a unit. There are a whole bunch of different directives listed in man systemd.unit. These are a few.</p>
 <ul>
  <li><code>Requires=</code> list of units to start. If any required units fail then abort this one</li>
  <li><code>Conflicts=</code> list of units to stop</li>
  <li><code>After=</code> the order in which units will start</li>
  <li><code>Before=</code> the order in which units will start</li>
  <li><code>Wants=</code> list of units to start. If any fail just continue anyway</li>
 </ul>
 <p>As well wanted units listed by the WANTS directive, there may also be a 'wants' directory below the unit directory. So the unit conf file <code>/etc/systemd/system/default.target</code> will cause two further unit conf files to be read in from the <code>/etc/systemd/system/default.target.wants/</code> directory.</p>
 <p>Each required unit and wanted unit from the directives, as well as those in the wants directory are added to a job queue. If directives cascade to other unit files containing more directives then all of these dependences are also added to the job queue. A directive may start or stop another unit, or that change the detail of a job already in the queue. All directives ultimately cascade down to starting or stopping one of the base units in <code>/usr/lib/systemd/system</code>.</p>
 <p>To get a feel for how this all pans out in practice I will walk us through the cascade of unit files from bootup.</p>
 <h2>From Bootup</h2>
 <p>First, the default.taget is activated, which on my system is just a link to graphical.target</p>
 <h3>graphical.target</h3><pre>
[Unit]
Description=Graphical Interface
Documentation=man:systemd.special(7)
Requires=multi-user.target
After=multi-user.target
Conflicts=rescue.target
Wants=display-manager.service
AllowIsolate=yes
</pre><p>Cascades to</p>
 <ul>
  <li>start multi-user.target</li>
  <li>start display-manager.service</li>
  <li>stop rescue.target</li>
 </ul>
 <p>Also we have a wants directory <code>/etc/systemd/system/graphical.target.wants/</code> that</p>
 <ul>
  <li>starts accounts-daemon.service (for logging)</li>
  <li>starts rtkit-daemon.service (for realtime scheduling)</li>
 </ul>
 <h3>multi-user.target</h3>
 <p>graphical target cascaded to multi-user.target.</p><pre>
[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
</pre><p>Cascades to</p>
 <ul>
  <li>start basic.target</li>
  <li>stop rescue.service</li>
  <li>stop rescue.target (again)</li>
 </ul>
 <p>Also we have a wants directory <code>/etc/systemd/system/multi-user.target.wants/</code> that</p><pre>
- abrt-ccpp.service
- abrtd.service
- abrt-oops.service
- abrt-vmcore.service
- abrt-xorg.service
- atd.service
- auditd.service
- avahi-daemon.service
- chronyd.service
- crond.service
- cups.path
- irqbalance.service
- libvirtd.service
- mcelog.service
- mdmonitor.service
- NetworkManager.service
- nfs.target
- remote-fs.target
- rngd.service
- rpcbind.service
- rsyslog.service
- smartd.service
- vmtoolsd.service
</pre><h3>display-manager.service</h3>
 <p>graphical.target also cascaded to display-manager.service which is not present on F20 so I guess we don't need it.</p>
 <h3>basic.target</h3>
 <p>So multiuser.target cascaded to basic.target, which itself cascades to</p><pre>
- sysinit.target
- sockets.target
- timers.target
- paths.target
- slices.target
- firewalld.service
</pre><h3>sysinit.target</h3>
 <p>basic.target cascaded to sysinit.target which itself cascades to</p><pre>
- local-fs.target 
- swap.target
- dmraid-activation.service
- iscsi.service
- lvm2-monitor.service
- multipathd.service ( which looks like all the file system daemons)
</pre><h3>sockets.target</h3>
 <p>basic.target also cascaded to sockets.target which itself cascades to</p><pre>
- avahi-daemon.socket
- cups.socket
- dm-event.socket
- iscsid.socket
- iscsiuio.socket
- lvm2-lvmetad.socket
- rpcbind.socket
</pre><h3>End point</h3>
 <p>Now we start reaching the end-points of this trail at</p><pre>
- systemd.sockets
- systemd.timer
- systemd.path
- systemd.slice
- systemd-fstab-generator
</pre><p>By the time all of that has finished, if I type the command</p>
 <p><code># systemctl list-units --type service</code></p>
 <p>I see that 58 services are listed as running</p>
 <h2>Running and Configuring Services</h2>
 <p>If we are going to work with systemd we will have to give it instructions. In systemd parlance</p>
 <ul>
  <li>active = running, currently in use</li>
  <li>loaded = enabled, available for use</li>
 </ul>
 <p>These terms crop up in the output from commands</p>
 <p>Many instructions are given to systemd by the systemctl command.</p>
 <p>Now to compare line up some common SysV init tasks with their systemd equivalent</p>
 <table border="1" cellpadding="5" cellspacing="5" width="500">
  <caption>Table 2. SysV init commands and their systemd equivalents</caption>
  <tr>
   <th>command</th>
   <th>SysV Init</th>
   <th>systemd</th>
  </tr>
  <tr>
   <td>Check status</td>
   <td><code># service bluetooth status</code></td>
   <td><code># systemctl status bluetooth</code></td>
  </tr>
  <tr>
   <td>Start</td>
   <td><code># service bluetooth start</code></td>
   <td><code># systemctl start bluetooth</code></td>
  </tr>
  <tr>
   <td>Stop</td>
   <td><code># service bluetooth stop</code></td>
   <td><code># systemctl stop bluetooth</code></td>
  </tr>
  <tr>
   <td>Enable</td>
   <td><code># chkconfig --level 35 ntpd on</code></td>
   <td><code># systemctl enable ntpd</code></td>
  </tr>
  <tr>
   <td>Disable</td>
   <td><code># chkconfig --level 35 ntpd off</code></td>
   <td><code># systemctl disable ntpd</code></td>
  </tr>
 </table>
 <h2>Journalctl Logging</h2>
 <p>Much has been said about the desirability or otherwise of binary logs, but systemd gives us these so we had better know what to do with them.</p>
 <p>Journal instructions are given to systemd by the journalctl command</p>
 <dl>
  <dt>To view all log entries in one go. This is verbose, mine came out at ~9000 lines</dt>
  <dd><code># journalctl</code></dd>
  <dt>To view from a specific date</dt>
  <dd><code># journalctl --since="2014-05-07"</code></dd>
  <dt>To view kernel logs</dt>
  <dd><code># journalctl -k</code></dd>
  <dt>To follow a log in realtime ... and then to close</dt>
  <dd><code># journalctl -f<br /> ...<br /> # ctl-c</code></dd>
  <dt>To view log entries associated with a given PID</dt>
  <dd><code># journalctl _PID=1</code></dd>
  <dt>To view log entries associated with a given service</dt>
  <dd><code># journatlctl -u bluetooth</code></dd>
 </dl>
 <h2>Interrogating the system</h2>
 <h3>More systemd information</h3>
 <dl>
  <dt>Get/Set system information. Works like uname, but is more verbose</dt>
  <dd><code># hostnamectl</code></dd>
  <dt>Get/Set timezone &amp; timedate info</dt>
  <dd><code># timedatectl</code></dd>
 </dl>
 <table border="1" cellpadding="5" cellspacing="5" width="800">
  <caption>Table 3. SysV init information and their systemd equivalents</caption>
  <tr>
   <th>SysV Init Info</th>
   <th width="250">SysV Init command</th>
   <th>systemd info</th>
   <th width="250">systemd command</th>
  </tr>
  <tr>
   <td>What services are available for init.d to manage</td>
   <td><code># ls /etc/init.d</code></td>
   <td>What service units are available for systemd to run</td>
   <td><code># systemctl list-units --type service --all</code></td>
  </tr>
  <tr>
   <td>What services are configured to be run by init.d for each run level</td>
   <td><code># chkconfig --list</code></td>
   <td>What service units are currently active</td>
   <td><code># systemctl list-units --type service</code></td>
  </tr>
 </table>
 <h2>References</h2>
 <ul>
  <li>man pages for systemd, systemd.unit, systemctl, journalctl.<br /> Plus a bunch of other man pages that are SEE ALSO in these man pages.</li>
  <li><a href="https://www.freedesktop.org/wiki/Software/systemd/">systemd homepage</a></li>
  <li><a href="https://blog.dmcip.com/2014/05/an-introduction-to-systemd/">A useful systemd blog posting, by Adam Dean of Asterisk Telephones</a></li>
  <li><a href="https://www.youtube.com/watch?v=BS8cdl7dOUI">A useful youtube vid by George Magklaras, UNiversity of Oslo, Norway</a></li>
 </ul>