Skip to content

systemd & journald

Scope: the course page's systemd section covers systemctl start/enable/status and a handful of journalctl flags — enough to run and check a service. It doesn't say why that service started when it did, what "boot" actually resolves to, how to schedule anything, or why last week's logs might be gone. This page fills in how units depend on and order around each other, what a target actually is (the boot path, not just multi-user.target as a magic word), timer units as the systemd-native alternative to cron, and journald's storage model — why logs sometimes vanish on reboot and how to make sure they don't.

TL;DR — in 30 seconds:

  • Wants=/Requires= answer whether a unit needs another; After=/Before= answer when it starts relative to it — a service that needs the network requires both an ordering and a pull-in line.
  • systemctl edit <unit> writes a drop-in on top of the vendor file instead of forking it — the next package update can't silently undo your change.
  • journald keeps logs in memory only, lost on reboot, until /var/log/journal/ exists on disk — that's the whole switch, not a config flag.

1. Dependencies and ordering — why a unit starts when it starts

A .service file's [Unit] section is mostly about its relationship to other units, not about the process it runs. Two separate questions get answered by two separate directive families:

  • Does it need the other unit at all? — Requires=, Wants=. Requires= is a hard dependency: if the required unit fails to start, this one fails too. Wants= is a soft dependency: the wanted unit is started alongside this one, but its failure doesn't block this one from starting.
  • In what order do they start, relative to each other? — After=, Before=. These say nothing about whether the other unit is needed — only about sequencing, if both are going to start anyway.
flowchart LR
    A["myapp.service\nWants=network-online.target\nAfter=network-online.target"]
    B["network-online.target"]
    B -->|"must start first"| A
    A -.->|"myapp still starts\neven if B fails\n(Wants, not Requires)"| C["running"]

This split is why a service that needs the network requires both lines: After=network-online.target alone only fixes ordering — without Wants= (or Requires=), systemd has no reason to pull that target in at all if nothing else did.

systemctl list-dependencies <unit> walks this graph for a single unit; systemd-analyze critical-chain shows which chain of After= relationships is actually on the boot's slowest path.

2. Targets — the boot path is a dependency graph, not a runlevel number

A .target is a synchronization point: a named unit with no process of its own, whose only job is to group other units together and be something they can depend on. Boot isn't a linear script — it's systemd resolving the dependency graph until default.target (usually a symlink to multi-user.target or graphical.target) is satisfied:

flowchart TB
    SYS["sysinit.target\n(mount, udev, cryptsetup)"] --> BASIC["basic.target\n(sockets, timers, paths ready)"]
    BASIC --> MULTI["multi-user.target\n(sshd, network services)"]
    MULTI --> GFX["graphical.target\n(display manager, if installed)"]
    MULTI --> DEFAULT["default.target\n(symlink to one of the above)"]

systemctl get-default / sudo systemctl set-default multi-user.target reads or changes that symlink — this is the systemd replacement for the old SysV runlevel file. A unit joins a target with WantedBy=multi-user.target in its [Install] section; systemctl enable is what actually creates the symlink that wires the unit into that target, which is why enable without --now changes what happens next boot but not the running system.

3. Editing units safely — drop-ins instead of forking the vendor file

Package-managed unit files live under /usr/lib/systemd/system/ and get overwritten on the next update. The safe way to change one setting is a drop-in override, not editing that file in place:

sudo systemctl edit sshd                    # opens an editor for a drop-in, creates the dir for you
sudo systemctl edit --full sshd             # edits a full local copy instead (rarely what you want)

systemctl edit writes to /etc/systemd/system/sshd.service.d/override.conf — a fragment that's merged on top of the vendor unit at daemon-reload time, so only your changed directives need to appear there. systemctl cat sshd shows the effective merged result (vendor file + every drop-in, in order), which is the fastest way to confirm what actually applies before debugging further.

systemctl mask <unit> goes a step further than disable — it symlinks the unit to /dev/null, so nothing (not even another unit's Wants=) can start it until systemctl unmask reverses it. Reach for mask when something keeps getting pulled back in by a dependency, not just started manually.

4. Timers — systemd's cron

A .timer unit triggers another unit (almost always the identically-named .service) on a schedule, without a separate cron daemon or crontab syntax to learn:

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup.service daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl enable --now backup.timer
systemctl list-timers                       # every timer, next/last run, which unit it triggers

The two scheduling styles cover most needs: OnCalendar= for wall-clock schedules (*-*-* 03:00:00, Mon,Wed,Fri 09:00), and OnBootSec= / OnUnitActiveSec= for relative schedules ("15 minutes after boot", "every 2 hours since this unit last ran") — closer to sleep-and-repeat than to a calendar. Persistent=true is what makes a missed run (host was off at 03:00) fire as soon as the host is back up, instead of silently waiting for the next scheduled time — the behavior most people actually want and cron never gave them for free. The advantage over a crontab line isn't just syntax: a timer's triggered service gets full systemd supervision (start/stop logged, failures visible in systemctl --failed, output captured by journald automatically) instead of a cron job's mail-or-silence result.

5. journald's storage model — why logs vanish, and how to keep them

By default on many RHEL installs, journald keeps logs only in memory / a small ring buffer, not on disk — a reboot loses everything (systemd calls this the auto storage mode). Whether logs persist is controlled by whether /var/log/journal/ exists:

flowchart LR
    APP["process writes to\nstdout/stderr or\nsd_journal API"] --> JD["journald"]
    JD -->|"/var/log/journal/ exists"| DISK["persistent storage\n(survives reboot)"]
    JD -->|"directory missing"| MEM["volatile ring buffer\nin /run\n(lost on reboot)"]
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

That's the whole switch — journald checks for the directory's existence, not a config flag, to decide where to write. Once persistent, size is bounded by /etc/systemd/journald.conf (SystemMaxUse=, SystemKeepFree=) or trimmed on demand with journalctl --vacuum-size=500M / journalctl --vacuum-time=7d — the same vacuum flag the course page already showed for time-based trimming.

Two more journald features are worth knowing even if you don't reach for them daily: structured fields — every entry carries metadata beyond the message text (_SYSTEMD_UNIT, _PID, _AUDIT_TYPE — the same field the course page's _AUDIT_TYPE=AVC query filters on), queryable with journalctl FIELD=value; and journalctl -o json-pretty, which dumps an entry's full field set when you need to see what's actually queryable. A plain shell script can log to the same system via systemd-cat mycommand, gaining a proper _SYSTEMD_UNIT-tagged entry without writing an integration by hand.

6. How this connects

The course page's systemctl status / enable --now / a handful of journalctl flags is what you'll type on almost every service, almost every day. This page is what to reach for once a unit's start order matters (a service that needs the network, or needs another service first), once you're writing a new .timer instead of reaching for cron, once a vendor unit needs one setting changed without forking it, or once "the logs are gone after reboot" needs an actual explanation instead of a guess.

Common gotchas

  • Adding After=network-online.target and assuming that's enough to guarantee the network is up. Fix: After= is ordering only — without Wants= (or Requires=) too, nothing pulls that target in at all unless something else already did.
  • Editing a vendor unit file directly under /usr/lib/systemd/system/. Fix: the next package update overwrites it silently — use systemctl edit <unit> for a drop-in override that survives updates.
  • Expecting systemctl enable (without --now) to start a service immediately. Fix: enable only wires the unit into its target for the next boot — pair it with --now, or run start separately, to affect the running system too.
  • Assuming logs from last week are just gone for good after "the logs vanished on reboot" once. Fix: check whether /var/log/journal/ exists — journald falls back to a volatile in-memory ring buffer when it doesn't, and creating the directory (plus a systemd-journald restart) makes logging persistent going forward.
Check yourself — a service has After=network-online.target but no Wants= or Requires= on it. What can go wrong?

Nothing pulls network-online.target in unless something else already needed it — After= only fixes ordering if the target is going to start anyway. Without a Wants=/Requires= line, the service can start before the network is even scheduled to come up.

Check yourself — systemctl enable backup.timer succeeds, but nothing runs at the scheduled time until you reboot. What did the command actually do, and how would you fix it without waiting for a reboot?

enable (without --now) only creates the symlink wiring the timer into its target for the next boot — it doesn't start it now. systemctl enable --now backup.timer (or systemctl start backup.timer separately) starts it on the running system immediately.

You can defend this when you can explain the difference between Wants=/Requires= (dependency) and After=/Before= (ordering) and why a unit can need both, what a target groups and how default.target determines the boot path, why systemctl edit writes a drop-in instead of touching the vendor file, what Persistent=true does for a timer that missed its scheduled run, and why journald logs disappear on reboot until /var/log/journal/ exists.