Diagnose a Failed Linux systemd Service

When a Linux service fails to start, systemctl shows its state and systemd records the service’s output and exit status in the journal. Follow the evidence from the failed unit to its recent logs, inspect the unit and its dependencies, then make a targeted correction and verify the recovery.

The examples use example.service; replace it with the actual unit name. These steps inspect configuration before restarting anything. For databases, queues, and other stateful services, follow the application’s recovery and change procedures before repeating start attempts.

---

Step 1: Identify the Failed Unit and Its State

01

List Failed Units and Inspect the Target

Initial Check

Start with the failed-unit list, then check the specific service. The status output includes whether the unit is loaded, its active state, the process exit result, and a small amount of recent journal output. Save this information before resetting a failed state or making changes.

systemctl --failed
systemctl status example.service --no-pager --full

❯ View Expected Console Output

● example.service - Example application
     Loaded: loaded (/etc/systemd/system/example.service; enabled)
     Active: failed (Result: exit-code)
    Process: 1234 ExecStart=/opt/example/bin/server (code=exited, status=1/FAILURE)

---

Step 2: Read the Service’s Journal Entries

02

Find the First Useful Error Message

Logs

Read this boot’s service log without a pager, then narrow it to the recent time window if needed. Look for the first application error before systemd’s final failure message; common causes include invalid configuration, unavailable dependencies, and permission errors.

sudo journalctl -b -u example.service --no-pager -n 100
sudo journalctl -u example.service --since "30 minutes ago" --no-pager

❯ View Expected Console Output

Oct 04 11:42:13 host server[1234]: Error: unable to read /etc/example/config.yml
Oct 04 11:42:13 host systemd[1]: example.service: Main process exited, code=exited, status=1/FAILURE

Figure 1: A failed unit’s status and journal identify a permission error reading its configuration file.

---

Step 3: Inspect the Effective Unit and Dependencies

03

Check the Unit File, Overrides, and Dependencies

Configuration

systemctl cat shows the main unit and any drop-ins, while systemctl show reports selected effective properties. Confirm the executable path, user, working directory, environment files, and required units exist. Unit files installed by packages usually live under /usr/lib/systemd/system or /lib/systemd/system; local overrides generally belong under /etc/systemd/system.

systemctl cat example.service
systemctl show example.service \
  -p FragmentPath -p DropInPaths -p User -p Group -p ExecStart -p WorkingDirectory
systemctl list-dependencies example.service

❯ View Expected Console Output

FragmentPath=/usr/lib/systemd/system/example.service
User=example
ExecStart={ path=/opt/example/bin/server ; argv[]=/opt/example/bin/server ... }

---

Step 4: Validate the Corrected Configuration

04

Check Unit Syntax and Application Inputs

Validation

After correcting a specific issue, validate the unit file if your systemd version provides systemd-analyze verify. Separately run the application’s own configuration check when available; unit syntax validation cannot confirm that the application config, credentials, network dependencies, or data are correct. If a unit file changed, reload systemd’s unit definitions before trying to start the service.

sudo systemd-analyze verify example.service
# Run the application's documented config-check command here, if available.
sudo systemctl daemon-reload

❯ View Expected Console Output

No unit-file errors reported.

---

Step 5: Start the Service and Confirm It Stays Healthy

05

Retry Once and Review the New Logs

Recovery

Once the cause has been corrected, start the unit and check its state and newest journal entries. Confirm the application is actually serving its expected function; an active systemd state only confirms that the process is running according to the unit. Avoid repeated restarts when the service is failing on persistent data or a dependency that remains unavailable.

sudo systemctl start example.service
systemctl is-active example.service
systemctl status example.service --no-pager --full
sudo journalctl -u example.service --since "5 minutes ago" --no-pager

❯ View Expected Console Output

active
Active: active (running)

Preserve the Failure Evidence

Capture the failed status and relevant journal entries before using systemctl reset-failed. Resetting clears the failed-state marker; it does not fix the underlying cause or remove the journal history.

For more detail, see the systemd project’s debugging guide and the systemctl manual.