Learn about the Linux command which can help you troubleshoot why apps and services fail to start.
If you’ve used any modern Linux distribution, chances are good you have become familiar with systemd. You might have also happened upon a systemd issue when an app or service refuses to start. When that’s the case, you might have found yourself in a situation where you’re not quite sure how to begin troubleshooting.
Fortunately for every Linux admin, there’s a built-in tool to help you with that. Said tool is journalctl. Journalctl is the utility used for querying the systemd logging utility, journald. With the help of journald and journalctl, you can begin the process of troubleshooting why a service is refusing to start. Journald also tracks logs to a specific boot. With this mechanism, you can compare system boots to see when a service was working properly versus when it wasn’t.
I want to show you how to use the journalctl command by way of a few examples. This should give you plenty of information to start troubleshooting your own systemd start issues.
What you’ll need
The only thing you’ll need to work with journalctl is a Linux distribution which makes use of systemd.
The basic command
Open a terminal window and issue the command journalctl. You should see all output from the systemd logs (Figure A).
Each log entry begins with the month, the day of the month, and the time. Following the date/time entries, you’ll see the system hostname. After the hostname, you’ll see the service associated with the entry and that service’s Process ID (PID). The last bit of the entry will be the actual journald information for the service. A single entry looks like:
Aug 21 07:59:22 bionicserver sshd: Protocol major versions differ for 192.168.1.50 port 58554: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 vs. SSH-1.99-OpenNMS_1.5
Scroll through enough of the output and you might come across an error (Figure B). Case in point, an error associated with VirtualBox indicating it cannot set config #1, which is error -32. Google that and you’ll find it is related to a specific external USB devices.
If you want to list out only the boot process and how services fared during the booting of the system, issue the command:
This will list entries of the boot journal for the most recent boot of the system (Figure C).
Say you want to see the journal output of the previous boot. If the current boot is 0, the previous boot would be -1. That command would be:
journalctl -b -1
The boot prior to that would be:
journalctl -b -2
Continue on like that, until you find the boot journal you’re looking for. That is a very handy way to compare your current boot (one that might have issues) to a previous boot instance where certain services failed to start.
If you’re not sure how many boots the system has experienced, issue the command:
You should see a list of boots which includes all of the information you need to know which boot journal you should view (Figure D).
Viewing by time
You can also use journalctl to view log entries by time. Say, for instance, you know an issue occurred within the last hour. To view all entries within that time frame, issue the command:
journalctl --since "1 hour ago"
You should see every journald entry from within the last 60 minutes (Figure E).
You can use the —since option to get very specific, using the date format “YYYY-MM-DD HH:MM:SS”
Viewing by service (unit)
If you know of a service that’s having issues, you don’t have to worry about combing through the entire journal entry. For instance, if OpenNMS isn’t starting, issue the command:
journalctl -u opennms.service
The above command will only list out those journald entries relating to the OpenNMS service (Figure F).
Finally, if you don’t like the default output format, you can use the -o switch to change the format of the journalctl output. The available formatting options are:
- json shows each journal entry in json format
- json-pretty shows each log entry in an easier-to-read json format
- verbose shows very detailed information for each entry
- cat shows messages in very short form
- shortis is the default, syslog, output format
- short-monotonic is similar to short, but includes the time stamp value
And that’s all there is to using journalctl. With the help of this tool, your Linux admin job is made significantly easier. To learn more about journalctl, issue the command man journalctl to read the manual page.