KB
Monitoring & Troubleshooting

Log Analysis with journalctl and rsyslog

6 min read1222 words20 code blocks

At a glance#

  • Purpose: Find, filter and interpret system and application logs during troubleshooting and incident response.
  • Applies to: Any systemd-based Linux distribution.
  • Risk: None for reading; Low when changing retention settings.
  • Time: 15–45 minutes.

Overview#

Modern Linux keeps logs in two places, and knowing which to search saves a lot of time:

systemd journalrsyslog
Location/var/log/journal/ (binary)/var/log/*.log (plain text)
Read withjournalctlgrep, less, tail
HoldsEverything systemd captures — services, kernel, bootWhat rsyslog is configured to write
Survives rebootOnly if persistent storage is enabledYes

Most distributions run both. The journal is authoritative for services; the text files are easier to search with standard tools and are what gets shipped to a central collector.

journalctl#

Basics#

bash
# everything, newest last
sudo journalctl

# follow live
sudo journalctl -f

# last 100 lines
sudo journalctl -n 100

# this boot only
sudo journalctl -b

# the previous boot - essential after an unexplained reboot
sudo journalctl -b -1

# list known boots
sudo journalctl --list-boots

journalctl -b -1 is the first command to run after a server rebooted unexpectedly. It shows what the machine was doing immediately before it went down.

Filtering#

bash
# by service
sudo journalctl -u nginx
sudo journalctl -u nginx -u php-fpm        # several at once

# by priority
sudo journalctl -p err                     # errors and worse
sudo journalctl -p warning..err            # a range

# by time
sudo journalctl --since "1 hour ago"
sudo journalctl --since "2026-08-11 09:00" --until "2026-08-11 10:30"
sudo journalctl --since today
sudo journalctl --since yesterday --until today

# kernel messages only
sudo journalctl -k

# by process, user or executable
sudo journalctl _PID=1234
sudo journalctl _UID=1000
sudo journalctl /usr/sbin/sshd

Priority levels, lowest number is most severe:

LevelNameMeaning
0emergSystem unusable
1alertImmediate action required
2critCritical failure
3errError
4warningWarning
5noticeNormal but significant
6infoInformational
7debugDebug detail

Combining filters#

The real value is in narrowing quickly:

bash
# errors from one service in the last hour
sudo journalctl -u nginx -p err --since "1 hour ago"

# everything around an incident window, no pager, for copying into a ticket
sudo journalctl --since "2026-08-11 14:00" --until "2026-08-11 14:30" --no-pager

# output as JSON for scripting
sudo journalctl -u nginx -o json-pretty -n 5

Useful output formats#

bash
sudo journalctl -o short-precise    # microsecond timestamps
sudo journalctl -o verbose          # every field, useful for finding filter keys
sudo journalctl -o cat              # message text only, no metadata

Journal persistence and size#

By default some distributions keep the journal in memory only, so it is lost on reboot — which is exactly when you need it.

Check:

bash
sudo journalctl --disk-usage
ls -d /var/log/journal 2>/dev/null || echo "journal is volatile - not persisted"

Make it persistent:

bash
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Cap the size so it cannot fill the disk:

bash
sudo nano /etc/systemd/journald.conf
ini
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
MaxRetentionSec=30day
bash
sudo systemctl restart systemd-journald

Manual cleanup:

bash
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=14d
Warning: An uncapped journal on a busy server will consume the root filesystem. Set SystemMaxUse as part of every server build.

Traditional log files#

Even with the journal, these remain important:

FileContents
/var/log/messages (RHEL) / /var/log/syslog (Debian)General system messages
/var/log/secure (RHEL) / /var/log/auth.log (Debian)Authentication, sudo, SSH
/var/log/cronCron execution
/var/log/maillogMail transport
/var/log/nginx/, /var/log/httpd/Web server access and errors
/var/log/audit/audit.logauditd records

Searching effectively#

bash
# failed SSH logins
sudo grep "Failed password" /var/log/secure | tail -20

# count failures by source IP - a brute force shows up immediately
sudo grep "Failed password" /var/log/secure | \
  grep -oE "from [0-9.]+" | awk '{print $2}' | sort | uniq -c | sort -rn | head

# successful logins
sudo grep "Accepted" /var/log/secure | tail -20

# sudo usage
sudo grep "sudo:" /var/log/secure | tail -20

# a time window
sudo awk '/Aug 11 14:00/,/Aug 11 14:30/' /var/log/messages

# follow several files at once
sudo tail -f /var/log/messages /var/log/secure

Compressed rotated logs#

bash
sudo zgrep "error" /var/log/messages-*.gz
sudo zcat /var/log/messages-20260801.gz | grep -i "oom"

Searching only the current file is a common mistake — if the incident was three days ago, it is in a rotated archive.

Log rotation#

bash
# global configuration
cat /etc/logrotate.conf

# per-service rules
ls /etc/logrotate.d/

# test a rule without applying it
sudo logrotate -d /etc/logrotate.d/nginx

# force a rotation
sudo logrotate -f /etc/logrotate.d/nginx

A custom rule for an application log:

bash
sudo nano /etc/logrotate.d/myapp
text
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}
Note: Without postrotate reloading the service, the application keeps writing to the deleted file handle. Disk space is never released and the new log stays empty. copytruncate is an alternative where the app cannot reopen its log.

Central log forwarding#

Logs on a compromised host cannot be trusted. Forward them off the machine:

bash
sudo nano /etc/rsyslog.d/50-forward.conf
text
# forward everything over TCP to the log collector
*.* @@logserver.example.local:514
bash
sudo systemctl restart rsyslog

@@ is TCP, a single @ is UDP. Use TCP so messages are not silently dropped.

Incident investigation sequence#

When something broke and you do not know why:

bash
# 1. anything failed?
systemctl --failed

# 2. errors across the system in the relevant window
sudo journalctl -p err --since "2 hours ago" --no-pager

# 3. did the machine reboot?
last reboot | head -5
uptime

# 4. what happened just before the last boot
sudo journalctl -b -1 -p err --no-pager | tail -50

# 5. kernel-level events - OOM, disk, hardware
sudo dmesg -T | grep -iE "error|fail|oom|i/o"

# 6. authentication activity
sudo grep -E "Failed|Accepted|sudo" /var/log/secure | tail -30

# 7. resource exhaustion at the time
sar -u -s 14:00:00 -e 15:00:00

Verification#

bash
# journal is persistent and capped
sudo journalctl --disk-usage
grep -E "Storage|SystemMaxUse" /etc/systemd/journald.conf

# rotation is working - expect dated, compressed files
ls -lh /var/log/nginx/

# forwarding reaching the collector
sudo logger "test message from $(hostname)"
# then confirm it arrived on the log server

Troubleshooting#

SymptomCause and fix
journalctl shows nothing after rebootJournal is volatile. Create /var/log/journal and restart journald.
No journal files were foundNot running journald, or permissions. Add your user to systemd-journal.
Disk filling with logsNo SystemMaxUse set, or rotation broken. Cap it and check logrotate.
Log file exists but stays emptyApplication writing to a deleted handle after rotation. Add a postrotate reload.
df and du disagree in /var/logDeleted log held open. Find with lsof +L1.
Timestamps in the wrong timezoneJournal stores UTC. Use journalctl --utc, or fix with timedatectl.
Cannot find an event from last weekIt is in a rotated archive. Use zgrep.
Rotation not runninglogrotate.timer disabled. Check systemctl status logrotate.timer.