Follow Journalctl Logs in Realtime to Monitor System and Services

The classic way of following a log file in real time is by using the tail -f command.

But you know that journalctl logs are a different entity and they are not like plain text files in syslogs.

So, how do you tail jounral logs then? It's simple. You use the similar -f option:

journalctl -f

The -f flag (short for "follow") works just like tail -f but for systemd journals. This shows the most recent log entries and keeps the output open, displaying new entries as they arrive.

Filter journalctl logs as you follow them

Now, just tailing logs as new entries are written can be overwhelming, specially on a busy Linux server. You are looking for particular information, so it is always better to filter them.

I understand that using grep is tempting here but let me tell you that journalctl has several built-in filters you can use to get relevant information from your logs.

Let's see various examples.

Filter journal logs by service

Want to watch only SSH logs? Use the -u (unit) flag! This is incredibly useful when troubleshooting specific services.

journalctl -u ssh -f

Unit should be a valid name. So if you are in doubt, list the systemd services and get the exact name.

Filter logs based on priority level

The -p option of journalctl allows you to filter logs of certain priority levels. These priority levels are:

• emerg (0): Emergency
• alert (1): Alert
• crit (2): Critical
• err (3): Error
• warning (4): Warning
• notice (5): Notice
• info (6): Info
• debug (7): Debug

Only show error messages and above.

journalctl -p err -f

The above command will show logs with priority 3 (error), 2, 1 and 0. You can also use numbers instead of mnemonics.

journalctl -p3 -f

Time-based filtering

You can tail logs in real time along with some older logs for context.

journalctl --since "1 hour ago" -f

The command above starts tailing from entries that occurred in the last hour. You can also use other time-based filters such as:

• --since "2023-01-01"
• --since "yesterday"
• --since "10 minutes ago"

Another example that watches both nginx and your application to help you troubleshoot:

journalctl -u nginx -u myapp -f --since "5 minutes ago"

Add initial lines for context

Another way to add context is by including last few lines.

journalctl -n 50 -f

This shows the last 50 lines and then follows new entries. Replace 50 with however many lines you want to see initially.

Filter kernel messages only

Focus solely on kernel messages using the -k flag.

journalctl -k -f

Combine multiple services and filters to narrow down your search

You can combine services as well as filters to add to be more specific with your log analysis.

journalctl -u service1 -u service2  -p error -f

For example, the command below will show warnings from apache2 and ssh in real time along with 20 lines from the earlier stored logs.

journalctl -u apache2 -u ssh -p warning -n 20 -f

This powerful combination shows:

• Only Apache logs (-u apache2)
• Warning level and above (-p warning)
• Last 20 lines initially (-n 20)
• Following new entries (-f)

Use grep for extra filtering

If you are lost, the classic grep command is always there. Combine with grep to filter for specific patterns.

journalctl -f | grep "ERROR"

📋 Summary

Effectively analyzing journal logs

Journalctl logs can take GBs of disk space. I advise cleaning them from time to time.

Tailing journalctl logs is an essential skill for any Linux administrator or developer. It's your window into what's happening on your system right now. Start with the basic -f flag, then gradually incorporate filters as you become more comfortable.

Remember: the key to effective log monitoring is knowing what to look for and filtering out the noise. Happy log hunting! 🕵️‍♂️