Complete Beginner's Guide to Docker Logging
Explore and learn some interesting ways to make log management easier on Docker.
Knowledge on logging with Docker can be very helpful for day-to-day system administration activities, be it troubleshooting or basic monitoring. Without tracking logs on Docker, mitigating issues would be a lot more difficult when investigating anomalies.
This write-up explores some important information on that area to make it easier to understand how to manage Docker associated log files on your Linux system.
Let me start by exploring the most basic parts and gradually go deeper into some specifics.
How to view Docker logs
You can use the docker logs command to fetch whatever is going on with the service inside a running container.
The syntax is simple:
docker logs container_name_or_IDYou probably already know that you can use the docker ps command to view the running containers' name and ID.
Let's take a real-world example. I have deployed Nextcloud with Docker. The container is named nextcloud, unsurprisingly.
Now, to view the logs of the container named nextcloud, use:
docker logs nextcloudHere's a truncated version of the output of the above command:
172.18.0.2 - - [23/Jul/2021:19:36:09 +0000] "HEAD /.env HTTP/1.1" 302 1571 "-" "python-requests/2.26.0"
172.18.0.2 - - [23/Jul/2021:19:49:52 +0000] "HEAD /c99.php HTTP/1.1" 302 1565 "-" "python-requests/2.26.0"
172.18.0.2 - - [24/Jul/2021:16:25:23 +0000] "HEAD /backup.tar HTTP/1.1" 302 1571 "-" "python-requests/2.26.0"
172.18.0.2 - - [24/Jul/2021:16:25:24 +0000] "HEAD /backup.zip HTTP/1.1" 302 1569 "-" "python-requests/2.26.0"
172.18.0.2 - - [25/Jul/2021:20:36:01 +0000] "GET / HTTP/1.1" 302 1590 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36"
172.18.0.2 - - [25/Jul/2021:20:36:07 +0000] "GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1" 200 721 "-" "Nextcloud Server Crawler"
172.18.0.2 - - [25/Jul/2021:20:36:04 +0000] "GET /login HTTP/1.1" 200 7501 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36"
Too much of logs? The good thing is that there are various options that make viewing docker logs a lot easier, just like logs in Linux.
Tail Docker logs to view only certain number of lines
Docker logs command has --tail attribute that can be used in a fashion similar to the tail command.
Which means, you can use it to display only a certain number of lines of Docker logs from the end.
For example, to view the last 50 lines of a container, you can use:
docker logs --tail 50 container_name_or_IDYou may use the above command in this fashion as well
docker logs -n=50 container_name_or_IDViewing Docker logs in real-time on a live container
You can totally view the container logs in real time. To “follow” the logs, use the --follow or the -f attribute.
docker logs -f container_name_or_ID
When you run docker logs with follow option, you'll notice that the new log lines will be reported from the container as time progresses.
To exit, you can simply use the Ctrl+C key combination and drop back to the terminal prompt.
Most of the options in the docker logs sub-command can be combined to produce a desired result.
For example, docker logs -f will flood the screen with all the logs from the beginning. You wouldn't want that. What you can do is to combine both tail and follow options like this:
docker logs -f --tail 20 container_name_or_IDThe above command will show the last 20 lines of the logs and then follow the logs in real time.
When updating your app containers with scaling, the above command can be very helpful before you remove an older container.
View timestamp in Docker logs
If you also want to see the timestamp of the log entries, you can use the -t option.
docker logs -t container_name_or_IDThis can easily be combined with other options like --follow and --tail.
The command below will show the last 10 lines of the log file of a given container with timestamps attached to each of them
docker logs -n=10 -t container_name_or_ID
Viewing Docker logs in a specified time period
When you are investigating an issue, time can be a critical factor, in which case the --since and --until flags could prove very helpful.
For example, if you are investigating an issue that occurred in the past 24 hours, the following command will show the logged contents of whatever happened only during that time:
docker logs --since 1440m -t container_name_or_ID1440m in the above command denotes 24×60 minutes. Adding the -t or the --timestamps flag ensures the reported lines get labeled with a timestamp for you to comprehend incidents or errors even more easily. I'd suggest adding it when inquiring any container.
Similarly, if you require the log details after the first 24 hours of initiating deployment, the equivalent command would be:
docker logs --until 1440m -t nextcloud
Other than specifying time in minutes, the above two flags can also be used with a specific timestamp that -t generates. The format is like 2021-07-28T06:20:00.
docker logs --since 2021-07-28 -t container_name_or_IDThe above command will show the logs from 28th July 2021.
Accessing Docker logs from within the container
In some interesting use-cases, you may want to go for a hybrid approach, where you access the application specific logs from within the containers.
To do that, you enter the Docker container:
docker exec -it container_name_or_ID shAnd then you can use the regular Linux tools to get application related logs. One approach would be to get the process ID of the desired service using the ps command:
ps auxGet the details on this process:
cat /proc/PID_of_process/fd/1Docker system service logs
If you wish to see the logs of the Docker service itself on the host, you can use journalctl if you are on Ubuntu 16.04 or later:
sudo journalctl -u docker
Abhishek Prakash
Where are Docker logs stored?
You should find the docker logs in the /var/lib/docker/containers directory on the host system. This directory contains the log files related to all containers in their individual directories. You can identify the directories with the container ID.
I used the docker ps to find that 70f19fde9076 is the container ID of the Nextcloud container.
If I look into the directory where docker logs are stored:
avimanyu@localhost:~$ sudo ls -lh /var/lib/docker/containers
total 16K
drwx------ 4 root root 4.0K Jul 27 18:48 0efe12b28562c42619e533ad5f524d56740a7a3739d9e082c758bac95ae2a46f
drwx------ 4 root root 4.0K Jul 27 18:57 12c55f365f93ecb7f91e40bc130ddc2409216a61bbb244cd045a22b4513416d3
drwx------ 4 root root 4.0K Jul 27 18:58 70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e
drwx------ 4 root root 4.0K Jul 27 18:57 a436399ef16a3efc0a909b9c3f725938e595e0b8fd34644b13f28b2c9bcb4ed7
There are currently four containers running, and you can see that the third one matches the one we need to look into (it begins with 70f19fde9076).
If you check the contents of this directory, you can see that our log file sits right there!
avimanyu@localhost:~$ sudo ls -lh /var/lib/docker/containers/70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e
total 192K
-rw-r----- 1 root root 150K Jul 27 18:58 70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e-json.log
drwx------ 2 root root 4.0K Jul 19 17:10 checkpoints
-rw------- 1 root root 5.5K Jul 27 18:58 config.v2.json
-rw-r--r-- 1 root root 1.5K Jul 27 18:58 hostconfig.json
-rw-r--r-- 1 root root 13 Jul 27 18:58 hostname
-rw-r--r-- 1 root root 198 Jul 27 18:58 hosts
drwx------ 3 root root 4.0K Jul 19 17:10 mounts
-rw-r--r-- 1 root root 79 Jul 27 18:58 resolv.conf
-rw-r--r-- 1 root root 71 Jul 27 18:58 resolv.conf.hashTherefore, the corresponding log file is 70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e-json.log which is the same file that you were reading when I ran the docker logs command in the beginning.
Let me quickly verify that:
avimanyu@localhost:~$ sudo cat /var/lib/docker/containers/70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e/70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e-json.log
{"log":"172.18.0.2 - - [23/Jul/2021:19:36:09 +0000] \"HEAD /.env HTTP/1.1\" 302 1571 \"-\" \"python-requests/2.26.0\"\n","stream":"stdout","time":"2021-07-23T19:36:09.837857567Z"}
{"log":"172.18.0.2 - - [23/Jul/2021:19:49:52 +0000] \"HEAD /c99.php HTTP/1.1\" 302 1565 \"-\" \"python-requests/2.26.0\"\n","stream":"stdout","time":"2021-07-23T19:49:52.996108799Z"}
{"log":"172.18.0.2 - - [24/Jul/2021:16:25:23 +0000] \"HEAD /backup.tar HTTP/1.1\" 302 1571 \"-\" \"python-requests/2.26.0\"\n","stream":"stdout","time":"2021-07-24T16:25:23.445225166Z"}
{"log":"172.18.0.2 - - [24/Jul/2021:16:25:24 +0000] \"HEAD /backup.zip HTTP/1.1\" 302 1569 \"-\" \"python-requests/2.26.0\"\n","stream":"stdout","time":"2021-07-24T16:25:24.772881041Z"}
{"log":"172.18.0.2 - - [25/Jul/2021:20:36:01 +0000] \"GET / HTTP/1.1\" 302 1590 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36\"\n","stream":"stdout","time":"2021-07-25T20:36:03.678967877Z"}
{"log":"172.18.0.2 - - [25/Jul/2021:20:36:07 +0000] \"GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1\" 200 721 \"-\" \"Nextcloud Server Crawler\"\n","stream":"stdout","time":"2021-07-25T20:36:07.404618408Z"}
{"log":"172.18.0.2 - - [25/Jul/2021:20:36:04 +0000] \"GET /login HTTP/1.1\" 200 7501 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36\"\n","stream":"stdout","time":"2021-07-25T20:36:07.619020402Z"}
The log contents are the same, but since I'm reading the log file directly from the host, this has a lot of jargon compared to the docker logs command used earlier.
Enabling Log Rotation for Docker (JSON)
By default, for JSON file based logging, log rotation on Docker is disabled.
This may pose a problem with the disk space if the log files grows bigger in size. Our Docker based Ghost instance saw its log files going up to 20 GB in size. You would want to avoid such situation by enabling log rotation.
To enable log rotation for Docker, edit the /etc/docker/daemon.json file:
sudo nano /etc/docker/daemon.json
Append the following lines and save the file:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}
Restart Docker daemon:
sudo systemctl restart docker
The above setup is of course relevant to a generic Docker installation, which has JSON logging set as default. So do note that every other logging mechanism (listed below) has its own way of implementing rotation.
A quick word about logging drivers
Did you notice “json” in the log filename from earlier? That's because JSON File is the default logging driver on Docker.
Apart from JSON, there are many other logging mechanisms to choose from as well:
- Local file
- Logentries
- Graylog Extended Format(GELF)
- Syslog
- Amazon Cloudwatch Logs(AWS)
- Event Tracing in Windows(ETW)
- Fluentd
- Google Cloud Project Logs(GCP)
- Journald
- Splunk
- Logstash
For large-scale deployments, GELF, AWS, GCP and Fluentd are recommended as they implement a centralized approach. But for small scale, JSON does the job just fine, while Local, Syslog and Journald are also quite suitable. Syslog and Logstash are particularly very helpful if complex log analysis becomes a requirement. ETW is Windows specific, whereas Splunk is focused on remote logging.
Starting from Docker Engine 20.10, a new feature called “dual-logging” has been introduced that ensures executing the docker logs command as well as performing log rotation irrespective of the logging driver in effect.
Summary
In this article, I've begun with the basics of how to view the logs for any Docker container, both at the container and host-system level. I've also showed you how to locate the physical location of container-specific log files residing on your host-system. I briefly showed you how to check the logs for the Docker service on the host itself.
Later, I've discussed logging drivers with focus on the default mechanism and also highlighted the new dual logging feature.
I hope these pointers help you in your daily monitoring or learning activities. Any more thoughts and suggestions are more than welcome! Please share them in the comment section below.
DevOps Geek at Linux Handbook | Doctoral Researcher on GPU-based Bioinformatics & author of 'Hands-On GPU Computing with Python' | Strong Believer in the role of Linux & Decentralization in Science