Context
Today, a quick and very simple post about managing the logs of my home lab containers :) The goal is to ensure that logs file generated by docker are not growing in size too much as docker by default will keep storing all logs in a log file per container.
There are multiple ways of doing it, and I will use the simplest one by leveraging logrotate (already installed by default in most distribution!) :). As logrotate man page1 says:
logrotate - rotates, compresses, and mails system logs
If you want a centralized log management solution where you can access all logs from containers in a single interface, with filter, search and sprinkles, then you should look at the ELK stack
Configuring Logrotate
First, we just need to tell logrotate to look at all log files generated
by containers. By default, docker will generate log files in
/var/lib/docker/containers/<containerID>/<containerID>-json.log
. This
means that we need to tell logrotate to check for all files finishing by
.log
in all subdirectory of /var/lib/docker/containers/
. To do so,
create a new file in logrotate configuration directory called docker
(/etc/logrotate.d/docker
) with the following:
/var/lib/docker/containers/*/*.log {
rotate 12
daily
compress
size=100M
missingok
delaycompress
copytruncate
}
Explaination
rotate 12
: Keep 12 months of logsdaily
: Run rotation dailycompress
: compress log file with gzipdelaycompress
: Wait for next rotation to compresssize=100M
: Max file size 100M. So rotation happens daily or when log file reach 100M.missingok
: Keep going in case of errorcopytruncate
: Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one.
More information on logrotate documentation.
To test that everything works correctly:
sudo logrotate -v /etc/logrotate.d/docker
The -v
is for a more verbose output of the command. You should get a
long output telling you it worked! You can check by looking at the log
files with ls /var/lib/docker/containers/*/*.log*
. You should see
rotation log file now (name <containerID>-log-<number>.gz
)
Reading logs
Logrotate is only helping with log file size to avoid having logs files of multiple GB that will pollute your disk. If you need to access the logs, you have multiple solution:
Use docker command line to see the logs of a swarm service:
docker services logs -f servicename_containername
Or you can use docker manager UI to access them like portainer or swarmpit.
Finally, you could also use dozzle to access all logs from all containers in real time in once place. It is not made for log storage or search, it is simply displaying all containers logs in a single web page. As it can be very handy for debugging, I have a docker-compose.yml defining a dozzle swarm stack ready for it any time, so maybe have a look at it :)
Just FYI, this is the docker-compose file for dozzle:
version: "3"
services:
dozzle:
image: amir20/dozzle:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- 8080:8080 # To change
deploy:
placement:
constraints:
- node.role == manager
Nota: Same warning as usual, you should either build your own docker images and deeply inspect the build file of the community containers you are using! :).
Conclusion
As said above, this is really the simplest thing to do (but still important to avoid filling up your disk with huge uncompressed log files :) The next step if you want a centralized place, look at ELK or equivalent. I don’t need that level of complexity, so for me it does the job :) But evolving that later on is always an option.
You can follow all posts about my Home Lab on the dedicated page.
*[ELK]: ElasticSearch Logstach Kibana