My Home Lab 2020, part 7: Keeping containers' log in rotation with logrotate

Context

Today, a quick and very simple post about managing the logs of my home lab containers :) The goal is to ensure that logs file generated by docker are not growing in size too much as docker by default will keep storing all logs in a log file per container.

There are multiple ways of doing it, and I will use the simplest one by leveraging logrotate (already installed by default in most distribution!) :). As logrotate man page¹ says:

logrotate - rotates, compresses, and mails system logs

If you want a centralized log management solution where you can access all logs from containers in a single interface, with filter, search and sprinkles, then you should look at the ELK stack

Configuring Logrotate

First, we just need to tell logrotate to look at all log files generated by containers. By default, docker will generate log files in /var/lib/docker/containers/<containerID>/<containerID>-json.log. This means that we need to tell logrotate to check for all files finishing by .log in all subdirectory of /var/lib/docker/containers/. To do so, create a new file in logrotate configuration directory called docker (/etc/logrotate.d/docker) with the following:

    /var/lib/docker/containers/*/*.log {
      rotate 12
      daily
      compress
      size=100M
      missingok
      delaycompress
      copytruncate
    }

Explaination

• rotate 12: Keep 12 months of logs
• daily: Run rotation daily
• compress: compress log file with gzip
• delaycompress: Wait for next rotation to compress
• size=100M: Max file size 100M. So rotation happens daily or when log file reach 100M.
• missingok: Keep going in case of error
• copytruncate: Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one.

More information on logrotate documentation.

To test that everything works correctly:

    sudo logrotate -v /etc/logrotate.d/docker

The -v is for a more verbose output of the command. You should get a long output telling you it worked! You can check by looking at the log files with ls /var/lib/docker/containers/*/*.log*. You should see rotation log file now (name <containerID>-log-<number>.gz)

Reading logs

Logrotate is only helping with log file size to avoid having logs files of multiple GB that will pollute your disk. If you need to access the logs, you have multiple solution:

Use docker command line to see the logs of a swarm service:

    docker services logs -f servicename_containername

Or you can use docker manager UI to access them like portainer or swarmpit.

Finally, you could also use dozzle to access all logs from all containers in real time in once place. It is not made for log storage or search, it is simply displaying all containers logs in a single web page. As it can be very handy for debugging, I have a docker-compose.yml defining a dozzle swarm stack ready for it any time, so maybe have a look at it :)

Just FYI, this is the docker-compose file for dozzle:

    version: "3"

    services:
      dozzle:
        image: amir20/dozzle:latest
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
        ports:
          - 8080:8080 # To change
        deploy:
          placement:
            constraints:
              - node.role == manager

Nota: Same warning as usual, you should either build your own docker images and deeply inspect the build file of the community containers you are using! :).

Conclusion

As said above, this is really the simplest thing to do (but still important to avoid filling up your disk with huge uncompressed log files :) The next step if you want a centralized place, look at ELK or equivalent. I don’t need that level of complexity, so for me it does the job :) But evolving that later on is always an option.

You can follow all posts about my Home Lab on the dedicated page.

*[ELK]: ElasticSearch Logstach Kibana