Log management for docker made easy (EN)

Tl;dr: logspout and syslogng are used to centralize all log files within one directory. See configuration for docker-compose.

Log management with Docker

Docker is love, docker is life.

But even if docker is easy to try, docker is not pleasant to have in production. One of the most painful things to do when using docker in production is how to manage logs.

Default docker logging configuration is good for testing but issues quickly arise when heavily used:

• Logs are often displayed in stdout and not in a file (default for nginx, postgres, ...);
• Log storage is within a json file located in a dark path (by default: /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log);
• The json file can get pretty heavy without any notice;
• No rotation / flush is performed by default;
• If the container is removed, so are the logs.

Many alternatives can be found on the Internet, but none of them was suitable for our organization:

• Using an external solution such as Elastic Cloud, AWS, PaperTrail; (Not acceptable as we need to own our data)
• Using heavy tools such as Elastic Search, Splunk; (Not acceptable as those solutions are very heavy - thank you Java Virtual Machine - and we don't need such complex features)
• Changing logging driver of Docker; (Not acceptable as docker log would not work anymore, and would depend on previously mentioned platforms).

And more than that, a 24/7 open connection for transporting logs "live" is clearly unwanted.

Context and needs

I use Docker a lot. In fact, most of servers that I manage are just docker hosts, with all services containerised with docker-compose configurations.

In this context, the list of containers within an host looks like the following:

# docker ps

CONTAINER ID   IMAGE             COMMAND    NAMES
4f18222c1055   image_1:latest    "..."      service1_app_1
250604934bfc   image_2:latest    "..."      service1_db_1
af4ce12af9d7   image_3:latest    "..."      service2_proxy_1
af5f32dc542e   image_4:latest    "..."      service3_app_1
2bff87b43ad8   image_5:latest    "..."      service4_front_1
a5a59dec1508   image_6:latest    "..."      service4_db_1

The purpose of this article is to get in a single folder (let say /logs/) all the logs from all services. Ideally, on a file per service, with a timestamp in the filename and automatic log rotation.

Solution used

In order to implement that, the following tools will be used:

• logspout is a tool that can retrieve stdout from containers and then push it toward a syslog service;
• syslogng is an implementation of syslog, and can write all retrieved logs in files.

logspout

As explained previously, logspout is a tool that can retrieve stdout from containers.

In order to do that, logspout will need access to the docker socket from the host (usually located in /var/run/docker.sock). Access to this file socket is very sensitive, so the access should be given with caution.

It is not detailed in the (poor) documentation, but it is strongly advised to give a read-only access to the socket by adding an :ro flag to the volume option.

The syslog service will be in simple UDP (as it will not be exposed on Internet, but only on internal docker network), so the command will simply be syslog+udp://syslogng:514.

syslog-ng

As explained previously, syslog-ng is an implementation of syslog. Its main purpose is to retrieve data sent by logspout and write it into files.

By default, syslog-ng will listen to 601/TCP, 514/UDP and 6514/TCP (for TLS, but will not work as no certificate are provided). The 514/UDP service will be used.

Logs files will be written in the /var/log/syslogng/ directory, and filenames will follow the {SERVICE}.{CONTAINER_ID}-{DATE}.log pattern.

In order to do this, the related configuration is file("/var/log/syslogng/${PROGRAM}.${HOST}-${YEAR}.${MONTH}.${DAY}.log")

Implementation

Configuration

In order to manage this feature with docker-compose, logspout and syslog-ng will be configured in the same docker-compose.yml file:

version: '3'
services:
  logspout:
    image: gliderlabs/logspout:latest
    restart: always
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    command: syslog+udp://syslogng:514
  syslogng:
    image: balabit/syslog-ng:latest
    restart: always
    command: -F --no-caps
    volumes:
      - "/logs/:/var/log/syslogng/"
      - "./syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf"

The syslog-ng configuration is in a syslog-ng.conf in the same directory :

@version: 3.19
@include "scl.conf"
source s_network {
  default-network-drivers();
};
destination d_local {
  file("/var/log/syslogng/${PROGRAM}.${HOST}-${YEAR}.${MONTH}.${DAY}.log");
};
log {
  source(s_network);
  destination(d_local);
};

And then, to create the logs directory: mkdir /logs/.

Finally, docker-compose up -d to launch everything, and wait a bit, in order to generate some logs.

Results

# ls /logs/
service1_app_1.4f18222c1055-2019.04.22.log         service1_db_1.250604934bfc-2019.04.22.log
service2_proxy_1.af4ce12af9d7-2019.04.22.log       service3_app_1.af5f32dc542e-2019.04.22.log
service4_front_1.2bff87b43ad8-2019.04.22.log       service4_db_1.a5a59dec1508-2019.04.22.log

Victory !

The following has been done:

• All logs are centralized in a clean directory on the host;
• No new service is needed, and there is no external network footprint of the operation;
• No change is needed on existing services, the configuration is clean and automatic.

However, this is not a total victory we don't have log flushing :-(

The documentation states that log rotation / flushing is not provided by syslog-ng, however, it can easily be implemented with two crons:

• find /var/log/syslogng/ -name "*.log" -daystart -ctime +2 -type f -exec gzip {} \; to compress 2 days old logs;
• find /var/log/syslogng/ -name "*.gz" -daystart -ctime +14 -type f -exec rm {} \; to remove 14 days old logs

This technique does not purge logs created by docker within the /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log file, so docker configuration must be changed by editing (or creating) the /etc/docker/daemon.json file with the following content:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  }
}

Also, I strongly advise you to backup those logs, but that is another issue... for another time maybe :-)

Comments