Log file rotation

In production environments it is common to rotate log files to prevent them from taking too much disk space, and to support log file archival.

By default Sync gateway will write log statements to stderr.

Normally stderr is redirected to a log file by starting Sync Gateway with a command similar to the following:

$ sync_gateway sync_gateway.json 2>> sg_error.log

On linux the logrotate tool can be used to monitor log files and rotate them at fixed time intervals or when they reach a certain size. Below is an example of a logrotate configuration that will rotate the Sync Gateway log file once a day or if it reaches 10M in size.

/home/sync_gateway/logs/* { 
    daily 
    rotate 1 
    size 10M  
    delaycompress 
    compress 
    notifempty 
    missingok 
} 

The log rotation is achieved by renaming the log file with an appended timestamp. The idea is that Sync Gateway should recreate the default log file and start writing to it again. The problem is Sync Gateway will follow the renamed file and keep writing to it until Sync gateway is restarted.

By adding the copy truncate option to the logrotate configuration, the log file will be rotated by making a copy of the log file, and then truncating the file to 0 bytes.

/home/sync_gateway/logs/* { 
    daily 
    rotate 1 
    size 10M
    copytruncate
    delaycompress 
    compress 
    notifempty 
    missingok 
}

But using this approach there is a possibility of loosing log entries between the copy and the truncate, on a busy Sync Gateway instance or when verbose logging is configured the number of lost entries could be large.

In Sync Gateway 1.1.0 a new configuration option has been added that gives Sync Gateway control over the log file rather than relaying on stderr. To use this option call Sync Gateway as follows:

$ sync_gateway -logFilePath=sg_error.log sync_gateway.json 

If the option is not used then the existing stderr logging behaviour is used. When the option is passed Sync Gateway will attempt to open and write to the log file directly.

If a Sync Gateway process is sent the SIGHUP signal it will close the open log file and then reopen it, on linux the SIGHUP signal can be sent using the following command:

$ kill -HUP <SYNC_GATEWAY_PID>

This means Sync Gateway can support the log rotation where the log file is renamed with a timestamp and no log entries will be lost.