Log buffer sizing

Log buffer

Tjahzi uses ManyToOneRingBuffer from Agrona library for a Log buffer component. It is placed between logging threads and thread that sends log entries to Loki.

                [via 10kB thread local buffer]
                           │                                          
+-------------+    Log  <──┘                                                
│ Application │----------------+                                          
│  Thread 1   │                │                                          
+-------------+                │                                          
                               │                                          
       .                      \│/                                          
                          +----+---------------+       
       .      Log         │                    │       
          --------------->┤    *Log buffer*    ├--> ...
       .                  │                    │       
                          +----------+---------+       
       .                            /│\                                    
                                     │                                    
+-------------+      Log             │                                    
│ Application │----------------------+                                    
│  Thread N   │                                                           
+-------------+                                                           

This code makes sure that putting log entry onto the buffer has low overhead. For this reason it avoids using integer division for calculating record offset and uses simple and fast binary math. The downside is that its size must be a power of two (plus some additional bytes at the end used for bookkeeping, see TRAILER_LENGTH in RingBufferDescriptor).

Tjahzi can be configured to create Log buffer with sizes from 1MB to 1GB and will automatically add few bytes required by ManyToOneRingBuffer at the end.

If configured size is not a power of two it will log a warning and will use nearest power of two greater than provided value.

When configuring log4j2 appender it cen be set like this:

    <appenders>
        <Loki name="Loki" bufferSizeMegabytes="64">
...
        </Loki>
    </appenders>

with default value of 32MB.

Note on thread local buffer (log4j)

Thread that processes application logic and wants to log a message will first enter code that formats the message using appropriate log4j2 pattern class and then will serialise that message to a thread local buffer. That buffer is copied onto shared log buffer that was described here.

The default size of that thread local buffer is 10kb. This is the maximum size of a log message, larger messages will be fragmented into multiples of 10kb - no data will be lost. The rationale is that there can be many threads logging and with thread local buffers and we do not want it to add up. Also 10kb per log line "should be enough for everyone". In case you disagree there is a configuration option (maxLogLineSizeKilobytes) for changing size of that buffer. The buffer is deliberately not autosizing so that large (malicious or erroneous) message cannot DoS the appender and the whole application.

Note on thread local buffer (Logback)

Mechanism is the sam as with log4j but size of the buffer is controlled by system property tjahzi.logback.layout.encoder.bufferSize with default of 10KiB.