Skip to content

Log buffer sizing

Tomasz Kowalczewski edited this page Dec 20, 2020 · 11 revisions

Tjahzi uses ManyToOneRingBuffer from Agrona library for a Log buffer component. It is placed between logging threads and thread that sends log entries to Loki.

                [via 10kB thread local buffer]
                           │                                          
+-------------+    Log  <──┘                                                
│ Application │----------------+                                          
│  Thread 1   │                │                                          
+-------------+                │                                          
                               │                                          
       .                      \│/                                          
                          +----+---------------+       
       .      Log         │                    │       
          --------------->┤    *Log buffer*    ├--> ...
       .                  │                    │       
                          +----------+---------+       
       .                            /│\                                    
                                     │                                    
+-------------+      Log             │                                    
│ Application │----------------------+                                    
│  Thread N   │                                                           
+-------------+                                                           

This code makes sure that putting log entry onto the buffer has low overhead. For this reason it avoids using integer division for calculating record offset and uses simple and fast binary math. The downside is that its size must be a power of two (plus some additional bytes at the end used for bookkeeping, see TRAILER_LENGTH in RingBufferDescriptor).

Tjahzi can be configured to create Log buffer with sizes from 1MB to 1GB and will automatically add few bytes required by ManyToOneRingBuffer at the end.

If configured size is not a power of two it will log a warning and will use nearest power of two greater than provided value.

When configuring log4j2 appender it cen be set like this:

    <appenders>
        <Loki name="Loki" bufferSizeMegabytes="64">
...
        </Loki>
    </appenders>
Clone this wiki locally