Skip to content

Commit 6f05b9e

Browse files
committed
loki.write: implement sharding
Add capability to perform sharding with loki.write
1 parent ea1a478 commit 6f05b9e

File tree

7 files changed

+750
-701
lines changed

7 files changed

+750
-701
lines changed

docs/sources/reference/components/loki/loki.write.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ You can use the following blocks with `loki.write`:
4646
| `endpoint` > [`basic_auth`][basic_auth] | Configure `basic_auth` for authenticating to the endpoint. | no |
4747
| `endpoint` > [`oauth2`][oauth2] | Configure OAuth 2.0 for authenticating to the endpoint. | no |
4848
| `endpoint` > `oauth2` > [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
49-
| `endpoint` > [`queue_config`][queue_config] | When WAL is enabled, configures the queue client. | no |
49+
| `endpoint` > [`queue_config`][queue_config] | Configure the queue used for endpoint. | no |
5050
| `endpoint` > [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
5151
| [`wal`][wal] | Write-ahead log configuration. | no |
5252

@@ -104,8 +104,9 @@ The following arguments are supported:
104104
If no `tenant_id` is provided, the component assumes that the Loki instance at `endpoint` is running in single-tenant mode and no X-Scope-OrgID header is sent.
105105

106106
When multiple `endpoint` blocks are provided, the `loki.write` component creates a client for each.
107-
Received log entries are fanned-out to these clients in succession.
108-
That means that if one client is bottlenecked, it may impact the rest.
107+
Received log entries are fanned-out to these endpoints in succession. That means that if one endpint is bottlenecked, it may impact the rest.
108+
109+
Each endpoint has a _queue_ of batches to be sent. The `queue_config` block can be used to customize the behavior of this queue.
109110

110111
Endpoints can be named for easier identification in debug metrics by using the `name` argument. If the `name` argument isn't provided, a name is generated based on a hash of the endpoint settings.
111112

@@ -127,17 +128,18 @@ When `retry_on_http_429` is enabled, the retry mechanism is governed by the back
127128

128129
### `queue_config`
129130

130-
{{< docs/shared lookup="stability/experimental_feature.md" source="alloy" version="<ALLOY_VERSION>" >}}
131-
132-
The optional `queue_config` block configures, when WAL is enabled, how the underlying client queues batches of logs sent to Loki.
133-
Refer to [Write-Ahead block](#wal) for more information.
131+
The optional `queue_config` block configures how the endpoint queues batches of logs sent to Loki.
134132

135133
The following arguments are supported:
136134

137135
| Name | Type | Description | Default | Required |
138136
| --------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
139137
| `capacity` | `string` | Controls the size of the underlying send queue buffer. This setting should be considered a worst-case scenario of memory consumption, in which all enqueued batches are full. | `10MiB` | no |
140138
| `drain_timeout` | `duration` | Configures the maximum time the client can take to drain the send queue upon shutdown. During that time, it enqueues pending batches and drains the send queue sending each. | `"1m"` | no |
139+
| `min_shards` | `number` | Minimum amount of concurrent shards sending samples to the endpoint. | `1` | no |
140+
141+
Each endpoint manages a number of concurrent _shards_ which is responsible for sending a fraction of batches, number of shards are controlled with `min_shards` argument.
142+
Each shard has a queue of batches it keeps in memory, controlled with the `capacity` argument.
141143

142144
### `tls_config`
143145

internal/component/common/loki/client/config.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ type QueueConfig struct {
4747
// is the 1 MiB default, and a capacity of 100 MiB, the underlying buffered channel would buffer up to 100 batches.
4848
Capacity int
4949

50+
// MinShards is the minimum amount of concurrent shards sending batches to the endpoint.
51+
MinShards int
52+
5053
// DrainTimeout controls the maximum time that draining the send queue can take.
5154
DrainTimeout time.Duration
5255
}

0 commit comments

Comments
 (0)