Skip to content

Feature/graylog server #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lfchris wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Feature/graylog server #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f77e15f
add possibility to add optional indices and inputs
lfchris Oct 30, 2025
572dfb8
fix tags
lfchris Oct 30, 2025
8d20fff
add missing things to readme
lfchris Oct 30, 2025
ea2ba15
refactor readme
lfchris Oct 31, 2025
f6f691e
refactor to use states
lfchris Nov 20, 2025
673791d
Update README.md
lfchris Nov 21, 2025
5d452e5
Update CHANGELOG.md
lfchris Nov 21, 2025
afa7f99
refactor(roles/graylog_server): remove dead code
NavidSassan Nov 27, 2025
1652521
fixes and improvements
lfchris Nov 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ Role:acme_sh

### Breaking Changes

Role:graylog_server
Copy link
Member

@NavidSassan NavidSassan Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong heading, this is not part of v4.0.0

* Removed creation a default index set and default input. Take a look at the README for the new intended method.
* Renamed `graylog_server__system_default_index_set` to `graylog_server__system_index_sets__*_var` & `graylog_server__system_inputs` to `graylog_server__system_inputs__*_var`
* Add state management for `graylog_server__system_index_sets__host_var` & `graylog_server__system_inputs__host_var`

Role:mariadb_server
* Removed support for EOL version 10.5
* Removed `mariadb_server__cnf_expire_logs_days__group_var` / `mariadb_server__cnf_expire_logs_days__host_var`, use `mariadb_server__cnf_binlog_expire_logs_seconds__group_var` / `mariadb_server__cnf_binlog_expire_logs_seconds__host_var` instead
Expand Down
86 changes: 48 additions & 38 deletions roles/graylog_server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,11 @@ If you use the ["Setup Graylog Server" Playbook](https://github.com/Linuxfabrik/

| Tag | What it does | Reload / Restart |
| --- | ------------ | ---------------- |
| `graylog_server` | Installs and configures Graylog Server | Restarts graylog-server.service |
| `graylog_server:configure` | Deploys the config files, manages the CA keystore, creates the system inputs and a default index set | Restarts graylog-server.service |
| `graylog_server:configure_defaults` | Only executed on demand. Configure Graylog Indices, Index Sets and Inputs. | - |
| `graylog_server:state` | Manages the state of the Graylog Server service | - |
| `graylog_server` | Installs and configures Graylog Server. | Restarts graylog-server.service |
| `graylog_server:configure` | Deploys the config files and manages the CA keystore. | Restarts graylog-server.service |
| `graylog_server:configure_system_inputs` | Only executed on demand. Configure Graylog Inputs. | - |
| `graylog_server:configure_system_index_sets` | Only executed on demand. Configure Graylog Index Sets. | - |
| `graylog_server:state` | Manages the state of the Graylog Server service. | - |


## Mandatory Role Variables
Expand Down Expand Up @@ -84,40 +85,46 @@ graylog_server__timezone: 'Europe/Zurich'

## Configure Graylog Indices, Index Sets and Inputs

Use the tag `graylog_server:configure_defaults` to configure Graylog indices, index sets and inputs.
Use the tags `graylog_server:configure_system_inputs` or `graylog_server:configure_system_index_sets` to configure Graylog index sets and inputs.

| Variable | Description | Default Value |
| -------- | ----------- | ------------- |
| `graylog_server__system_default_index_set` | Creates a default index set. Subkeys: <ul><li>`can_be_default`: Mandatory, boolean. Whether this index set can be default.</li><li>`creation_date`: Mandatory, date. Date in iso8601 format.</li><li>`description`: Mandatory, string. Description of index set.</li><li>`field_type_refresh_interval`: Mandatory, integer. Refresh interval in milliseconds.</li><li>`index_analyzer`: Mandatory, string. Elasticsearch/Opensearch analyzer for this index set.</li><li>`index_optimization_max_num_segments`: Mandatory, integer. Maximum number of segments per Elasticsearch/Opensearch index after optimization (force merge).</li><li>`index_optimization_disabled`: Mandatory, boolean. Whether Elasticsearch/Opensearch index optimization (force merge) after rotation is disabled.</li><li>`index_prefix`: Mandatory, string. A unique prefix used in Elasticsearch/Opensearch indices belonging to this index set. The prefix must start with a letter or number, and can only contain letters, numbers, `_`, `-` and `+`.</li><li>`replicas`: Mandatory, integer. Number of Elasticsearch/Opensearch replicas used per index in this index set.</li><li>`retention_strategy_class`: Mandatory, string. Retention strategy class to clean up old indices.</li><li>`retention_strategy`<ul><li>`max_number_of_indices`: Mandatory, integer. Maximum number of indices to keep before retention strategy gets triggered.</li><li>`type`: Mandatory, string. Retention strategy type to clean up old indices.</li></ul><li>`rotation_strategy_class`: Mandatory, string. Graylog uses multiple indices to store documents in. You can configure the strategy it uses to determine when to rotate the currently active write index.</li><li>`rotation_strategy`<ul><li>`rotation_period`: Mandatory, string. How long an index gets written to before it is rotated. (i.e. "P1D" for 1 day, "PT6H" for 6 hours).</li><li>`rotate_empty_index_set`: Mandatory, boolean. Apply the rotation strategy even when the index set is empty (not recommended).</li><li>`type`: Mandatory, string. The type of the Rotation Strategy.</li></ul><li>`shards`: Mandatory, integer. Number of Elasticsearch/Opensearch shards used per index in this index set.</li><li>`title`: Mandatory, string. Descriptive name of the index set.</li><li>`writable`: Mandatory, boolean. Whether this Index Set is writable.</li></ul> | One index per day; 365 indices max |
| `graylog_server__system_inputs` | Creates system inputs. Subkeys: <ul><li>`configuration`: Mandatory, dictionay. Specific configuration of corresponding input. Please refer to the [API documentation](https://go2docs.graylog.org/current/setting_up_graylog/rest_api.html).</li><li>`global`: Mandatory, boolean. Whether this input should start on all nodes.</li><li>`title`: Mandatory, string. The title for this input.</li><li>`type`: Mandatory, string. The type of the input.</li></ul> | Gelf (12201/TCP), Gelf (12201/UDP), Syslog (1514/UDP) |
| `graylog_server__system_index_sets__host_var` / `graylog_server__system_index_sets__group_var` | Creates additional index sets. Subkeys: <ul><li>`index`:<ul><li>`can_be_default`: Mandatory, boolean. Whether this index set can be be set to default via the Graylog GUI.</li><li>`data_tiering` (will be `null` if using legacy rotation)<ul><li>`index_lifetime_min`: Mandatory, string. The earliest age an index must reach before it becomes eligible for lifecycle actions (e.g., migration to a cooler tier, rollover, or deletion). Prevents moving/acting on indices too soon.</li><li>`index_lifetime_max`: Mandatory, string. The latest age an index may remain in its current tier. Once this age is hit, the next lifecycle action is enforced (migrate/rollover/delete), even if other triggers (size, docs) haven’t fired.</li><li>`type`: Mandatory, string. Data-tiering strategy for the index set. It defines which tiers (hot/warm/cold) are eligible and the order/behavior of migrations as indices age.</li></ul></li><li>`field_type_refresh_interval`: Mandatory, integer. Refresh interval in milliseconds.</li><li>`description`: Mandatory, string. Description of index set.</li><li>`field_type_refresh_interval`: Mandatory, integer. Refresh interval in milliseconds.</li><li>`index_analyzer`: Mandatory, string. Elasticsearch/Opensearch analyzer for this index set.</li><li>`index_optimization_max_num_segments`: Mandatory, integer. Maximum number of segments per Elasticsearch/Opensearch index after optimization (force merge).</li><li>`index_optimization_disabled`: Mandatory, boolean. Whether Elasticsearch/Opensearch index optimization (force merge) after rotation is disabled.</li><li>`index_prefix`: Mandatory, string. A unique prefix used in Elasticsearch/Opensearch indices belonging to this index set. The prefix must start with a letter or number, and can only contain letters, numbers, `_`, `-` and `+`. Note: As `index_prefix` must be unique, the index_prefix can not be updated after creation. If updated, a new index set would be created instead. It must not start with the same word, e.g. `linuxfabrik` and `linuxfabrik02` would conflict with each other.</li><li>`replicas`: Mandatory, integer. Number of Elasticsearch/Opensearch replicas used per index in this index set.</li><li>`retention_strategy_class`: Mandatory, string. Retention strategy class to clean up old indices.</li><li>`retention_strategy`<ul><li>`max_number_of_indices`: Mandatory, integer. Maximum number of indices to keep before retention strategy gets triggered.</li><li>`type`: Mandatory, string. Retention strategy type to clean up old indices.</li></ul><li>`rotation_strategy_class`: Mandatory, string. Graylog uses multiple indices to store documents in. You can configure the strategy it uses to determine when to rotate the currently active write index.</li><li>`rotation_strategy`<ul><li>`index_lifetime_max`: Mandatory, string. The maximum number of days the data in this index is kept before it is retained (i.e. "P1D" for 1 day, "PT6H" for 6 hours).</li><li>`index_lifetime_min`: Mandatory, string. The minimum number of days the data in this index is kept before it is retained (i.e. "P1D" for 1 day, "PT6H" for 6 hours).</li><li>`type`: Mandatory, string. The type of the Rotation Strategy.</li></ul><li>`shards`: Mandatory, integer. Number of Elasticsearch/Opensearch shards used per index in this index set.</li><li>`title`: Mandatory, string. Descriptive name of the index set.</li><li>`use_legacy_rotation`: Mandatory, boolean. Whether to use the legacy rotation method, i.e. "P1D".</li><li>`writable`: Mandatory, boolean. Whether this Index Set is writable.</li></li></ul><li>`state`: Optional, string. State of the index set, one of `present`, `absent`. Defaults to `'present'`</li></ul> | `unset` |
| `graylog_server__system_inputs__host_var` / `graylog_server__system_inputs__group_var` | Creates system inputs. Subkeys: <ul><li>`input`:<ul><li>`configuration`: Mandatory, dictionay. Specific configuration of corresponding input. Please refer to the [API documentation](https://go2docs.graylog.org/current/setting_up_graylog/rest_api.html). Note: As `port` must be unique, the port can not be updated after creation. If updated, a new input would be created instead.</li><li>`global`: Mandatory, boolean. Whether this input should start on all nodes.</li><li>`title`: Mandatory, string. The title for this input.</li><li>`type`: Mandatory, string. The type of the input.</li></ul></li><li>`state`: Optional, string. State of the index set, one of `present`, `absent`. Defaults to `'present'`</li></ul> | `unset` |

Example:
```yaml
# optional
graylog_server__system_default_index_set:
can_be_default: true
creation_date: '{{ ansible_date_time.iso8601 }}'
description: 'One index per day; 365 indices max'
field_type_refresh_interval: 5000
index_analyzer: 'standard'
index_optimization_disabled: false
index_optimization_max_num_segments: 1
index_prefix: 'lfops-default'
replicas: 0
retention_strategy:
max_number_of_indices: 365
type: 'org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig'
retention_strategy_class: 'org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy'
rotation_strategy:
rotation_period: 'P1D'
rotate_empty_index_set: false
type: 'org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig'
rotation_strategy_class: 'org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategy'
shards: 4
title: 'Linuxfabrik Index Set (managed by Ansible - do not edit)'
writable: true
graylog_server__system_inputs:
- configuration:
graylog_server__system_index_sets__host_var:
- index_prefix: 'linuxfabrik' # <-- Must be unique. Must not start with the same phrase, e.g. linuxfabrik and linuxfabrik02 would conflict with each other
can_be_default: true
data_tiering:
index_lifetime_min: 'P30D'
index_lifetime_max: 'P40D'
type: 'hot_only'
description: 'Another Index Set'
field_type_refresh_interval: 5000
index_analyzer: 'standard'
index_optimization_disabled: false
index_optimization_max_num_segments: 1
replicas: 0
retention_strategy:
max_number_of_indices: 20
type: 'org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig'
retention_strategy_class: 'org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy'
rotation_strategy:
index_lifetime_min: 'P7D'
index_lifetime_max: 'P10D'
type: 'org.graylog2.indexer.rotation.strategies.TimeBasedSizeOptimizingStrategyConfig'
rotation_strategy_class: 'org.graylog2.indexer.rotation.strategies.TimeBasedSizeOptimizingStrategy'
shards: 1
title: 'Linuxfabrik Index Set (managed by Ansible - do not edit)'
use_legacy_rotation: false
writable: true
state: 'present'
graylog_server__system_inputs__host_var:
- title: 'Beats (5044/TCP - managed by Ansible - do not edit)' # <-- Must be unique
configuration:
bind_address: '0.0.0.0'
number_worker_threads: 4
override_source: ''
Expand All @@ -131,9 +138,10 @@ graylog_server__system_inputs:
tls_key_file: ''
tls_key_password: ''
global: true
title: 'Beats (5044/TCP - managed by Ansible - do not edit)'
type: 'org.graylog.plugins.beats.Beats2Input'
- configuration:
state: 'present'
- title: 'Gelf (12201/TCP - managed by Ansible - do not edit)' # <-- Must be unique
configuration:
bind_address: '0.0.0.0'
decompress_size_limit: 8388608
max_message_size: 2097152
Expand All @@ -150,19 +158,21 @@ graylog_server__system_inputs:
tls_key_password: ''
use_null_delimiter: true
global: true
title: 'Gelf (12201/TCP - managed by Ansible - do not edit)'
type: 'org.graylog2.inputs.gelf.tcp.GELFTCPInput'
- configuration:
state: 'present'
- title: 'Gelf (12201/UDP - managed by Ansible - do not edit)' # <-- Must be unique
configuration:
bind_address: '0.0.0.0'
decompress_size_limit: 8388608
number_worker_threads: 4
override_source: ''
port: 12201
recv_buffer_size: 1048576
global: true
title: 'Gelf (12201/UDP - managed by Ansible - do not edit)'
type: 'org.graylog2.inputs.gelf.udp.GELFUDPInput'
- configuration:
state: 'present'
- title: 'Syslog (1514/UDP - managed by Ansible - do not edit)' # <-- Must be unique
configuration:
allow_override_date: true
bind_address: '0.0.0.0'
decompress_size_limit: 8388608
Expand All @@ -174,8 +184,8 @@ graylog_server__system_inputs:
recv_buffer_size: 1048576
store_full_message: false
global: true
title: 'Syslog (1514/UDP - managed by Ansible - do not edit)'
type: 'org.graylog2.inputs.syslog.udp.SyslogUDPInput'
state: 'present'
```


Expand Down
106 changes: 23 additions & 83 deletions roles/graylog_server/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,89 +8,29 @@ graylog_server__service_enabled: true
graylog_server__stale_leader_timeout_ms: 2000
graylog_server__timezone: 'Europe/Zurich'

graylog_server__system_default_index_set:
can_be_default: true
creation_date: '{{ ansible_date_time.iso8601 }}'
description: 'One index per day; 365 indices max'
field_type_refresh_interval: 5000
index_analyzer: 'standard'
index_optimization_disabled: false
index_optimization_max_num_segments: 1
index_prefix: 'lfops-default'
replicas: 0
retention_strategy:
max_number_of_indices: 365
type: 'org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig'
retention_strategy_class: 'org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy'
rotation_strategy:
rotation_period: 'P1D'
rotate_empty_index_set: false
type: 'org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig'
rotation_strategy_class: 'org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategy'
shards: 4
title: 'Linuxfabrik Index Set (managed by Ansible - do not edit)'
writable: true
graylog_server__system_index_sets__dependent_var: []
graylog_server__system_index_sets__group_var: []
graylog_server__system_index_sets__host_var: []
graylog_server__system_index_sets__role_var: []
graylog_server__system_index_sets__combined_var: '{{ (
graylog_server__system_index_sets__role_var +
graylog_server__system_index_sets__dependent_var +
graylog_server__system_index_sets__group_var +
graylog_server__system_index_sets__host_var
) | linuxfabrik.lfops.combine_lod(unique_key="index_prefix")
}}'

graylog_server__system_inputs:
- configuration:
bind_address: '0.0.0.0'
number_worker_threads: 4
override_source: ''
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: ''
tls_client_auth: 'disabled'
tls_client_auth_cert_file: ''
tls_enable: false
tls_key_file: ''
tls_key_password: ''
global: true
title: 'Beats (5044/TCP - managed by Ansible - do not edit)'
type: 'org.graylog.plugins.beats.Beats2Input'
- configuration:
bind_address: '0.0.0.0'
decompress_size_limit: 8388608
max_message_size: 2097152
number_worker_threads: 4
override_source: ''
port: 12201
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: ''
tls_client_auth: 'disabled'
tls_client_auth_cert_file: ''
tls_enable: false
tls_key_file: ''
tls_key_password: ''
use_null_delimiter: true
global: true
title: 'Gelf (12201/TCP - managed by Ansible - do not edit)'
type: 'org.graylog2.inputs.gelf.tcp.GELFTCPInput'
- configuration:
bind_address: '0.0.0.0'
decompress_size_limit: 8388608
number_worker_threads: 4
override_source: ''
port: 12201
recv_buffer_size: 1048576
global: true
title: 'Gelf (12201/UDP - managed by Ansible - do not edit)'
type: 'org.graylog2.inputs.gelf.udp.GELFUDPInput'
- configuration:
allow_override_date: true
bind_address: '0.0.0.0'
decompress_size_limit: 8388608
expand_structured_data: false
force_rdns: false
number_worker_threads: 4
override_source: ''
port: 1514
recv_buffer_size: 1048576
store_full_message: false
global: true
title: 'Syslog (1514/UDP - managed by Ansible - do not edit)'
type: 'org.graylog2.inputs.syslog.udp.SyslogUDPInput'
graylog_server__system_inputs__dependent_var: []
graylog_server__system_inputs__group_var: []
graylog_server__system_inputs__host_var: []
graylog_server__system_inputs__role_var: []
Copy link
Member

@NavidSassan NavidSassan Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did we remove the default values here? please explain the intended way to use the role in an "Post-Installation Steps" section in the README.md (have a look at the elasticsearch role)

Copy link
Member

@NavidSassan NavidSassan Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still missing

Copy link
Contributor Author

@lfchris lfchris Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know what you mean. they are there

graylog_server__system_inputs__combined_var: '{{ (
graylog_server__system_inputs__role_var +
graylog_server__system_inputs__dependent_var +
graylog_server__system_inputs__group_var +
graylog_server__system_inputs__host_var
) | linuxfabrik.lfops.combine_lod(unique_key=["title", "type"])
}}'

# ---------------------------------------------------

Expand All @@ -99,7 +39,7 @@ graylog_server__kernel_settings__sysctl__dependent_var:
# `WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input ... should be >= 1048576 but is 425984`
# this warning is related to how the operating system is configured (udp receive buffer size)
- name: 'net.core.rmem_max'
value: 1048576 # default: 212992
value: 1048576 # default: 212992
graylog_server__kernel_settings__transparent_hugepages__dependent_var: 'never'

graylog_server__selinux__booleans__dependent_var:
Expand Down
Loading