Galera cluster

docs/severity.md

@@ -80,6 +80,7 @@
 - [smart-agent_dns](#smart-agent_dns)
 - [smart-agent_docker](#smart-agent_docker)
 - [smart-agent_elasticsearch](#smart-agent_elasticsearch)
+- [smart-agent_galera](#smart-agent_galera)
 - [smart-agent_genericjmx](#smart-agent_genericjmx)
 - [smart-agent_haproxy](#smart-agent_haproxy)
 - [smart-agent_health-checker](#smart-agent_health-checker)
@@ -878,6 +879,16 @@
 |Elasticsearch max time spent by task in queue rate of change|-|X|X|-|-|
 
 
+## smart-agent_galera
+
+|Detector|Critical|Major|Minor|Warning|Info|
+|---|---|---|---|---|---|
+|Galera node|X|-|-|-|-|
+|Galera node state|X|-|-|-|-|
+|Galera replication paused ratio|-|X|X|-|-|
+|Galera recv queue length|-|X|X|-|-|
+
+
 ## smart-agent_genericjmx
 
 |Detector|Critical|Major|Minor|Warning|Info|

modules/smart-agent_galera/README.md

@@ -0,0 +1,185 @@
+# GALERA SignalFx detectors
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+:link: **Contents**
+
+- [How to use this module?](#how-to-use-this-module)
+- [What are the available detectors in this module?](#what-are-the-available-detectors-in-this-module)
+- [How to collect required metrics?](#how-to-collect-required-metrics)
+  - [Metrics](#metrics)
+- [Notes](#notes)
+- [Related documentation](#related-documentation)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## How to use this module?
+
+This directory defines a [Terraform](https://www.terraform.io/)
+[module](https://www.terraform.io/language/modules/syntax) you can use in your
+existing [stack](https://github.com/claranet/terraform-signalfx-detectors/wiki/Getting-started#stack) by adding a
+`module` configuration and setting its `source` parameter to URL of this folder:
+
+```hcl
+module "signalfx-detectors-smart-agent-galera" {
+  source = "github.com/claranet/terraform-signalfx-detectors.git//modules/smart-agent_galera?ref={revision}"
+
+  environment   = var.environment
+  notifications = local.notifications
+}
+```
+
+Note the following parameters:
+
+* `source`: Use this parameter to specify the URL of the module. The double slash (`//`) is intentional  and required.
+  Terraform uses it to specify subfolders within a Git repo (see [module
+  sources](https://www.terraform.io/language/modules/sources)). The `ref` parameter specifies a specific Git tag in
+  this repository. It is recommended to use the latest "pinned" version in place of `{revision}`. Avoid using a branch
+  like `master` except for testing purpose. Note that every modules in this repository are available on the Terraform
+  [registry](https://registry.terraform.io/modules/claranet/detectors/signalfx) and we recommend using it as source
+  instead of `git` which is more flexible but less future-proof.
+
+* `environment`: Use this parameter to specify the
+  [environment](https://github.com/claranet/terraform-signalfx-detectors/wiki/Getting-started#environment) used by this
+  instance of the module.
+  Its value will be added to the `prefixes` list at the start of the [detector
+  name](https://github.com/claranet/terraform-signalfx-detectors/wiki/Templating#example).
+  In general, it will also be used in the `filtering` internal sub-module to [apply
+  filters](https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance#filtering) based on our default
+  [tagging convention](https://github.com/claranet/terraform-signalfx-detectors/wiki/Tagging-convention) by default.
+
+* `notifications`: Use this parameter to define where alerts should be sent depending on their severity. It consists
+  of a Terraform [object](https://www.terraform.io/language/expressions/type-constraints#object) where each key represents an available
+  [detector rule severity](https://docs.splunk.com/observability/alerts-detectors-notifications/create-detectors-for-alerts.html#severity)
+  and its value is a list of recipients. Every recipients must respect the [detector notification
+  format](https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/detector#notification-format).
+  Check the [notification binding](https://github.com/claranet/terraform-signalfx-detectors/wiki/Notifications-binding)
+  documentation to understand the recommended role of each severity.
+
+These 3 parameters alongs with all variables defined in [common-variables.tf](common-variables.tf) are common to all
+[modules](../) in this repository. Other variables, specific to this module, are available in
+[variables-gen.tf](variables-gen.tf).
+In general, the default configuration "works" but all of these Terraform
+[variables](https://www.terraform.io/language/values/variables) make it possible to
+customize the detectors behavior to better fit your needs.
+
+Most of them represent usual tips and rules detailled in the
+[guidance](https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance) documentation and listed in the
+common [variables](https://github.com/claranet/terraform-signalfx-detectors/wiki/Variables) dedicated documentation.
+
+Feel free to explore the [wiki](https://github.com/claranet/terraform-signalfx-detectors/wiki) for more information about
+general usage of this repository.
+
+## What are the available detectors in this module?
+
+This module creates the following SignalFx detectors which could contain one or multiple alerting rules:
+
+|Detector|Critical|Major|Minor|Warning|Info|
+|---|---|---|---|---|---|
+|Galera node|X|-|-|-|-|
+|Galera node state|X|-|-|-|-|
+|Galera replication paused ratio|-|X|X|-|-|
+|Galera recv queue length|-|X|X|-|-|
+
+## How to collect required metrics?
+
+This module deploys detectors using metrics reported by the
+[SignalFx Smart Agent Monitors](https://github.com/signalfx/signalfx-agent#monitors).
+
+Even if the [Smart Agent is deprecated](https://github.com/signalfx/signalfx-agent/blob/main/docs/smartagent-deprecation-notice.md)
+it remains an efficient, lightweight and simple monitoring agent which still works fine.
+See the [official documentation](https://docs.splunk.com/Observability/gdi/smart-agent/smart-agent-resources.html) for more information
+about this agent.
+You might find the related following documentations useful:
+- the global level [agent configuration](https://github.com/signalfx/signalfx-agent/blob/main/docs/config-schema.md)
+- the [monitor level configuration](https://github.com/signalfx/signalfx-agent/blob/main/docs/monitor-config.md)
+- the internal [agent configuration tips](https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance#agent-configuration).
+- the full list of [monitors available](https://github.com/signalfx/signalfx-agent/tree/main/docs/monitors) with their own specific documentation.
+
+In addition, all of these monitors are still available in the [Splunk Otel Collector](https://github.com/signalfx/splunk-otel-collector),
+the Splunk [distro of OpenTelemetry Collector](https://opentelemetry.io/docs/concepts/distributions/) which replaces SignalFx Smart Agent,
+thanks to the internal [Smart Agent Receiver](https://github.com/signalfx/splunk-otel-collector/tree/main/internal/receiver/smartagentreceiver).
+
+As a result:
+- any SignalFx Smart Agent monitor are compatible with the new agent OpenTelemetry Collector and related modules in this repository keep `smart-agent` as source name.
+- any OpenTelemetry receiver not based on an existing Smart Agent monitor is not available from old agent so related modules in this repository use `otel-collector` as source name.
+
+
+Check the [Related documentation](#related-documentation) section for more detailed and specific information about this module dependencies.
+
+This module is only based on the [sql](https://docs.splunk.com/Observability/gdi/sql/sql.html) monitor.
+
+Here is a full configuration example working with these detectors:
+
+```yaml
+  - type: sql
+    dbDriver: mysql
+    host: &mysqlHost localhost
+    port: &mysqlPort 3306
+    extraDimensions:
+      mysql_port: *mysqlPort
+    params:
+      host: *mysqlHost
+      port: *mysqlPort
+      dbname: mysql
+      username: {"#from": "vault:secret/my-database[username]"}
+      password: {"#from": "vault:secret/my-database[password]"}
+    connectionString: '{{.username}}:{{.password}}@tcp({{.host}}:{{.port}})/{{.dbname}}'
+    queries:
+      - query: 'SHOW STATUS LIKE "wsrep_ready";'
+        datapointExpressions:
+          - 'GAUGE("mysql_wsrep_ready", {}, Value== "ON" ? 1: 0)'
+
+      - query: 'SHOW STATUS LIKE "wsrep_local_state";'
+        metrics:
+          - metricName: "mysql_wsrep_local_state"
+            valueColumn: "Value"
+
+      - query: 'SHOW STATUS LIKE "wsrep_flow_control_paused";'
+        metrics:
+          - metricName: "mysql_wsrep_flow_control_paused"
+            valueColumn: "Value"
+
+      - query: 'SHOW STATUS LIKE "wsrep_local_recv_queue_avg";'
+        metrics:
+          - metricName: "mysql_wsrep_local_recv_queue_avg"
+            valueColumn: "Value"
+```
+
+If you already use the MySQL module, you can just merge the `queries` attribute with the one already present in your configuration.
+
+
+### Metrics
+
+
+To filter only required metrics for the detectors of this module, add the
+[datapointsToExclude](https://docs.splunk.com/observability/gdi/smart-agent/smart-agent-resources.html#filtering-data-using-the-smart-agent)
+parameter to the corresponding monitor configuration:
+
+```yaml
+    datapointsToExclude:
+      - metricNames:
+        - '*'
+        - '!mysql_wsrep_flow_control_paused'
+        - '!mysql_wsrep_local_recv_queue_avg'
+        - '!mysql_wsrep_local_state'
+        - '!mysql_wsrep_ready'
+
+```
+
+## Notes
+
+To prepare the Mysql server allowing monitoring you can use [this terraform module](https://github.com/claranet/terraform-signalfx-integrations/tree/master/database/mysql).
+
+This module is usually used in addition with the [MySQL](../smart-agent_galera/) module.
+
+
+## Related documentation
+
+* [Terraform SignalFx provider](https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs)
+* [Terraform SignalFx detector](https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/detector)
+* [Splunk Observability integrations](https://docs.splunk.com/Observability/gdi/get-data-in/integrations.html)
+* [Official SQL documentation](https://docs.signalfx.com/en/latest/integrations/integrations-reference/integrations.mysql.html)
+* [Galera Status variables](https://galeracluster.com/library/documentation/galera-status-variables.html)
+* [Galera monitoring](https://galeracluster.com/library/documentation/monitoring-cluster.html)
+* [Galera node state](https://galeracluster.com/library/documentation/node-states.html#node-state-changes)

modules/smart-agent_galera/common-filters.tf

@@ -0,0 +1 @@
+../../common/module/filters-smart-agent.tf

modules/smart-agent_galera/common-locals.tf

@@ -0,0 +1 @@
+../../common/module/locals.tf

modules/smart-agent_galera/common-modules.tf

@@ -0,0 +1 @@
+../../common/module/modules.tf

modules/smart-agent_galera/common-variables.tf

@@ -0,0 +1 @@
+../../common/module/variables.tf

modules/smart-agent_galera/common-versions.tf

@@ -0,0 +1 @@
+../../common/module/versions.tf

modules/smart-agent_galera/conf/01-node.yaml

@@ -0,0 +1,13 @@
+module: galera
+name: "node"
+transformation: true
+aggregation: true
+signals:
+  signal:
+    metric: "mysql_wsrep_ready"
+rules:
+  critical:
+    lasting_duration: 10m
+    description: "is not ready"
+    threshold: 1
+    comparator: "<"

modules/smart-agent_galera/conf/02-state.yaml

@@ -0,0 +1,14 @@
+module: galera
+name: "node state"
+transformation: true
+aggregation: true
+disable: true
+signals:
+  signal:
+    metric: "mysql_wsrep_local_state"
+rules:
+  critical:
+    lasting_duration: 10m
+    description: "is not Synced"
+    threshold: 4
+    comparator: "!="

modules/smart-agent_galera/conf/03-flow-control.yaml

@@ -0,0 +1,18 @@
+module: galera
+name: "replication paused ratio"
+transformation: true
+aggregation: true 
+disabled: true
+signals:
+  signal:
+    metric: "mysql_wsrep_flow_control_paused"
+rules:
+  major:
+    lasting_duration: 10m
+    threshold: 0.05
+    comparator: ">"
+  minor:
+    lasting_duration: 10m
+    threshold: 0
+    comparator: ">"
+    dependency: major

modules/smart-agent_galera/conf/04-recv-queue.yaml

@@ -0,0 +1,18 @@
+module: galera
+name: "Recv queue length"
+transformation: true
+aggregation: true
+disabled: true
+signals:
+  signal:
+    metric: "mysql_wsrep_local_recv_queue_avg"
+rules:
+  major:
+    lasting_duration: 10m
+    threshold: 0.1
+    comparator: ">"
+  minor:
+    lasting_duration: 10m
+    threshold: 0
+    comparator: ">"
+    dependency: major

modules/smart-agent_galera/conf/readme.yaml

@@ -0,0 +1,57 @@
+documentations:
+  - name: Official SQL documentation
+    url: https://docs.signalfx.com/en/latest/integrations/integrations-reference/integrations.mysql.html
+  - name: Galera Status variables
+    url: https://galeracluster.com/library/documentation/galera-status-variables.html
+  - name: Galera monitoring
+    url: https://galeracluster.com/library/documentation/monitoring-cluster.html
+  - name: Galera node state
+    url: https://galeracluster.com/library/documentation/node-states.html#node-state-changes
+notes: |
+  To prepare the Mysql server allowing monitoring you can use [this terraform module](https://github.com/claranet/terraform-signalfx-integrations/tree/master/database/mysql).
+
+  This module is usually used in addition with the [MySQL](../smart-agent_galera/) module.
+
+source_doc: |
+  This module is only based on the [sql](https://docs.splunk.com/Observability/gdi/sql/sql.html) monitor.
+
+  Here is a full configuration example working with these detectors:
+
+  ```yaml
+    - type: sql
+      dbDriver: mysql
+      host: &mysqlHost localhost
+      port: &mysqlPort 3306
+      extraDimensions:
+        mysql_port: *mysqlPort
+      params:
+        host: *mysqlHost
+        port: *mysqlPort
+        dbname: mysql
+        username: {"#from": "vault:secret/my-database[username]"}
+        password: {"#from": "vault:secret/my-database[password]"}
+      connectionString: '{{.username}}:{{.password}}@tcp({{.host}}:{{.port}})/{{.dbname}}'
+      queries:
+        - query: 'SHOW STATUS LIKE "wsrep_ready";'
+          datapointExpressions:
+            - 'GAUGE("mysql_wsrep_ready", {}, Value== "ON" ? 1: 0)'
+
+        - query: 'SHOW STATUS LIKE "wsrep_local_state";'
+          metrics:
+            - metricName: "mysql_wsrep_local_state"
+              valueColumn: "Value"
+
+        - query: 'SHOW STATUS LIKE "wsrep_flow_control_paused";'
+          metrics:
+            - metricName: "mysql_wsrep_flow_control_paused"
+              valueColumn: "Value"
+
+        - query: 'SHOW STATUS LIKE "wsrep_local_recv_queue_avg";'
+          metrics:
+            - metricName: "mysql_wsrep_local_recv_queue_avg"
+              valueColumn: "Value"
+  ```
+
+  If you already use the MySQL module, you can just merge the `queries` attribute with the one already present in your configuration.
+
+