commonlib: Support combined metricsSource (#1317)

* Add support that varMetric arg to be provided as an array * Fix typo * Add info label test * Update multi combine test * Fix typo in query (JVM survival memory) * Update asPanelExpression/asRuleExpression Support multiple metricsSources * Add info support * Fix memoryUsedHeapPercent/memoryUsedNonHeapPercent when using multiple metricSources * Update signal * Remove info label * Update exprBase to expr in sourceMaps * Move exprWrappers to sourceMaps * Add combineUniqueExpressions * Move expr to sourceMaps[].expr * Move valueMappings to sourceMaps * Move rangeFunction to sourceMaps * Move aggKeepLabels to sourceMaps * Refactor: move legendCustomTemplate to sourceMaps * Update README * Fix stub * Fix stub generation for multi * Fix gc metric * Deduplicate valuemappings * Update libs to support multiple sources * Fix kafka metrics * Update hello-world
grafana · Aug 28, 2024 · d7f91f1 · d7f91f1
1 parent 8a25b0e
commit d7f91f1
Show file tree

Hide file tree

Showing 40 changed files with 783 additions and 309 deletions.
diff --git a/common-lib/common/signal/README.md b/common-lib/common/signal/README.md
@@ -59,15 +59,15 @@ Signal's level:
 |type|Signal's type. Depending on the type, some opinionated autotransformations would happen with queries, units. |gauge,counter,histogram,info,raw|gauge|-|
 |unit| Signal's units. |*|bytes|``|
 |description| Signal's description. Used to populate panel's description. |*|CPU usage time in percent.|``|
-|expr| Signal's BASE expression in simplest form. Simplified jsonnet templating is supported (see below). Depending on signal's type(not `raw`) could autotransform to different form. |*|network_bytes_received_total{%(queriesSelector)s}|-|
-|exprWrappers| Signal's additional wrapper functions that could be added as an array, [<left_part>, <right_part>]. Functions would be applied AFTER any autotransformation takes place.  |*|`['topk(10,',')']`|[]|
+|sourceMaps[].expr| Signal's BASE expression in simplest form. Simplified jsonnet templating is supported (see below). Depending on signal's type(not `raw`) could autotransform to different form. |*|network_bytes_received_total{%(queriesSelector)s}|-|
+|sourceMaps[].exprWrappers| Signal's additional wrapper functions that could be added as an array, [<left_part>, <right_part>]. Functions would be applied AFTER any autotransformation takes place.  |*|`['topk(10,',')']`|[]|
 |aggLevel| Metrics aggregation level. |none, instance, group|`group`|`none`|
 |aggFunction| A function used to aggregate metrics. |avg,min,max,sum...|`sum`|`avg`|
-|aggKeepLabels| Extra labels to keep when aggregating with by() clause.  |`['pool','level']`|`[]`|
-|infoLabel| Only applicable to `info` metrics. Points to label name used to extract info. |*|-|-|
-|valueMappings| Define signal's valueMappings in the same way defined in Grafana Dashboard Schema. |*|-|-|
-|legendCustomTemplate| A custom legend template could be defined with this to override automatic legend's generation|*|`null`|`{{instance}}`|
-|rangeFunction| Rate function to use for counter metrics.|rate,irate,delta,idelta,increase|`rate`|`increase`|
+|sourceMaps[].aggKeepLabels| Extra labels to keep when aggregating with by() clause.  |`['pool','level']`|`[]`|
+|sourceMaps[].infoLabel| Only applicable to `info` metrics. Points to label name used to extract info. |*|-|-|
+|sourceMaps[].valueMappings| Define signal's valueMappings in the same way defined in Grafana Dashboard Schema. |*|-|-|
+|sourceMaps[].legendCustomTemplate| A custom legend template could be defined with this to override automatic legend's generation|*|`null`|`{{topic}}`|
+|sourceMaps[].rangeFunction| Rate function to use for counter metrics.|rate,irate,delta,idelta,increase|`rate`|`increase`|
 
 ## Expressions templating
 
@@ -83,7 +83,121 @@ The following is supported in expressions and legends:
 - `%(interval)s` - expands to `interval` value
 - `%(alertsInterval)s` - expands to `interval` value
 
-## Example 1
+
+## Example 1: From JSON
+
+Define signals in json/jsonnet and unmarshall to signals. You can define multiple signals' sources:
+
+```jsonnet
+//config.libsonnet
+
+local jsonSignals =
+  {
+    aggLevel: 'group',
+    groupLabels: ['job'],
+    instanceLabels: ['instance'],
+    filteringSelector: 'job="integrations/agent"',
+    discoveryMetric: {
+      prometheus: 'up',
+      opentelemetry: 'target_info',
+    },
+    signals: {
+      abc: {
+        name: 'ABC',
+        type: 'gauge',
+        description: 'ABC',
+        unit: 's',
+        sources: {
+          "prometheus": {
+            expr: 'abc_total{%(queriesSelector)s}',
+          },
+          "opentelemetry": {
+            expr: 'otel_metric{%(queriesSelector)s}',
+          }
+        },
+      },
+      bar: {
+        name: 'BAR',
+        type: 'counter',
+        description: 'BAR',
+        unit: 'ns',
+        sources: {
+          "prometheus": {
+            expr: 'bar_total{%(queriesSelector)s}',
+          },
+          "opentelemetry": {
+            expr: 'otel_bar{%(queriesSelector)s}',
+          }
+        },
+        
+      },
+      foo_info: {
+        name: 'foo info',
+        type: 'info',
+        description: 'foo',
+        unit: 'short',
+        optional: true, // set optional:true, if not all sources are available.
+        sources: {
+          "prometheus": {
+            expr: 'foo_info{%(queriesSelector)s}',
+            infoLabel: 'version',
+          },
+        },
+        
+      },
+      status: {
+        name: 'Status',
+        type: 'gauge',
+        description: 'status',
+        unit: 'short',
+        optional: true,
+        sources: {
+          "prometheus": {
+            expr: 'status{%(queriesSelector)s}',
+            valueMappings: 
+              [
+                {
+                  type: 'value',
+                  options: {
+                    '1': {
+                      text: 'Up',
+                      color: 'light-green',
+                      index: 1,
+                    },
+                    '0': {
+                      text: 'Down',
+                      color: 'light-red',
+                      index: 0,
+                    },
+                  }
+                },
+              ],
+          },
+        },
+        
+      },
+    },
+  };
+
+local signals = signal.unmarshallJsonMulti(jsonSignals,type=['prometheus']);
+
+```
+
+## Example 2: Generate 'combined' dashboard/alerts
+
+Same as example 1, but you can actually merge multiple sources together in one query by using 'or':
+
+```jsonnet
+local jsonSignals = <see example1>;
+local signals = signal.unmarshallJsonMulti(jsonSignals,type=['prometheus','opentelemetry]);
+```
+
+This would make your dashboards and alerts universal, suitable for multiple sources.
+
+
+## Example 3: low-level
+
+Rarely used, but it is also possible to skip JSON unmarshalling: 
 
 ```
 local g = import 'g.libsonnet';
@@ -105,14 +219,29 @@ local g = import 'g.libsonnet';
         type='counter',
         unit='bytes',
         description='Bytes in through interface',
-        expr='node_network_receive_bytes_total{%(queriesSelector)s}',
+        sourceMaps=[
+          {
+            expr: 'node_network_receive_bytes_total{%(queriesSelector)s}',
+            rangeFunction: null,
+            aggKeepLabels: [],
+            legendCustomTemplate: null,
+          },
+        ],
     ),
     bytesOut: s.addSignal(
         name='Bytes out',
         type='counter',
         unit='bytes',
         description='Bytes out through interface',
-        expr='node_network_transmit_bytes_total{%(queriesSelector)s}',
+        sourceMaps=[
+          {
+            expr: 'node_network_transmit_bytes_total{%(queriesSelector)s}',
+            rangeFunction: null,
+            aggKeepLabels: [],
+            legendCustomTemplate: null,
+          },
+        ],
+        
     ),
 };
 
@@ -135,80 +264,10 @@ g.dashboard.new('Device')
 )
 ```
 
-
-## Example 2
-
-Define signals in json/jsonnet and unmarshall to signals:
-
-```jsonnet
-//config.libsonnet
-
-local jsonSignals =
-  {
-    aggLevel: 'group',
-    groupLabels: ['job'],
-    instanceLabels: ['instance'],
-    filteringSelector: 'job="integrations/agent"',
-    signals: {
-      abc: {
-        name: 'ABC',
-        type: 'gauge',
-        description: 'ABC',
-        unit: 's',
-        expr: 'abc{%(queriesSelector)s}',
-      },
-      bar: {
-        name: 'BAR',
-        type: 'counter',
-        description: 'BAR',
-        unit: 'ns',
-        expr: 'bar{%(queriesSelector)s}',
-      },
-      foo_info: {
-        name: 'foo info',
-        type: 'info',
-        description: 'foo',
-        unit: 'short',
-        infoLabel: 'version',
-        expr: 'foo_info{%(queriesSelector)s}',
-      },
-      status: {
-        name: 'Status',
-        type: 'gauge',
-        description: 'status',
-        unit: 'short',
-        expr: 'status{%(queriesSelector)s}',
-        valueMappings: 
-          [
-            {
-              type: 'value',
-              options: {
-                '1': {
-                  text: 'Up',
-                  color: 'light-green',
-                  index: 1,
-                },
-                '0': {
-                  text: 'Down',
-                  color: 'light-red',
-                  index: 0,
-                },
-              }
-            },
-          ],
-      },
-    },
-  };
-
-local signals = signal.unmarshallJson(jsonSignals);
-
-```
-
-
 ## Running tests
 
 See test files
 ```
 cd common-lib
 jsonnet -J vendor common/signal/test_histogram.libsonnet 
-```
+```