new segment: analysis/toptalkers-metrics (#66)

* added first working version of toptalkers_metrics segment * moved ticker from records to DB, records are now tracking if they are elegible for export, cleanup runs as separate goroutine, bucket duration is now configurable * forward flows based on traffic levels * added a metric for the DB size after cleanup * defined the segment as a filter segment, so drops can be used in the pipeline * added documentation and example for toptalkers-metrics * go mod tidy * go mod tidy * go version bumped from 1.18 to 1.20 * fuck yaml * updated Dockerfile to go 1.20 * updated dependencies --------- Co-authored-by: Sebastian Neuner <[email protected]>
BelWue · Feb 24, 2023 · 66562c2 · 66562c2
1 parent acb7492
commit 66562c2
Show file tree

Hide file tree

Showing 9 changed files with 734 additions and 36 deletions.
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
@@ -27,7 +27,7 @@ jobs:
     - name: setup go
       uses: actions/setup-go@v3
       with:
-        go-version: 1.18
+        go-version: '1.20'
 
     - name: test
       run: go test ./...

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -25,7 +25,7 @@ jobs:
     - name: setup go
       uses: actions/setup-go@v3
       with:
-        go-version: 1.18
+        go-version: '1.20'
 
     - name: test before release
       run: go test ./...

diff --git a/CONFIGURATION.md b/CONFIGURATION.md
@@ -70,6 +70,55 @@ conditional, limiting payload data, and multiple receivers.
 [godoc](https://pkg.go.dev/github.com/bwNetFlow/flowpipeline/segments/alert/http)
 [examples using this segment](https://github.com/search?q=%22segment%3A+http%22+extension%3Ayml+repo%3AbwNetFlow%2Fflowpipeline%2Fexamples&type=Code)
 
+### Analysis Group
+Segments in this group do higher level analysis on flow data. They usually
+export or print results in some way, but might also filter given flows.
+
+#### toptalkers-metrics
+The `toptalkers-metrics` segment calculates statistics about traffic levels
+per IP address and exports them in OpenMetrics format via HTTP.
+
+Traffic is counted in bits per second and packets per second, categorized into
+forwarded and dropped traffic. By default, only the destination IP addresses
+are accounted, but the configuration allows using the source IP address or
+both addresses. For the latter, a flows number of bytes and packets are
+ccounted for both addresses.
+
+Thresholds for bits per second or packets per second can be configured. Only
+metrics for addresses that exceeded this threshold during the last window size
+are exported. This can be used for detection of unusual or unwanted traffic
+levels. This can also be used as a flow filter: While the average traffic for
+an address is above threshold, flows are passed, other flows are dropped.
+
+The averages are calculated with a sliding window. The window size (in number
+of buckets) and the bucket duration can be configured. By default, it uses
+60 buckets of 1 second each (1 minute of sliding window). Optionally, the
+window size for the exported metrics calculation and for the threshold check
+can be configured differently.
+
+The parameter "traffictype" is passed as OpenMetrics label, so this segment
+can be used multiple times in one pipeline without metrics getting mixed up.
+
+```
+- segment: toptalkers-metrics
+  config:
+    # the lines below are optional and set to default
+    traffictype: ""
+    buckets: 60
+    BucketDuration: 1
+    Thresholdbuckets: 60
+    reportbuckets: 60
+    thresholdbps: 0
+    thresholdpps: 0
+    endpoint: ":8080"
+    metricspath: "/metrics"
+    flowdatapath: "/flowdata"
+    relevantaddress: "destination"
+```
+
+[godoc](https://pkg.go.dev/github.com/bwNetFlow/flowpipeline/segments/analysis/toptalkers-metrics)
+[examples using this segment](https://github.com/search?q=%22segment%3A+toptalkers-metrics%22+extension%3Ayml+repo%3AbwNetFlow%2Fflowpipeline%2Fexamples&type=Code)
+
 ### Controlflow Group
 Segments in this group have the ability to change the sequence of segments any
 given flow traverses.

diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM golang:1.18 AS builder
+FROM golang:1.20 AS builder
 RUN apt-get update
 
 # add local repo into the builder

diff --git a/examples/analysis/toptalkers-metrics.yml b/examples/analysis/toptalkers-metrics.yml
@@ -0,0 +1,35 @@
+---
+###############################################################################
+# Consume flow messages, it's best to use an enriched topic as flowdump
+# printing involves interface descriptions.
+- segment: kafkaconsumer
+  config:
+    server: kafka01.example.com:9093
+    topic: flow-messages-enriched
+    group: myuser-flowdump
+    user: myuser
+    pass: $KAFKA_SASL_PASS
+
+###############################################################################
+# filter for some interesting traffic, in this case something
+# that is likely used for DDoS attacks (UDP with source port 123
+# is seen during NTP amplification attacks)
+- segment: flowfilter
+  config:
+    filter: "proto udp and src port 123"
+
+###############################################################################
+# creates OpenMetrics endpoints for traffic data
+# default endpoints are:
+# <host>:8080/flowdata
+# <host>:8080/metrics
+# the given labels in this example are the default ones.
+# They are also applied if the labels field is omitted.
+- segment: prometheus
+  config:
+    endpoint: ":8080"
+    # 12 buckets at 5 seconds each -> 1 minute of sliding window
+    buckets: 12
+    bucketduration: 5
+    # set some thresholds (here 1 Gbps)
+    thresholdbps: 1000000000
diff --git a/go.mod b/go.mod
@@ -1,11 +1,11 @@
 module github.com/bwNetFlow/flowpipeline
 
-go 1.18
+go 1.20
 
 require (
 	github.com/BelWue/bgp_routeinfo v0.0.0-20221004100427-d8095fc566dd
-	github.com/ClickHouse/clickhouse-go/v2 v2.6.1
-	github.com/Shopify/sarama v1.37.2
+	github.com/ClickHouse/clickhouse-go/v2 v2.6.3
+	github.com/Shopify/sarama v1.38.1
 	github.com/Yawning/cryptopan v0.0.0-20170504040949-65bca51288fe
 	github.com/alouca/gosnmp v0.0.0-20170620005048-04d83944c9ab
 	github.com/asecurityteam/rolling v2.0.4+incompatible
@@ -16,31 +16,32 @@ require (
 	github.com/dustin/go-humanize v1.0.1
 	github.com/google/gopacket v1.1.19
 	github.com/hashicorp/logutils v1.0.0
-	github.com/influxdata/influxdb-client-go/v2 v2.11.0
-	github.com/mattn/go-sqlite3 v1.14.15
+	github.com/influxdata/influxdb-client-go/v2 v2.12.2
+	github.com/mattn/go-sqlite3 v1.14.16
 	github.com/netsampler/goflow2 v1.1.1
 	github.com/oschwald/maxminddb-golang v1.10.0
 	github.com/patrickmn/go-cache v2.1.0+incompatible
-	github.com/prometheus/client_golang v1.13.0
+	github.com/prometheus/client_golang v1.14.0
 	github.com/rs/dnscache v0.0.0-20211102005908-e0241e321417
 	google.golang.org/protobuf v1.28.1
 	gopkg.in/yaml.v2 v2.4.0
 )
 
 require (
-	github.com/ClickHouse/ch-go v0.51.2 // indirect
+	github.com/ClickHouse/ch-go v0.53.0 // indirect
 	github.com/alecthomas/participle/v2 v2.0.0-beta.1 // indirect
 	github.com/alouca/gologger v0.0.0-20120904114645-7d4b7291de9c // indirect
-	github.com/andybalholm/brotli v1.0.4 // indirect
+	github.com/andybalholm/brotli v1.0.5 // indirect
+	github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cespare/xxhash/v2 v2.1.2 // indirect
-	github.com/cilium/ebpf v0.9.3 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
+	github.com/cilium/ebpf v0.10.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/deepmap/oapi-codegen v1.11.0 // indirect
+	github.com/deepmap/oapi-codegen v1.12.4 // indirect
 	github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
 	github.com/eapache/channels v1.1.0 // indirect
 	github.com/eapache/go-resiliency v1.3.0 // indirect
-	github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
+	github.com/eapache/go-xerial-snappy v0.0.0-20230111030713-bf00bc1b83b6 // indirect
 	github.com/eapache/queue v1.1.0 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
 	github.com/go-faster/city v1.0.1 // indirect
@@ -58,42 +59,45 @@ require (
 	github.com/jcmturner/gofork v1.7.6 // indirect
 	github.com/jcmturner/gokrb5/v8 v8.4.3 // indirect
 	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/k-sone/critbitgo v1.4.0 // indirect
-	github.com/klauspost/compress v1.15.14 // indirect
+	github.com/klauspost/compress v1.15.15 // indirect
 	github.com/libp2p/go-reuseport v0.2.0 // indirect
-	github.com/magiconair/properties v1.8.6 // indirect
-	github.com/matttproud/golang_protobuf_extensions v1.0.2 // indirect
+	github.com/magiconair/properties v1.8.7 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
-	github.com/osrg/gobgp/v3 v3.7.0 // indirect
-	github.com/paulmach/orb v0.8.0 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/osrg/gobgp/v3 v3.11.0 // indirect
+	github.com/paulmach/orb v0.9.0 // indirect
 	github.com/pelletier/go-toml v1.9.5 // indirect
-	github.com/pelletier/go-toml/v2 v2.0.5 // indirect
+	github.com/pelletier/go-toml/v2 v2.0.6 // indirect
 	github.com/pierrec/lz4/v4 v4.1.17 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/prometheus/client_model v0.3.0 // indirect
-	github.com/prometheus/common v0.37.0 // indirect
-	github.com/prometheus/procfs v0.8.0 // indirect
+	github.com/prometheus/common v0.40.0 // indirect
+	github.com/prometheus/procfs v0.9.0 // indirect
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
 	github.com/segmentio/asm v1.2.0 // indirect
 	github.com/shopspring/decimal v1.3.1 // indirect
 	github.com/sirupsen/logrus v1.9.0 // indirect
-	github.com/spf13/afero v1.9.2 // indirect
+	github.com/spf13/afero v1.9.4 // indirect
 	github.com/spf13/cast v1.5.0 // indirect
 	github.com/spf13/jwalterweatherman v1.1.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	github.com/spf13/viper v1.13.0 // indirect
-	github.com/subosito/gotenv v1.4.1 // indirect
+	github.com/spf13/viper v1.15.0 // indirect
+	github.com/subosito/gotenv v1.4.2 // indirect
 	github.com/vishvananda/netlink v1.2.1-beta.2 // indirect
-	github.com/vishvananda/netns v0.0.0-20220913150850-18c4f4234207 // indirect
-	go.opentelemetry.io/otel v1.11.2 // indirect
-	go.opentelemetry.io/otel/trace v1.11.2 // indirect
-	golang.org/x/crypto v0.1.0 // indirect
-	golang.org/x/net v0.1.0 // indirect
-	golang.org/x/sync v0.0.0-20220923202941-7f9b1623fab7 // indirect
-	golang.org/x/sys v0.1.0 // indirect
-	golang.org/x/text v0.4.0 // indirect
-	google.golang.org/genproto v0.0.0-20221024183307-1bc688fe9f3e // indirect
-	google.golang.org/grpc v1.50.1 // indirect
+	github.com/vishvananda/netns v0.0.4 // indirect
+	go.opentelemetry.io/otel v1.13.0 // indirect
+	go.opentelemetry.io/otel/trace v1.13.0 // indirect
+	golang.org/x/crypto v0.6.0 // indirect
+	golang.org/x/net v0.7.0 // indirect
+	golang.org/x/sync v0.1.0 // indirect
+	golang.org/x/sys v0.5.0 // indirect
+	golang.org/x/text v0.7.0 // indirect
+	google.golang.org/genproto v0.0.0-20230222225845-10f96fb3dbec // indirect
+	google.golang.org/grpc v1.53.0 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )