Merge pull request #90 from BelWue/v2

New Major Version v2 ### Maintenance - Moved GitHub orgs from bwNetFlow to BelWue @9er #89 - Upgraded golang version to 1.23.2 ### Bug fixes - waitgroup in lumberjack segment. (@JustKiddingCode #79) - vacuum_interval for prometheus Export (@ynHuber #85) ### New segments - diskbuffer (@JustKiddingCode #78) - mongodb (@ynHuber #81) ### Breaking Changes: - Protobuffer upgrade to match changes [introduced in goflow2 v2.2.1](netsampler/goflow2@3326554) (@ynHuber #86)
BelWue · Feb 3, 2025 · 48a66f9 · 48a66f9
2 parents 9cbc5e8 + 1bbb2af
commit 48a66f9
Show file tree

Hide file tree

Showing 103 changed files with 6,313 additions and 2,001 deletions.
diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml
@@ -29,9 +29,6 @@ jobs:
       packages: write
 
     steps:
-      - name: Checkout repository
-        uses: actions/checkout@v2
-
       # Login against a Docker registry except on PR
       # https://github.com/docker/login-action
       - name: Log into registry ${{ env.REGISTRY }}
@@ -42,6 +39,9 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
+      - name: Checkout repository
+        uses: actions/checkout@v2
+
       # Extract metadata (tags, labels) for Docker
       # https://github.com/docker/metadata-action
       - name: Extract Docker metadata
@@ -53,7 +53,7 @@ jobs:
       # Build and push Docker image with Buildx (don't push on PR)
       # https://github.com/docker/build-push-action
       - name: Build and push Docker image
-        uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
+        uses: docker/build-push-action@v6
         with:
           context: .
           push: ${{ github.event_name != 'pull_request' }}

diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
@@ -25,9 +25,9 @@ jobs:
       run: sudo apt-get install -y libpcap-dev
 
     - name: setup go
-      uses: actions/setup-go@v3
+      uses: actions/setup-go@v5.3.0
       with:
-        go-version: '1.20'
+        go-version: '1.23.2'
 
     - name: test
       run: go test ./...

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -23,9 +23,9 @@ jobs:
       run: sudo apt-get install -y libpcap-dev
 
     - name: setup go
-      uses: actions/setup-go@v3
+      uses: actions/setup-go@v5.3.0
       with:
-        go-version: '1.20'
+        go-version: '1.23.2'
 
     - name: test before release
       run: go test ./...
@@ -49,6 +49,10 @@ jobs:
     runs-on: ubuntu-latest
     needs: build
     steps:
+    - name: setup go
+      uses: actions/setup-go@v3
+      with:
+        go-version: '1.23.2'
     - name: get binaries from previous job
       uses: actions/download-artifact@v3
     - name: display downloaded files

diff --git a/CONFIGURATION.md b/CONFIGURATION.md
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM golang:1.20 AS builder
+FROM golang:1.23.2 AS builder
 RUN apt-get update
 
 # add local repo into the builder

diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
-# Flow Pipeline
+# Flow PipelineSrcAS
 
-[godoc](https://pkg.go.dev/github.com/bwNetFlow/flowpipeline)
+[godoc](https://pkg.go.dev/github.com/BelWue/flowpipeline)
 
 ## About The Project
 
@@ -17,10 +17,10 @@ processing stack into a single piece of software which can be configured to
 serve any function:
 
 * accepting raw Netflow (using [goflow2](https://github.com/netsampler/goflow2))
-* enriching the resulting flow messages ([examples/enricher](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/enricher))
-* writing to and reading from Kafka ([examples/localkafka](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/localkafka))
-* dumping flows to cli (e.g. [flowdump](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/flowdump))
-* providing metrics and insights ([examples/prometheus](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/prometheus))
+* enriching the resulting flow messages ([examples/enricher](https://github.com/BelWue/flowpipeline/tree/master/examples/enricher))
+* writing to and reading from Kafka ([examples/localkafka](https://github.com/BelWue/flowpipeline/tree/master/examples/localkafka))
+* dumping flows to cli (e.g. [flowdump](https://github.com/BelWue/flowpipeline/tree/master/examples/flowdump))
+* providing metrics and insights ([examples/prometheus](https://github.com/BelWue/flowpipeline/tree/master/examples/prometheus))
 * and many more...
 
 ## Getting Started
@@ -35,19 +35,19 @@ you'll either want to create one or call it from any example directory (and
 maybe follow the instructions there).
 
 ### Binary Releases
-Download our [latest release](https://github.com/bwNetFlow/flowpipeline/releases)
+Download our [latest release](https://github.com/BelWue/flowpipeline/releases)
 and run it, same as if you compiled it yourself.
 
 The default, dynamically linked version requires a reasonably recent system
-(glibc 2.32+, linux 5.11+ for `bpf`, ...) and comes with all features.
+(glibc 2.32+, linux 5.11+ for `bpf`, `mongodb` ...) and comes with all features.
 As a fallback option, the static binaries will work in older environments
 (CentOS 7, Debian 10, ...), but come without the segments that require
-CGO/dynamically linked code (`bpf`, `sqlite`, and plugin support, check
-[CONFIGURATION.md](https://github.com/bwNetFlow/flowpipeline/blob/master/CONFIGURATION.md)).
+CGO/dynamically linked code (`bpf`, `sqlite`, `mongodb` and plugin support, check
+[CONFIGURATION.md](https://github.com/BelWue/flowpipeline/blob/master/CONFIGURATION.md)).
 
 ### Container Releases
 A ready to use container is provided as `bwnetflow/flowpipeline`, you can check
-it out on [GitHub container registry](https://github.com/bwNetFlow/flowpipeline/pkgs/container/flowpipeline).
+it out on [GitHub container registry](https://github.com/BelWue/flowpipeline/pkgs/container/flowpipeline).
 
 Configurations referencing other files (geolocation databases for instance)
 will work in a container without extra edits. This is because the volume
@@ -62,7 +62,7 @@ docker run -v ./examples/xy:/config flowpipeline
 
 ## Configuration
 
-Refer to [CONFIGURATION.md](https://github.com/bwNetFlow/flowpipeline/blob/master/CONFIGURATION.md)
+Refer to [CONFIGURATION.md](https://github.com/BelWue/flowpipeline/blob/master/CONFIGURATION.md)
 for the full guide. Other than that, looking at the examples should give you a
 good idea what the config looks like in detail and what the possible
 applications are. For sake of completeness, here's another minimal example
@@ -85,7 +85,7 @@ If you find that the existing segments lack some functionality or you require
 some very specific behaviour, it is possible to include segments as a plugin.
 This is done using the `-p yourplugin.so` commandline option and your own
 custom module. See
-[examples/plugin](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/plugin)
+[examples/plugin](https://github.com/BelWue/flowpipeline/tree/master/examples/plugin)
 for a basic example and instructions on how to compile your plugin.
 
 Note that this requires CGO and thus will not work using the static binary

diff --git a/examples/README.md b/examples/README.md
@@ -2,7 +2,7 @@
 
 This collection of example configs is supposed to help users get started using
 different use cases. A grouped and alphabetically sorted
-[reference](https://github.com/bwNetFlow/flowpipeline/blob/master/CONFIGURATION.md),
+[reference](https://github.com/BelWue/flowpipeline/blob/master/CONFIGURATION.md),
 might be the best resource when trying to achieve a specific outcome, this
 short guide however tries to give new users some idea of what is possible with
 this tool and present existing users with additional options.
@@ -17,25 +17,25 @@ all inputs.
 This segment accesses local network interfaces using raw sockets, as for instance tcpdump does.
 
 Relevant examples are:
-* [./flowdump/bpf.yml](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/flowdump/bpf.yml) -- create a tcpdump style view with custom filtering from CLI using local
+* [./flowdump/bpf.yml](https://github.com/BelWue/flowpipeline/tree/master/examples/flowdump/bpf.yml) -- create a tcpdump style view with custom filtering from CLI using local
   interfaces
 
 
 ## `goflow`
 This segment allows listening for raw IPFIX, Netflow, or sFlow by using goflow2's API.
 
 Relevant examples are:
-* [./localkafka/write.yml](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/localkafka) -- emulate plain goflow2 and write flows to a Kafka topic for the following section to use
+* [./localkafka/write.yml](https://github.com/BelWue/flowpipeline/tree/master/examples/localkafka) -- emulate plain goflow2 and write flows to a Kafka topic for the following section to use
 
 
 ## `kafkaconsumer`
 This segment accesses streams of flows generated by another pipeline using
 `kafkaproducer` or [goflow2](https://github.com/netsampler/goflow2).
 
 Relevant examples are:
-* [./flowdump/kafkaflowdump.yml](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/flowdump/kafkaflowdump.yml) -- create a tcpdump style view with custom filtering from CLI
-* [./flowdump/highlight.yml](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/flowdump/highlight.yml) -- create a tcpdump style view but use the filtering conditional to highlight desired flows instead of dropping undesired flows
-* [./enricher](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/enricher) -- enrich flows with various bits of data and store them back in Kafka
-* [./reducer](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/reducer) -- strip flows of fields and store them back in Kafka
-* [./splitter](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/splitter) -- distribute flows to multiple Kafka topics based on a field
-* [./anonymizer](https://github.com/bwNetFlow/flowpipeline/tree/master/examples/anonymizer) -- anonymize IP addresses using Crypto PAn
+* [./flowdump/kafkaflowdump.yml](https://github.com/BelWue/flowpipeline/tree/master/examples/flowdump/kafkaflowdump.yml) -- create a tcpdump style view with custom filtering from CLI
+* [./flowdump/highlight.yml](https://github.com/BelWue/flowpipeline/tree/master/examples/flowdump/highlight.yml) -- create a tcpdump style view but use the filtering conditional to highlight desired flows instead of dropping undesired flows
+* [./enricher](https://github.com/BelWue/flowpipeline/tree/master/examples/enricher) -- enrich flows with various bits of data and store them back in Kafka
+* [./reducer](https://github.com/BelWue/flowpipeline/tree/master/examples/reducer) -- strip flows of fields and store them back in Kafka
+* [./splitter](https://github.com/BelWue/flowpipeline/tree/master/examples/splitter) -- distribute flows to multiple Kafka topics based on a field
+* [./anonymizer](https://github.com/BelWue/flowpipeline/tree/master/examples/anonymizer) -- anonymize IP addresses using Crypto PAn
diff --git a/examples/plugin/printcustom.go b/examples/plugin/printcustom.go
@@ -6,7 +6,7 @@ import (
 	"fmt"
 	"sync"
 
-	"github.com/bwNetFlow/flowpipeline/segments"
+	"github.com/BelWue/flowpipeline/segments"
 )
 
 // This is a short example on how to write custom segments and load them as a plugin.

diff --git a/go.mod b/go.mod
@@ -1,57 +1,77 @@
-module github.com/bwNetFlow/flowpipeline
+module github.com/BelWue/flowpipeline
 
-go 1.20
+go 1.23.2
+
+replace github.com/BelWue/flowpipeline => github.com/BelWue/flowpipeline v1.3.1-0.20250127122013-c865e669d527
 
 require (
 	github.com/BelWue/bgp_routeinfo v0.0.0-20221004100427-d8095fc566dd
-	github.com/ClickHouse/clickhouse-go/v2 v2.6.3
-	github.com/Shopify/sarama v1.38.1
+	github.com/BelWue/flowfilter v0.0.0-20250129140213-b11ce00e4643
+	github.com/ClickHouse/clickhouse-go/v2 v2.30.1
+	github.com/IBM/sarama v1.45.0
 	github.com/Yawning/cryptopan v0.0.0-20170504040949-65bca51288fe
 	github.com/alouca/gosnmp v0.0.0-20170620005048-04d83944c9ab
 	github.com/asecurityteam/rolling v2.0.4+incompatible
-	github.com/banviktor/asnlookup v0.1.0
-	github.com/bwNetFlow/bpf_flowexport v0.0.0-20220515112212-cd8128615c05
-	github.com/bwNetFlow/flowfilter v0.0.0-20221025122858-60746fa15915
+	github.com/banviktor/asnlookup v0.1.1
 	github.com/bwNetFlow/ip_prefix_trie v0.0.0-20210830112018-b360b7b65c04
-	github.com/bwNetFlow/protobuf/go v0.0.0-20211004083441-61e193b4b342
 	github.com/dustin/go-humanize v1.0.1
 	github.com/elastic/go-lumber v0.1.1
+	github.com/go-co-op/gocron/v2 v2.15.0
 	github.com/google/gopacket v1.1.19
 	github.com/hashicorp/logutils v1.0.0
-	github.com/influxdata/influxdb-client-go/v2 v2.12.2
-	github.com/klauspost/compress v1.15.15
-	github.com/mattn/go-sqlite3 v1.14.16
-	github.com/netsampler/goflow2 v1.1.1
-	github.com/oschwald/maxminddb-golang v1.10.0
+	github.com/influxdata/influxdb-client-go/v2 v2.14.0
+	github.com/mattn/go-sqlite3 v1.14.24
+	github.com/netsampler/goflow2/v2 v2.2.1
+	github.com/oschwald/maxminddb-golang v1.13.1
 	github.com/patrickmn/go-cache v2.1.0+incompatible
-	github.com/prometheus/client_golang v1.14.0
-	github.com/rs/dnscache v0.0.0-20211102005908-e0241e321417
-	google.golang.org/protobuf v1.28.1
+	github.com/prometheus/client_golang v1.20.5
+	github.com/rs/dnscache v0.0.0-20230804202142-fc85eb664529
+	go.mongodb.org/mongo-driver v1.17.2
+	google.golang.org/protobuf v1.36.4
 	gopkg.in/yaml.v2 v2.4.0
 )
 
 require (
-	github.com/ClickHouse/ch-go v0.53.0 // indirect
-	github.com/alecthomas/participle/v2 v2.0.0-beta.1 // indirect
+	github.com/Shopify/sarama v1.38.1 // indirect
+	github.com/alecthomas/participle/v2 v2.1.1 // indirect
+	github.com/banviktor/go-mrt v0.0.0-20230515165434-0ce2ad0d8984 // indirect
+	github.com/jonboulle/clockwork v0.5.0 // indirect
+	github.com/montanaflynn/stats v0.7.1 // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/oapi-codegen/runtime v1.1.1 // indirect
+	github.com/robfig/cron/v3 v3.0.1 // indirect
+	github.com/rogpeppe/go-internal v1.13.1 // indirect
+	github.com/sagikazarmark/locafero v0.7.0 // indirect
+	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
+	github.com/sourcegraph/conc v0.3.0 // indirect
+	github.com/xdg-go/pbkdf2 v1.0.0 // indirect
+	github.com/xdg-go/scram v1.1.2 // indirect
+	github.com/xdg-go/stringprep v1.0.4 // indirect
+	github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
+	go.uber.org/multierr v1.11.0 // indirect
+	golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20250127172529-29210b9bc287 // indirect
+)
+
+require (
+	github.com/ClickHouse/ch-go v0.63.1 // indirect
 	github.com/alouca/gologger v0.0.0-20120904114645-7d4b7291de9c // indirect
-	github.com/andybalholm/brotli v1.0.5 // indirect
+	github.com/andybalholm/brotli v1.1.1 // indirect
 	github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cespare/xxhash/v2 v2.2.0 // indirect
-	github.com/cilium/ebpf v0.10.0 // indirect
-	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/deepmap/oapi-codegen v1.12.4 // indirect
-	github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/cilium/ebpf v0.17.2
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da // indirect
 	github.com/eapache/channels v1.1.0 // indirect
-	github.com/eapache/go-resiliency v1.3.0 // indirect
-	github.com/eapache/go-xerial-snappy v0.0.0-20230111030713-bf00bc1b83b6 // indirect
+	github.com/eapache/go-resiliency v1.7.0 // indirect
+	github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
 	github.com/eapache/queue v1.1.0 // indirect
-	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/fsnotify/fsnotify v1.8.0 // indirect
 	github.com/go-faster/city v1.0.1 // indirect
-	github.com/go-faster/errors v0.6.1 // indirect
-	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/go-faster/errors v0.7.1 // indirect
 	github.com/golang/snappy v0.0.4 // indirect
-	github.com/google/uuid v1.3.0 // indirect
+	github.com/google/uuid v1.6.0
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
 	github.com/hashicorp/go-uuid v1.0.3 // indirect
@@ -60,47 +80,40 @@ require (
 	github.com/jcmturner/aescts/v2 v2.0.0 // indirect
 	github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
 	github.com/jcmturner/gofork v1.7.6 // indirect
-	github.com/jcmturner/gokrb5/v8 v8.4.3 // indirect
+	github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
 	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
-	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/k-sone/critbitgo v1.4.0 // indirect
-	github.com/kaorimatz/go-mrt v0.0.0-20210326003454-aa11f3646f93 // indirect
-	github.com/libp2p/go-reuseport v0.2.0 // indirect
-	github.com/magiconair/properties v1.8.7 // indirect
-	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
+	github.com/klauspost/compress v1.17.11
+	github.com/libp2p/go-reuseport v0.4.0 // indirect
+	github.com/magiconair/properties v1.8.9 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
-	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v1.0.2 // indirect
-	github.com/osrg/gobgp/v3 v3.11.0 // indirect
-	github.com/paulmach/orb v0.9.0 // indirect
-	github.com/pelletier/go-toml v1.9.5 // indirect
-	github.com/pelletier/go-toml/v2 v2.0.6 // indirect
-	github.com/pierrec/lz4/v4 v4.1.17 // indirect
+	github.com/osrg/gobgp/v3 v3.33.0 // indirect
+	github.com/paulmach/orb v0.11.1 // indirect
+	github.com/pelletier/go-toml/v2 v2.2.3 // indirect
+	github.com/pierrec/lz4/v4 v4.1.22 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
-	github.com/prometheus/client_model v0.3.0 // indirect
-	github.com/prometheus/common v0.40.0 // indirect
-	github.com/prometheus/procfs v0.9.0 // indirect
+	github.com/prometheus/client_model v0.6.1 // indirect
+	github.com/prometheus/common v0.62.0 // indirect
+	github.com/prometheus/procfs v0.15.1 // indirect
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
 	github.com/segmentio/asm v1.2.0 // indirect
-	github.com/shopspring/decimal v1.3.1 // indirect
-	github.com/sirupsen/logrus v1.9.0 // indirect
-	github.com/spf13/afero v1.9.4 // indirect
-	github.com/spf13/cast v1.5.0 // indirect
-	github.com/spf13/jwalterweatherman v1.1.0 // indirect
+	github.com/shopspring/decimal v1.4.0 // indirect
+	github.com/sirupsen/logrus v1.9.3 // indirect
+	github.com/spf13/afero v1.12.0 // indirect
+	github.com/spf13/cast v1.7.1 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	github.com/spf13/viper v1.15.0 // indirect
-	github.com/subosito/gotenv v1.4.2 // indirect
-	github.com/vishvananda/netlink v1.2.1-beta.2 // indirect
-	github.com/vishvananda/netns v0.0.4 // indirect
-	go.opentelemetry.io/otel v1.13.0 // indirect
-	go.opentelemetry.io/otel/trace v1.13.0 // indirect
-	golang.org/x/crypto v0.6.0 // indirect
-	golang.org/x/net v0.7.0 // indirect
-	golang.org/x/sync v0.1.0 // indirect
-	golang.org/x/sys v0.5.0 // indirect
-	golang.org/x/text v0.7.0 // indirect
-	google.golang.org/genproto v0.0.0-20230222225845-10f96fb3dbec // indirect
-	google.golang.org/grpc v1.53.0 // indirect
+	github.com/spf13/viper v1.19.0 // indirect
+	github.com/subosito/gotenv v1.6.0 // indirect
+	github.com/vishvananda/netlink v1.3.0 // indirect
+	github.com/vishvananda/netns v0.0.5 // indirect
+	go.opentelemetry.io/otel v1.34.0 // indirect
+	go.opentelemetry.io/otel/trace v1.34.0 // indirect
+	golang.org/x/crypto v0.32.0 // indirect
+	golang.org/x/net v0.34.0 // indirect
+	golang.org/x/sync v0.10.0 // indirect
+	golang.org/x/sys v0.29.0
+	golang.org/x/text v0.21.0 // indirect
+	google.golang.org/grpc v1.70.0 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
-Original file line number
+Diff line change
@@ Expand Up / @@ -6,7 +6,7 @@ import ( @@
     	"fmt"
     	"sync"
-    	"github.com/bwNetFlow/flowpipeline/segments"
+    	"github.com/BelWue/flowpipeline/segments"
     )
     // This is a short example on how to write custom segments and load them as a plugin.
@@ Expand Down @@