From cff3121a6df6bfb69d78d5533ab649f66f9e69d9 Mon Sep 17 00:00:00 2001 From: Michal Rostecki Date: Wed, 6 Apr 2022 14:28:07 +0200 Subject: [PATCH] Split README into mdbook documentation README was getting really long. This change moves specific information about building and using PacketStreamer to separate mdbook subpages. It also adds information about using PacketStreamer with Suricata. Fixes: #4 Signed-off-by: Michal Rostecki --- .github/workflows/mdbook.yml | 26 +++ .gitignore | 3 + README.md | 282 +++++------------------------- docs/book.toml | 6 + docs/src/README.md | 1 + docs/src/SUMMARY.md | 12 ++ docs/src/configuration.md | 32 ++++ docs/src/quickstart/README.md | 7 + docs/src/quickstart/building.md | 34 ++++ docs/src/quickstart/docker.md | 46 +++++ docs/src/quickstart/kubernetes.md | 12 ++ docs/src/quickstart/local.md | 50 ++++++ docs/src/quickstart/vagrant.md | 53 ++++++ docs/src/tools/README.md | 3 + docs/src/tools/suricata.md | 24 +++ 15 files changed, 349 insertions(+), 242 deletions(-) create mode 100644 .github/workflows/mdbook.yml create mode 100644 docs/book.toml create mode 120000 docs/src/README.md create mode 100644 docs/src/SUMMARY.md create mode 100644 docs/src/configuration.md create mode 100644 docs/src/quickstart/README.md create mode 100644 docs/src/quickstart/building.md create mode 100644 docs/src/quickstart/docker.md create mode 100644 docs/src/quickstart/kubernetes.md create mode 100644 docs/src/quickstart/local.md create mode 100644 docs/src/quickstart/vagrant.md create mode 100644 docs/src/tools/README.md create mode 100644 docs/src/tools/suricata.md diff --git a/.github/workflows/mdbook.yml b/.github/workflows/mdbook.yml new file mode 100644 index 0000000..1ab1bfd --- /dev/null +++ b/.github/workflows/mdbook.yml @@ -0,0 +1,26 @@ +name: mdBook + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + +jobs: + build-and-deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Build book + run: | + cargo install mdbook + pushd docs + mdbook build + mdbook test + popd + - name: Deploy website + if: github.event_name == 'push' + uses: JamesIves/github-pages-deploy-action@4.1.4 + with: + branch: gh-pages + folder: book \ No newline at end of file diff --git a/.gitignore b/.gitignore index be56bca..b9bbb2f 100644 --- a/.gitignore +++ b/.gitignore @@ -25,3 +25,6 @@ go.work # Binary packetstreamer + +# Documentation +docs/book/ diff --git a/README.md b/README.md index 38f89cb..5fda83d 100644 --- a/README.md +++ b/README.md @@ -1,278 +1,76 @@ +[![Book](https://img.shields.io/website?url=https%3A%2F%2Fdeepfence.github.io%2FPacketStreamer%2F)](https://deepfence.github.io/PacketStreamer/) [![GitHub license](https://img.shields.io/github/license/deepfence/PacketStreamer)](https://github.com/deepfence/PacketStreamer/blob/master/LICENSE) [![GitHub stars](https://img.shields.io/github/stars/deepfence/PacketStreamer)](https://github.com/deepfence/PacketStreamer/stargazers) [![GitHub issues](https://img.shields.io/github/issues/deepfence/PacketStreamer)](https://github.com/deepfence/PacketStreamer/issues) [![Slack](https://img.shields.io/badge/slack-@deepfence-blue.svg?logo=slack)](https://join.slack.com/t/deepfence-community/shared_invite/zt-podmzle9-5X~qYx8wMaLt9bGWwkSdgQ) - # PacketStreamer -Deepfence PacketStreamer is a high-performance remote packet capture and collection tool. It is used by Deepfence's [ThreatStryker](https://deepfence.io/threatstryker/) security observability platform to gather network traffic on demand from cloud workloads for forensic analysis. +Deepfence PacketStreamer is a high-performance remote packet capture and +collection tool. It is used by Deepfence's [ThreatStryker](https://deepfence.io/threatstryker/) +security observability platform to gather network traffic on demand from cloud +workloads for forensic analysis. Primary design goals: * Stay light, capture and stream, no additional processing -* Portability, works across virtual machines, Kubernetes and AWS Fargate. Linux and Windows. - -PacketStreamer **sensors** are started on the target servers. Sensors capture traffic, apply filters, and then stream the traffic to a central reciever. Traffic streams may be compressed and/or encrypted using TLS. - -The PacketStreamer **reciever** accepts PacketStreamer streams from multiple remote sensors, and writes the packets to a local `pcap` capture file - -

- -PacketStreamer sensors collect raw network packets on remote hosts. It selects packets to capture using a BPF filter, and forwards them to a central reciever process where they are written in pcap format. Sensors are very lightweight and impose little performance impact on the remote hosts. PacketStreamer sensors can be run on bare-metal servers, on Docker hosts, and on Kubernetes nodes. - -The PacketStreamer receiver accepts network traffic from multiple sensors, collecting it into a single, central `pcap` file. You can then process the pcap file or live feed the traffic to the tooling of your choice, such as `Zeek`, `Wireshark` `Suricata`, or as a live stream for Machine Learning models. - -### When to use PacketStreamer - -PacketStreamer meets more general use cases than existing alternatives. For example, [PacketBeat](https://github.com/elastic/beats/tree/master/packetbeat) captures and parses the packets on multiple remote hosts, assembles transactions, and ships the processed data to a central ElasticSearch collector. [ksniff](https://github.com/eldadru/ksniff) captures raw packet data from a single Kubernetes pod. - -Use PacketStreamer if you need a lightweight, efficient method to collect raw network data from multiple machines for central logging and analysis. - -### Who uses PacketStreamer? - - * Deepfence [ThreatStryker](https://deepfence.io/threatstryker/) uses PacketStreamer to capture traffic from production platforms for forensics and anomaly detection. - - -# Quickstart: Build and Run PacketStreamer - -## Building PacketStreamer - -Build the `packetstreamer` binary using the `go` toolchain as follows: - -```bash -make -``` - -## Run a PacketStreamer receiver - -```bash -packetstreamer receiver --config [configuration_file] -``` - -You can use an example configuration file: - -```bash -packetstreamer receiver --config ./contrib/config/receiver-local.yaml -``` - -You can process the `pcap` output in a variety of ways: - -```bash -# pass the output file /tmp/dump_file to tcpdump: -tail -c +1 -f /tmp/dump_file | tcpdump -r - -``` - -```bash -# Edit the configuration to write to /dev/stdout, and pipe output to tcpdump: -./packet-streamer receiver --config contrib/receiver-stdout.yaml | tcpdump -r - -``` - -## Run a PacketStreamer sensor - -```bash -sudo packetstreamer sensor --config [configuration_file] -``` - -You can use an example configuration file: - -```bash -sudo packetstreamer sensor --config ./contrib/config/sensor-local.yaml -``` - -When running the sensor remotely, edit the configuration file to target the remote receiver. - -## Testing PacketStreamer - -You can generate some test traffic using your favorite load generator - `ab`, `wrk`, `httperf`, `vegeta`. For example, to use vegeta: - -```bash -# install vegeta -go install github.com/tsenart/vegeta@latest - -echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report -``` - -# Advanced Build Options - -Use the `RELEASE` parameter to strip the binary for a production environment: - -```bash -make RELEASE=1 -``` - -Use the `STATIC` parameter to statically-link the binary: - -```bash -make STATIC=1 -``` - -## Build using Docker - -Use the `docker-bin` target to build `packetstreamer` with Docker. The binary will be statically linked with `musl` and `libpcap`, making it portable across Linux distributions: - -```bash -make docker-bin - -# Alternatively, build a stripped release binary -make docker-bin RELEASE=1 -``` - -## Build a Container Image - -Use the `docker-image` target to build a container image: - -```bash -make docker-image - -# Alternatively, build a stripped release binary -make docker-image RELEASE=1 -``` - -## Deploy on Kubernetes - -PacketStreamer can be deployed on Kubernetes using Helm: - -```bash -kubectl apply -f ./contrib/kubernetes/namespace.yaml -helm install packetstreamer ./contrib/helm/ --namespace packetstreamer -``` - -By default, the Helm chart deploys a DaemonSet with sensor on all nodes and one receiver instance. For the custom configuration values, please refer to the [values.yaml file](/contrib/helm/values.yaml). - -# Advanced Test Scenarios - -## Testing on Docker - -PacketStreamer container images can be tested locally on Docker. - -### Receiver side - -```bash -docker run --rm -it \ - -v $(pwd)/contrib/config:/etc/packetstreamer \ - -v $HOME/container_tmp:/tmp \ - -p 8081:8081 \ - deepfenceio/deepfence_packetstreamer \ - receiver --config /etc/packetstreamer/receiver.yaml -``` - -### Sensor side - -```bash -docker run --rm -it \ - --cap-add=NET_ADMIN --net=host \ - -v $(pwd)/contrib/config:/etc/packetstreamer \ - deepfenceio/deepfence_packetstreamer \ - sensor --config /etc/packetstreamer/sensor-local.yaml -``` - -The sensor requires `--net=host` and `NET_ADMIN` capability in order to capture all of the packets on the host. - -```bash -echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report -``` - -The `pcap` dump file is available in `$HOME/container_tmp/dump_file`. - -## Testing on Vagrant - -On a single host, you may use [Vagrant](https://www.vagrantup.com) to run sensor and receiver hosts easily: - -Install Vagrant according to [the official instructions](https://www.vagrantup.com/downloads). By default, Vagrant uses Virtualbox; you should install [vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), using `vagrant plugin install vagrant-libvirt`. - -Start the two Vagrant VMs, `receiver` and `sensor`: - -```bash -vagrant up - -vagrant status -# Current machine states: -# -# receiver running (libvirt) -# sensor running (libvirt) -``` - -SSH to those VMs (in separate terminals) by using the following commands: - -```bash -vagrant ssh receiver -``` - -```bash -vagrant ssh sensor -``` - -On each, enter the source code directory: +* Portability, works across virtual machines, Kubernetes and AWS Fargate. Linux + and Windows. -### Receiver side +PacketStreamer **sensors** are started on the target servers. Sensors capture +traffic, apply filters, and then stream the traffic to a central reciever. +Traffic streams may be compressed and/or encrypted using TLS. -```bash -cd PacketStreamer -./packetstreamer receiver --config ./contrib/config/receiver-vagrant.yaml -``` +The PacketStreamer **receiver** accepts PacketStreamer streams from multiple +remote sensors, and writes the packets to a local `pcap` capture file -### Sensor side +

-```bash -cd PacketStreamer -sudo ./packetstreamer --config ./contrib/config/sensor-vagrant.yaml -``` +PacketStreamer sensors collect raw network packets on remote hosts. It selects packets +to capture using a BPF filter, and forwards them to a central reciever process +where they are written in pcap format. Sensors are very lightweight and impose +little performance impact on the remote hosts. PacketStreamer sensors can be +run on bare-metal servers, on Docker hosts, and on Kubernetes nodes. -Generate some live traffic +The PacketStreamer receiver accepts network traffic from multiple sensors, +collecting it into a single, central `pcap` file. You can then process the +pcap file or live feed the traffic to the tooling of your choice, such as +`Zeek`, `Wireshark` `Suricata`, or as a live stream for Machine Learning models. -```bash -echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report -``` +## When to use PacketStreamer -# PacketStreamer Configuration +PacketStreamer meets more general use cases than existing alternatives. For +example, [PacketBeat](https://github.com/elastic/beats/tree/master/packetbeat) +captures and parses the packets on multiple remote hosts, assembles +transactions, and ships the processed data to a central ElasticSearch +collector. [ksniff](https://github.com/eldadru/ksniff) captures raw packet +data from a single Kubernetes pod. -`packetstreamer` is configured using a yaml-formatted configuration file. +Use PacketStreamer if you need a lightweight, efficient method to collect raw +network data from multiple machines for central logging and analysis. -```yaml -input: # required in 'receiver' mode - address: _ip-address_ - port: _listen-port_ -output: - server: # required in 'sensor' mode - address: _ip-address_ - port: _listen-port_ - file: # required in 'receiver' mode - path: _filename_ -tls: # optional - enable: _true_|_false_ - certfile: _filename_ - keyfile: _filename_ -auth: # optional; receiver and sensor must use same shared key - enable: _true_|_false_ - key: _string_ -compressBlockSize: _integer_ # optional; default: 65 -inputPacketLen: _integer_ # optional; default: 65535 -logFilename: _filename_ # optional -pcapMode: _Allow_|_Deny_|_All_ # optional -capturePorts: _list-of-ports_ # optional -captureInterfacesPorts: _map: interface-name:port_ # optional -ignorePorts: _list-of-ports_ # optional -``` +## Who uses PacketStreamer? -You can find example configuration files in the [`/contrib/config/`](contrib/config/) folder. + * Deepfence [ThreatStryker](https://deepfence.io/threatstryker/) uses + PacketStreamer to capture traffic from production platforms for forensics + and anomaly detection. -# Get in touch +## Get in touch Thank you for using PacketStreamer. * [](https://join.slack.com/t/deepfence-community/shared_invite/zt-podmzle9-5X~qYx8wMaLt9bGWwkSdgQ) Got a question, need some help? Find the Deepfence team on Slack -* https://github.com/deepfence/PacketStreamer/issues: Got a feature request or found a bug? Raise an issue -* [productsecurity at deepfence dot io](SECURITY.md): Found a security issue? Share it in confidence +* https://github.com/deepfence/PacketStreamer/issues: Got a feature request or found a bug? Raise an issue +* [productsecurity *at* deepfence *dot* io](SECURITY.md): Found a security issue? Share it in confidence * Find out more at [deepfence.io](https://deepfence.io/) -# Security and Support +## Security and Support For any security-related issues in the PacketStreamer project, contact [productsecurity *at* deepfence *dot* io](SECURITY.md). Please file GitHub issues as needed, and join the Deepfence Community [Slack channel](https://join.slack.com/t/deepfence-community/shared_invite/zt-podmzle9-5X~qYx8wMaLt9bGWwkSdgQ). -# License +## License The Deepfence PacketStreamer project (this repository) is offered under the [Apache2 license](https://www.apache.org/licenses/LICENSE-2.0). diff --git a/docs/book.toml b/docs/book.toml new file mode 100644 index 0000000..c6bc03d --- /dev/null +++ b/docs/book.toml @@ -0,0 +1,6 @@ +[book] +authors = ["Michal Rostecki"] +language = "en" +multilingual = false +src = "src" +title = "PacketStreamer" diff --git a/docs/src/README.md b/docs/src/README.md new file mode 120000 index 0000000..fe84005 --- /dev/null +++ b/docs/src/README.md @@ -0,0 +1 @@ +../../README.md \ No newline at end of file diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md new file mode 100644 index 0000000..6a80a4f --- /dev/null +++ b/docs/src/SUMMARY.md @@ -0,0 +1,12 @@ +# Summary + +- [Introduction](./README.md) +- [Quickstart](./quickstart/README.md) + - [Building](./quickstart/building.md) + - [Using locally](./quickstart/local.md) + - [Using with Docker](./quickstart/docker.md) + - [Using on Kubernetes](./quickstart/kubernetes.md) + - [Using on Vagrant](./quickstart/vagrant.md) +- [Using with other tools](./tools/README.md) + - [Suricata](./tools/suricata.md) +- [Configuration](./configuration.md) diff --git a/docs/src/configuration.md b/docs/src/configuration.md new file mode 100644 index 0000000..2cc190c --- /dev/null +++ b/docs/src/configuration.md @@ -0,0 +1,32 @@ +# Configuration + +`packetstreamer` is configured using a yaml-formatted configuration file. + +```yaml +input: # required in 'receiver' mode + address: _ip-address_ + port: _listen-port_ +output: + server: # required in 'sensor' mode + address: _ip-address_ + port: _listen-port_ + file: # required in 'receiver' mode + path: _filename_ +tls: # optional + enable: _true_|_false_ + certfile: _filename_ + keyfile: _filename_ +auth: # optional; receiver and sensor must use same shared key + enable: _true_|_false_ + key: _string_ +compressBlockSize: _integer_ # optional; default: 65 +inputPacketLen: _integer_ # optional; default: 65535 +logFilename: _filename_ # optional +pcapMode: _Allow_|_Deny_|_All_ # optional +capturePorts: _list-of-ports_ # optional +captureInterfacesPorts: _map: interface-name:port_ # optional +ignorePorts: _list-of-ports_ # optional +``` + +You can find example configuration files in the [`/contrib/config/`](https://github.com/deepfence/PacketStreamer/tree/main/contrib/config) +folder. diff --git a/docs/src/quickstart/README.md b/docs/src/quickstart/README.md new file mode 100644 index 0000000..d05a582 --- /dev/null +++ b/docs/src/quickstart/README.md @@ -0,0 +1,7 @@ +# Quickstart + +- [Building](./building.md) +- [Using locally](./local.md) +- [Using with Docker](./docker.md) +- [Using on Kubernetes](./kubernetes.md) +- [Using on Vagrant](./vagrant.md) diff --git a/docs/src/quickstart/building.md b/docs/src/quickstart/building.md new file mode 100644 index 0000000..f566ebc --- /dev/null +++ b/docs/src/quickstart/building.md @@ -0,0 +1,34 @@ +# Building + +Build the `packetstreamer` binary using the `go` toolchain as follows: + +```bash +make +``` + +## Advanced Build Options + +Use the `RELEASE` parameter to strip the binary for a production environment: + +```bash +make RELEASE=1 +``` + +Use the `STATIC` parameter to statically-link the binary: + +```bash +make STATIC=1 +``` + +## Build using Docker + +Use the `docker-bin` target to build `packetstreamer` with Docker. The binary +will be statically linked with `musl` and `libpcap`, making it portable across +Linux distributions: + +```bash +make docker-bin + +# Alternatively, build a stripped release binary +make docker-bin RELEASE=1 +``` diff --git a/docs/src/quickstart/docker.md b/docs/src/quickstart/docker.md new file mode 100644 index 0000000..7bfd958 --- /dev/null +++ b/docs/src/quickstart/docker.md @@ -0,0 +1,46 @@ +# Using with Docker + +## Build a Container Image + +Use the `docker-image` target to build a container image: + +```bash +make docker-image + +# Alternatively, build a stripped release binary +make docker-image RELEASE=1 +``` + +## Testing on Docker + +PacketStreamer container images can be tested locally on Docker. + +### Receiver side + +```bash +docker run --rm -it \ + -v $(pwd)/contrib/config:/etc/packetstreamer \ + -v $HOME/container_tmp:/tmp \ + -p 8081:8081 \ + deepfenceio/deepfence_packetstreamer \ + receiver --config /etc/packetstreamer/receiver.yaml +``` + +### Sensor side + +```bash +docker run --rm -it \ + --cap-add=NET_ADMIN --net=host \ + -v $(pwd)/contrib/config:/etc/packetstreamer \ + deepfenceio/deepfence_packetstreamer \ + sensor --config /etc/packetstreamer/sensor-local.yaml +``` + +The sensor requires `--net=host` and `NET_ADMIN` capability in order to capture +all of the packets on the host. + +```bash +echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report +``` + +The `pcap` dump file is available in `$HOME/container_tmp/dump_file`. diff --git a/docs/src/quickstart/kubernetes.md b/docs/src/quickstart/kubernetes.md new file mode 100644 index 0000000..94bf7fa --- /dev/null +++ b/docs/src/quickstart/kubernetes.md @@ -0,0 +1,12 @@ +# Using on Kubernetes + +PacketStreamer can be deployed on Kubernetes using Helm: + +```bash +kubectl apply -f ./contrib/kubernetes/namespace.yaml +helm install packetstreamer ./contrib/helm/ --namespace packetstreamer +``` + +By default, the Helm chart deploys a DaemonSet with sensor on all nodes and one +receiver instance. For the custom configuration values, please refer to the +[values.yaml file](/contrib/helm/values.yaml). diff --git a/docs/src/quickstart/local.md b/docs/src/quickstart/local.md new file mode 100644 index 0000000..a9220a4 --- /dev/null +++ b/docs/src/quickstart/local.md @@ -0,0 +1,50 @@ +# Using locally + +## Run a PacketStreamer receiver + +```bash +packetstreamer receiver --config [configuration_file] +``` + +You can use an example configuration file: + +```bash +packetstreamer receiver --config ./contrib/config/receiver-local.yaml +``` + +You can process the `pcap` output in a variety of ways: + +```bash +# pass the output file /tmp/dump_file to tcpdump: +tail -c +1 -f /tmp/dump_file | tcpdump -r - +``` + +```bash +# Edit the configuration to write to /dev/stdout, and pipe output to tcpdump: +./packet-streamer receiver --config contrib/receiver-stdout.yaml | tcpdump -r - +``` + +## Run a PacketStreamer sensor + +```bash +sudo packetstreamer sensor --config [configuration_file] +``` + +You can use an example configuration file: + +```bash +sudo packetstreamer sensor --config ./contrib/config/sensor-local.yaml +``` + +When running the sensor remotely, edit the configuration file to target the remote receiver. + +## Testing PacketStreamer + +You can generate some test traffic using your favorite load generator - `ab`, `wrk`, `httperf`, `vegeta`. For example, to use vegeta: + +```bash +# install vegeta +go install github.com/tsenart/vegeta@latest + +echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report +``` diff --git a/docs/src/quickstart/vagrant.md b/docs/src/quickstart/vagrant.md new file mode 100644 index 0000000..a195fcf --- /dev/null +++ b/docs/src/quickstart/vagrant.md @@ -0,0 +1,53 @@ +# Using on Vagrant + +On a single host, you may use [Vagrant](https://www.vagrantup.com) to run +sensor and receiver hosts easily: + +Install Vagrant according to [the official instructions](https://www.vagrantup.com/downloads). +By default, Vagrant uses Virtualbox; you should install +[vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), using +`vagrant plugin install vagrant-libvirt`. + +Start the two Vagrant VMs, `receiver` and `sensor`: + +```bash +vagrant up + +vagrant status +# Current machine states: +# +# receiver running (libvirt) +# sensor running (libvirt) +``` + +SSH to those VMs (in separate terminals) by using the following commands: + +```bash +vagrant ssh receiver +``` + +```bash +vagrant ssh sensor +``` + +On each, enter the source code directory: + +### Receiver side + +```bash +cd PacketStreamer +./packetstreamer receiver --config ./contrib/config/receiver-vagrant.yaml +``` + +### Sensor side + +```bash +cd PacketStreamer +sudo ./packetstreamer --config ./contrib/config/sensor-vagrant.yaml +``` + +Generate some live traffic + +```bash +echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report +``` diff --git a/docs/src/tools/README.md b/docs/src/tools/README.md new file mode 100644 index 0000000..2b5e81a --- /dev/null +++ b/docs/src/tools/README.md @@ -0,0 +1,3 @@ +# Using with other tools + +- [Suricata](./suricata.md) diff --git a/docs/src/tools/suricata.md b/docs/src/tools/suricata.md new file mode 100644 index 0000000..9a29c11 --- /dev/null +++ b/docs/src/tools/suricata.md @@ -0,0 +1,24 @@ +# Suricata + +PacketStreamer receiver file output can be used for threat detection with +Suricata with the following command: + +```bash +tail -f /tmp/dump_file | suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin +``` + +It assumes that: + +* Suricata's configuration file is `/etc/suricata/suricata.yaml`. +* PacketStreamer receiver is configured with output file to `/tmp/dump_file`. + +Example receiver configuration: + +```yaml +input: + address: 0.0.0.0 + port: 8081 +output: + file: + path: /tmp/dump_file +```