Releases: streamdal/plumber
Releases · streamdal/plumber
Maintenance release
- Fixing version injection in build process
Shallow Envelope Protobuf Schemas / NATS JetStream
Features:
- Support for NATS JetStream
- Shallow Envelope Protobuf reading/writing
- For more information about shallow envelope protobufs, see https://www.confluent.io/blog/spring-kafka-protobuf-part-1-event-data-modeling/#shallow-envelope
Pass additional queue arguments to Rabbit using the --queue-arg flag
Example:
plumber read rabbit --exchange-name events --queue-name myqueue --binding-key \# --queue-durable --queue-arg x-dead-letter-exchange=mydeadletterv1.0.2
Help is now the default behavior when no commands or flags are specified
v1.0.1
Plumber version 1.0!
v1.0.0 Release Notes
This release has a number of breaking changes, be sure to read these release notes fully.
- Improvements
- Major code architecture overhaul
- Improved Kafka lag monitoring
- Improved read output
- Improved protobuf flags to remove verbosity
- Improved CLI flag parsing
- Rearchitected message bus interface for easier addition of new messaging systems in the future
- Features
- Kafka Sampling
New Feature: Kafka Sampling
We've added two new flags for kafka --sample-rate and --sample-interval-seconds
With these flags, you can now sample messages over a time period from a high-traffic kafka topic rather trying to read through a firehose of messages like it's the Matrix!
Command changes
plumber azure ...is nowplumber azure-service-bus ...plumber azure-eventhub ...is nowplumber azure-event-hub ...plumber cdc-mongo ...is nowplumber mongo ...plumber cdc-posgres ...is nowplumber postgres ...
CLI Flag Changes
Plumber has undergone a major CLI refactor and some flags have changed.
Changed flags
| Message Bus | Old Flag | New Flag | Notes |
|---|---|---|---|
--json |
--pretty |
||
--follow |
--continuous |
||
--avro-schema |
--avro-schema-file |
||
--protobuf-dir |
--protobuf-dirs |
||
--convert |
--convert-output |
||
--json-array |
--input-as-json-array |
||
--skip-verify |
--tls-skip-verify |
||
--insecure-tls |
--tls-skip-verify |
||
| Azure Service Bus | --subscription |
--subscription-name |
|
| GCP PubSub | --ack |
--ack-messages |
|
| GCP PubSub | --sub-id |
--subscription-id |
|
| Kafka | --username |
--sasl-username |
|
| Kafka | --password |
--sasl-Password |
|
| Kafka | --auth-type |
--sasl-type |
|
| Kafka | --rebalance-timeout |
--rebalance-timeout-seconds |
|
| Kafka | --commit-interval |
--commit-interval-seconds |
|
| Kafka | --group-id |
--consumer-group-name |
|
| MQTT | --tls-ca-file |
-tls-ca-cert |
Now accepts file path or certificate as a string |
| MQTT | --tls-client-cert-file |
--tls-client-cert |
Now accepts file path or certificate as a string |
| MQTT | --tls-client-key-file |
-tls-client-key |
Now accepts file path or certificate as a string |
| MQTT | --qos |
--qos-level |
|
| MQTT | --timeout |
--con-timeout-seconds |
|
| Nats | --creds-file |
--user-credentials |
|
| Nats | --tls-ca-file |
--tls-ca-cert |
Now accepts file path or certificate as a string |
| Nats | --tls-cert-file |
--tls-client-cert |
Now accepts file path or certificate as a string |
| Nats | --tls-key-file |
--tls-client-key |
Now accepts file path or certificate as a string |
| Nats | --address |
--dsn |
|
| Nats Streaming | --creds-file |
--user-credentials |
|
| Nats Streaming | --last |
--read-last-available |
|
| Nats Streaming | --all |
--read-all |
|
| Nats Streaming | --seq |
--read-sequence-number |
|
| Nats Streaming | --since |
--read-since |
|
| NSQ | --tls-ca-file |
--tls-ca-cert |
Now accepts file path or certificate as a string |
| NSQ | --tls-cert-file |
--tls-client-cert |
Now accepts file path or certificate as a string |
| NSQ | --tls-key-file |
--tls-client-key |
Now accepts file path or certificate as a string |
| Postgres CDC | --slot |
--replication-slot |
|
| Postgres CDC | --publisher |
--publisher-name |
|
| Pulsar | --name |
--subscription-name |
|
| Pulsar | --type |
--subscription-type |
|
| Pulsar | --auth-certificate-file |
--tls-client-cert |
|
| Pulsar | --auth-key-file |
--tls-client-key |
|
| RabbitMQ | --exchange |
--exchange-name |
|
| RabbitMQ | --queue-auto-delete |
--queue-delete |
|
| RabbitMQ | --routing-key |
--binding-key |
Read mode only. Write mode still uses correct nomenclature of --routing-key |
| Redis Streams | --id |
--write-id |
|
| Redis Streams | --start-id |
--ofset-start |
Options: oldest, latest |
Removed Flags
| Message Bus | Flag | Notes |
|---|---|---|
--thrift |
Moved to an option under --decode-type |
New Flags
| Message Bus | Flag | Description |
|---|---|---|
--decode-type |
Options: protobuf, avro, thrift |
|
| RabbitMQ | --exchange-type |
Exchange type to use when declaring a new exchange. Ex: topic, direct, fanout |
| RabbitMQ | --exchange-delete |
Whether to remove the exchange at disconnect |
| RabbitMQ | --exhcnage-declare |
Whether to declare a new exchange when connecting |
| RabbitMQ | --exchange-durable |
Create exchange as durable when declaring a new exchange |
| Kafka | --include-offset-info |
Displays consumer offset stats when reading |
| Kafka | --lag-consumer-group |
What consumer group to lookup lag for (required if --lag is set) |
Read/Relay from multiple GCP PubSub subscriptions at once
v0.35.1 Forgot to remove debugging statement
Read/Relay from multiple GCP PubSub subscriptions at once
Merge pull request #186 from batchcorp/blinktag/gcp_multisub Support for reading/relaying from multiple GCP pubsub subscriptions
Maintenance release
Bumping gjson version to v1.10.2 to address CVE
Fix RabbitMQ dynamic mode
v0.34.3 Fixing rabbit dynamic mode