v1.1.1 - Requirements fix and Examples

Fixed some broken requirements, added an example/demo for Current 2023

v1.1.0 - Allow altering topic configs; metrics manager bugfix

You can now alter topic configs with the FluviiToolbox + CLI. Operates similarly to creating topics.

You can now pipe stuff into the Fluvii CLI (i.e "{my topic_dict}" | fluvii topics create)

Small bugfix around the metrics manager config class default where it wasn't actually using the defined class variable to initialize with.

Remove username/password requirement

Remove the requirement to provide a username and password for Kafka authentication (meant to remove this before).

Various bugfixes, cleanup, small internal feature expansions

Cleanup and Tweaks

• Add a few more generalized class-level attributes for default classes in the various Fluvii factories to enable further customization for those who need it.
• Further cleaned up the MetricsManager class
• Cleaned up some Transaction object arguments to be more generalized.
• Other various cleanup.

Bugfixes:

• Fixed incorrect AUTH env var references in the readme.
• Fixed incorrect environment variable config path for MetricsManagerConfig

v1.0.0 - configuration overhaul and factories

Insights/Context

While not as encompassing as we desired (proper unit testing is coming!), we felt this deserved the v1.0 mark as it solidifies the last really "shaky" part of Fluvii, which was how it was configured (wanted to get something out before Current 2022). More specifically, the user interface changed enough that we felt it warranted a major version bump.

Not to say the interface won't change from here, but we've put a lot of thought into how the code is organized, how users should interact with it, and how intuitive app configuration is. We don't envision anything drastic changing from here (we knew 1.0 was gonna bring fairly large changes, but it ended up even larger than expected!). Hopefully this structure will hold up for a while!

The main reason we went with a "Factory" approach is that we wanted to have an independent configuration layer; the factories only take config objects rather than any component classes or instances as arguments, and then the factory makes the components for you, making it less for the user to manage overall. You also know where to look for configuration as no settings are managed by FluviiApps themselves anymore, thus clearly delineating configuration from operation. This is also going to significantly aid in our unit testing efforts

We hope this is a step in the right direction for feeling like a proper library!

Large refactoring/shuffling of various objects:

• New folder: /fluvii/components
  • Now houses consumer, producer, schema-registry, sqlite, metrics, admin, auth, which were originally standalone
• Renamed folder: /fluvii/fluvii_app to /fluvii/app
• New folder: /fluvii/apps/transactions
  • Now houses the special transaction classes: Transaction, TransactionalConsumer, TransactionalProducer
• New folder: /fluvii/apps/helpers
  • Houses any helpers to the FluviiApp classes, keeping just the usable apps in the /fluvii/apps folder

Configuration changes:

• Every .config file now uses Pydantic
  • Every config can also be set via a single dotenv (see README)
  • Some configs were moved around, or removed from various component inits
• Allowed environment-level authentication configuration for both Kafka clients and schema registry

Interface/usage changes

• New "Factory" classes are now the primary app entrypoints for users rather than FluviiApp classes directly.
  • This leads to a minor interface/usage change from <1.0 versions. See README for details.
  • Producer and Consumer also have factories, should you need them
• (More of an FYI) components now have an explicit component.start() command so that you may inspect them before they actually initialize their various client-level behaviors. Defaults to automatically starting at object init.

README additions

• Updated previous examples based on above changes
• New examples and explanations around new config options
• Examples on how to use the CLI

new CLI; table offset tracking adjustment

• table offsets now follow the typical offset model of tracking "next to be added" rather than "last successfully added". This is to help avoid confusion and more easily track bugs, should they arise. By doing this, we fix a rare edge case where table recovery misses the last entry on the topic due to the tracking being "off by 1".

• added a new argument to SqliteFluvii to not allow table commits by default (allow_commits=False), which also added a step to avoid an initial table write and avoid issues with trying to use multiple readers at once.

• Added a small step for tabling apps in the abort_transaction step so that it can also track failed offsets in the table for the partitions kept during rebalances (transactions are aborted any time a rebalance happens). This is a QoL adjustment that helps avoid some unneccesary rebalances.

• There's now a Fluvii CLI! It allows you to add, delete, consume from, and produce to topics (assuming your environment is set up correctly). More documentation/examples coming soon around how to use it.

schema registry url parsing fix

Fixed a bug around how the schema registry auth url was being generated, along with some additional cleanup/hardening around that parsing.

have sqlite recover from a corrupt db

Sometimes, thing just happen, and the sqlite dict becomes corrupt. Added a feature to automatically delete it and kill the application so that it will rebuild upon restart.

Also added a producer poll operation during table recovery so the logs do not get spammed with authentication errors during long recovery times with Oauth clients.

fix to multi-msg-app

Small fix to multi-msg-app so that it doesn't throw an error when no messages are pulled

Singleton to Batch process swapping

Messages will now be handled as singletons (dictated by the consumer) until batching (via _batch_consume=False) is required. This will help speed up processing during low data flow situations.

While in singleton mode, after every message consume, the consumer will check what the message delay is ("now" - msg timestamp). If the delta is ever greater than allowed configured setting batch_consume_trigger_message_age_seconds, then batch mode will be initiated (_batch_consume=True)

Once batch consuming, it will ever only be swapped back to singletons when:
a) No messages are consumed in a poll, or
b) the message count of a finalized batch does not meet the max batch count.

This is to avoid doing time delta checks every message when in batch mode, and it's pretty safe to assume you want to "catch up" as much as possible before going back to singleton mode.

Additionally includes some minor bugfixes and exception handling around producer/transaction timeouts, ensuring the application gracefully terminates rather than the producer hanging in a FENCED state.