Stream Reactor 8.1.16

Improvements to the HTTP Sink (#163)

* Improvements to the HTTP Sink

This PR introduces the following enhancements to the HTTP Sink:

1. **Queue Limiting**: We've set a limit on the queue size per topic to reduce the chances of an Out-of-Memory (OOM) issue. Previously the queue was unbounded and in a scenario where the http calls are slow and the sink gets more records than it clears, it would eventually lead to OOM.

2. **Offering Timeout**: The offering to the queue now includes a timeout. If there are records to be offered, but the timeout is exceeded, a retriable exception is thrown. Depending on the connector's retry settings, the operation will be attempted again. This helps avoid situations where the sink gets stuck processing a slow or unresponsive batch.

3. **Duplicate Record Handling**: To prevent the same records from being added to the queue multiple times, we've introduced a `Map[TopicPartition, Offset]` to track the last processed offset for each topic-partition. This ensures that the sink does not attempt to process the same records repeatedly.

4. **Batch Failure Handling**: The changes also address a situation where an HTTP call fails due to a specific input, but the batch is not removed from the queue. This could have led to the batch being retried indefinitely, which is now prevented.

In the near future, there will be a new PR to further reduce the code complexity around the batching approach and the boilerplate code.

* fix the unit test

* Rename variable

* Removes the invalid functional tests. a failed batch request is not retried anymore.

* Remove unused functions

---------

Co-authored-by: stheppi <[email protected]>

Stream Reactor 8.1.15

Ignore null records (#156)

* Ignore null records

Sometimes, custom made SMTs overwrite the payload and pass nulls to the connector leading to unintended results.

* Fix the linting error

---------

Co-authored-by: stheppi <[email protected]>

Stream Reactor 8.1.14

HTTP Sink Reporter Fixes

Stream Reactor 8.1.13

HTTP Sink Improvement

Stream Reactor 8.1.12

Data lake sink improvements. HTTP sink batching fix.

Stream Reactor 8.1.11

HTTP Sink Improvements

Stream Reactor 8.1.10

Merge pull request #144 from lensesio-dev/chore/dependency-upgrades

Chore/dependency upgrades

Stream Reactor 8.1.9

ServiceBus: Allowing for null contentType and other optional values (…

Stream Reactor 8.1.8

Release 8.1.8 : NPE fix;Store success/failure response codes in http …

Stream Reactor 8.1.7

NPE fix in reporter code