Skip to content

Releases: Azure/azure-event-hubs-spark

azure-eventhubs-databricks_2.11:3.4.0

28 Nov 23:31
c9eb2bc
Compare
Choose a tag to compare

A new package is available that is compatible with Databricks Runtime Version 3.4. Internals of the EventHubsSource were changed to cast strings and maps to the internal types used in Structured Streaming. In our case we cast strings to UTF8String and maps to ArrayBasedMapData. This should avoid any casting issues users experience in their streaming jobs.

<groupdId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-databricks_2.11</artifactId>
<version>3.4.0</version>   


<!-- 
3.4 to signify the compatible Databricks version. 
We'll use the final digit for any patches to this release. 
-->

azure-eventhubs-spark_2.11:2.1.6

07 Nov 20:23
2769616
Compare
Choose a tag to compare

This is the final planned release supporting Spark 2.1. Development efforts will now be focused on Spark 2.2 and future releases. In this release, we:

  • Updated naming conventions to align with other artifacts in the EventHubs ecosystem.
  • Added URI support (in addition namespace name) to allow use with non-public clouds running EventHubs.

azure-eventhubs-databricks_2.11:3.3.0

07 Nov 01:08
4978458
Compare
Choose a tag to compare

A new package is available that is compatible with Databricks Runtime Version 3.3. Internals of the EventHubSource were changed to accommodate the version of Spark used by Databricks. No other changes were made.

<groupdId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-databricks_2.11</artifactId>
<version>3.3.0</version>   
<!-- 3.3 to signify the compatible Databricks version. 
     We'll use the final digit for future releases involving this Databricks Runtime 
-->

spark-streaming-eventhubs_2.11:2.1.5

17 Oct 23:10
Compare
Choose a tag to compare

This release includes the following changes to the spark-eventhubs library:

  • GetPartitionRuntimeInfo API call is made every batch to ensure RDDs are note made with stale data
  • Metadata progress files have been added for faster initialization
  • Recever-based streams are no longer available
  • Restful EventHubs Client has been removed
  • Upgraded to version 0.15.1 of the Event Hubs Java Client (see https://github.com/Azure/azure-event-hubs-java/releases)
  • Spark streaming jobs can now shutdown gracefully
  • Additional traces have been added
  • scalafmt has been adopted
  • misc. code clean up
  • repo improvements (re-written README, Contributor's Guide, Issue and PR templates, etc)

The corresponding Maven package can be found here:
http://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cspark-streaming-eventhubs_2.11%7C2.1.5%7C

This works with Spark 2.1.x and Spark 2.0.x. Open an issue if you have any questions/concerns/feedback of any kind! 👍

Spark Streaming Eventhubs Receiver v1.6.0

05 Oct 18:09
Compare
Choose a tag to compare

Spark Streaming Eventhubs Receiver v1.1.0

04 Oct 22:59
Compare
Choose a tag to compare
Merge pull request #7 from ppatierno/master

Added the credits parameter handling as "eventhubs.receiver.credits"

Spark Streaming Eventhubs Receiver v1.0.0

14 Jul 15:44
Compare
Choose a tag to compare
Merge pull request #5 from wmeints/api-improvements

Add implicit conversion methods