Skip to content

Conversation

@techienidhipatel
Copy link

@techienidhipatel techienidhipatel commented Dec 15, 2025

Upgrade Bigtable Connector to Apache Flink 2.1

Fixes # (no existing issue - opening PR for discussion)

Summary

This PR upgrades the Flink Bigtable connector from Apache Flink 1.20.0 to 2.1.0, ensuring compatibility with the latest Flink 2.x series released in December 2025.

Changes Made

1. Core API Updates

  • Removed deprecated Sink.InitContext (BigtableSink.java)
    • Deleted the createWriter(Sink.InitContext) method which was already deprecated and throwing UnsupportedOperationException
    • The connector now uses only createWriter(WriterInitContext) which is the supported API in Flink 2.x
    • Reference: Flink 2.0 API Evolution

2. Version Upgrade

  • Updated Flink dependency from 1.20.0 to 2.1.0 in pom.xml

3. Example Code Fixes

  • Fixed import paths in all 8 example files to reflect Flink 2.0 package restructuring:
    • Changed: org.apache.flink.api.java.utils.ParameterTool
    • To: org.apache.flink.util.ParameterTool
    • Affected files:
      • WriteExactlyOnce.java
      • WriteGenericRecord.java
      • WriteGenericRecordNested.java
      • WriteRowData.java
      • WriteRowDataNested.java
      • WriteTableAPI.java
      • WriteTableAPINested.java
      • WriteWithFunction.java

4. Shaded JAR Configuration

  • Added shade-pom.xml for building a self-contained connector JAR
    • Bundles all Google Cloud dependencies
    • Relocates packages to avoid conflicts with Flink's internal libraries
    • Excludes Flink runtime dependencies (marked as provided)
    • Follows the same pattern as other Flink connectors

Testing

  • All 55 existing tests pass
  • Compiled successfully with Flink 2.1.0
  • Examples compile without errors
  • Shaded JAR builds successfully (63MB with all dependencies)
  • Tested locally with Flink 2.1 cluster writing to Bigtable staging instance

Compatibility Notes

  • Breaking Changes: None for end users - the connector API remains the same
  • Minimum Flink Version: Now requires Flink 2.1.0 or later
  • Java Version: Continues to support Java 11+ (as per parent POM)

Checklist

  • Tests pass
  • Code follows existing style and conventions
  • Examples updated and functional
  • No documentation changes needed (connector usage remains the same)
  • Shaded JAR configuration added for easier deployment

image

@google-cla
Copy link

google-cla bot commented Dec 15, 2025

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants