Skip to content

Add python side input pattern #36883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Add python side input pattern #36883

wants to merge 4 commits into from
+293 −16

Conversation

@Arunodoy18
Copy link

@Arunodoy18 Arunodoy18 commented Nov 24, 2025

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@Arunodoy18
BEAM-13231: Add glob wildcard escaping utilities to FileSystems
385217f 
This commit adds support for escaping and unescaping glob wildcard characters
in file path specifications, addressing the issue where files with literal
glob metacharacters (*, ?, {, }) in their names cannot be matched.

Changes:
- Added escapeGlobWildcards(String spec) method to escape glob metacharacters
  by prefixing them with backslash
- Added unescapeGlobWildcards(String spec) method to remove backslash prefixes
  from escaped glob characters
- Added comprehensive test cases for both methods including round-trip testing

These utilities provide the foundation for allowing users to treat glob
metacharacters as literals when they appear in actual filenames.

Fixes BEAM-13231
@Arunodoy18
[BEAM-31422] Fix IllegalStateException in STORAGE_WRITE_API with CDC
bf37c0f 
When CDC (Change Data Capture) is enabled with STORAGE_WRITE_API method,
the system was incorrectly using PENDING streams instead of the default
stream, causing an IllegalStateException due to checkState validation.

Changes:
- Fixed StorageApiWriteUnshardedRecords to use default stream when CDC is enabled
- Added comprehensive test case to prevent regression

CDC requires default streams because PENDING streams don't support the
RowMutationInformation functionality needed for upserts and deletes.

Fixes apache#31422
@Arunodoy18
Fix triggeringFrequency documentation and validation for STORAGE_API_…
0950492 
…AT_LEAST_ONCE

- Updated JavaDoc to reflect that triggeringFrequency applies to FILE_LOADS,
  STORAGE_WRITE_API, and STORAGE_API_AT_LEAST_ONCE methods
- Fixed validation logic in expand() to require triggeringFrequency for
  STORAGE_API_AT_LEAST_ONCE when writing unbounded PCollections
- Removed conflicting warning that incorrectly stated STORAGE_API_AT_LEAST_ONCE
  ignores triggeringFrequency
- Added comprehensive test cases to verify proper validation behavior:
  * Test failure when triggeringFrequency is missing for unbounded collections
  * Test success when triggeringFrequency is provided for unbounded collections
  * Test success for bounded collections without triggeringFrequency requirement
- Updated error messages to include all three supported methods

This ensures consistent behavior across all BigQuery write methods that support
triggered writes for unbounded collections.
@Arunodoy18
Add Python side input pattern for slowly updating global window
16062d3 
- Added Python implementation of slowly updating global window side inputs pattern
- Uses PeriodicSequence instead of GenerateSequence for broader compatibility
- Implements Latest.Globally().without_defaults() as mentioned in issue apache#35934
- Added test coverage for the new pattern
- Updated documentation to include Python example alongside Java

Fixes missing Python side input pattern in documentation.
Addresses issue apache#35934 by using Latest.Globally().without_defaults() for non-global windowing scenarios and ensuring Dataflow compatibility.
@gemini-code-assist
Copy link
Contributor

gemini-code-assist bot commented Nov 24, 2025

Summary of Changes

Hello @Arunodoy18, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances Apache Beam's capabilities across both its Python and Java SDKs. It provides a valuable new Python example for managing dynamic side inputs that update periodically, which is crucial for scenarios requiring fresh reference data. Concurrently, it refines the BigQueryIO connector in Java by improving the configuration and validation for streaming writes, particularly with the STORAGE_API_AT_LEAST_ONCE method, and by explicitly supporting Change Data Capture (CDC) operations. Additionally, new file system utilities are introduced to better handle special characters in file paths. These changes collectively improve the flexibility, robustness, and ease of use for developers building data pipelines with Apache Beam.

Highlights

  • New Python Side Input Pattern: Introduced a new Python example (side_input_slow_update_global_window) demonstrating how to implement a periodically updating side input using PeriodicSequence and global windows in Apache Beam. This pattern is useful for scenarios where side input data needs to be refreshed dynamically.
  • BigQueryIO Write Method Enhancements: Extended validation and configuration for BigQueryIO's STORAGE_API_AT_LEAST_ONCE write method, ensuring proper handling of triggeringFrequency for unbounded PCollections and removing a redundant warning.
  • BigQueryIO CDC Support: Enabled Change Data Capture (CDC) functionality for BigQueryIO's STORAGE_WRITE_API by passing the usesCdc flag, and added a test to prevent IllegalStateException when using CDC with this write method.
  • Glob Wildcard Utilities: Added escapeGlobWildcards and unescapeGlobWildcards utility methods to the Java FileSystems class for robust handling of file paths containing glob characters, ensuring they can be treated as literals.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@Arunodoy18
Copy link
Author

Arunodoy18 commented Nov 24, 2025

I have done some changes in this issues. Please review this and let me know any changes is needed.
Thank you mate.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 24, 2025

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

examples gcp io java python website

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Arunodoy18