Release 2.50.0 lyft #65
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nder pushed by GH tests every 6 weeks. (apache#27807)
…he#27815) * Use correct concurrency condition to avoid incorrect interrupts * Another one
…GES.md (apache#27816) * mention prism in changes.md * ws lint * review comments --------- Co-authored-by: lostluck <[email protected]>
Removing myself from the IO reviewer list as I don't have much context on IO changes. With the recent reorg, I would have even less opportunity to explore this side :(
Previously these were implicitly lifted from the base dictionary. We may want to consider allowing that again, but it is easier to loosen restrictions than tighten them up once this is declared stable.
* Add a new gradle file. * Removed all legacy worker related tasks in build.gradle, adjust tasks for multiarch containers * Created new files for Java Dataflow ARM tests, still need to figure out Java version and Java home issues. * Remove unused code in build.gradle. * remove unused target, add build multiarch Java container flags, adjust build.gradle file. * Add the TAG as env variable shared by diff Java version tests. * Adjust dependencies. * Change READNE for Java versions and yml file for auth. * Made changes based on PR comments. * Made changes based on comments, standardize yml file. * Change github_job. * Update .github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml Co-authored-by: Danny McCormick <[email protected]> * Update .github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml Co-authored-by: Danny McCormick <[email protected]> * remove redundant underscore --------- Co-authored-by: Danny McCormick <[email protected]>
…e (Cron, Commit, Phrase) (apache#27482) * GoPortable PreCommit * GoPortable PreCommit * GoPortable PreCommit * beam_PreCommit_GoPortable * fix readme and run * timeout-minutes * step name added * new fix cron and worflow dispatch
…he#27826) Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.12.0 to 1.12.1. - [Release notes](https://github.com/mongodb/mongo-go-driver/releases) - [Commits](mongodb/mongo-go-driver@v1.12.0...v1.12.1) --- updated-dependencies: - dependency-name: go.mongodb.org/mongo-driver dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Upgrade Kryo coder from Kryo v4 to Kryo v5 * Update CHANGES.md
… in an explicit 'config' parameter.
…rtifacts containing guava (apache#27766)
* Add model manager class to hold many models at once * Remove extraneous always_proxy addition * Format/lint * Use ordereddict + one more tested behavior * use move_to_end * Type hints + doc update
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v3...v4) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix comment trigger against PR branch * encapsulate checkout logic into the action * updated setup action and readme file * updated setup action and readme file * updated workflow file
… Github Actions (apache#27541) * added beam_PreCommit_Java_Examples_Dataflow_Java11.yml job * added beam_PreCommit_Java_Examples_Dataflow_Java11.yml job * fixed formatting * added timeout for the beam_PreCommit_Java_Examples_Dataflow_Java11.yml job * updated the README file * added additional changes * merge master into beam_PreCommit_Java_Examples_Dataflow_Java11
Co-authored-by: lostluck <[email protected]>
Rtop logging
added more logs to bundle_processor.py
This reverts commit 8765f5a.
This reverts commit 8b736ff.
Remove all waittimeaggregation logging
added back rtop logs + more
remove pipeline.py logs
This reverts commit 13692c3.
[RTOP-2169] Adding support for kinesis consumer synchronization
Update S3Input to align with S3AndKinesisInput
This pull request includes significant updates to the `runners/flink` module, primarily focusing on upgrading dependencies and refactoring the Kafka-related code to use updated classes and methods. The most important changes include updating the dependencies in `flink_runner.gradle`, refactoring the `LyftFlinkStreamingPortableTranslations` class to use the `LyftKafkaSourceBuilder` and `KafkaSource`, and modifying the `ByteArrayWindowedValueSchema` class to implement `KafkaRecordDeserializationSchema`. --------- Co-authored-by: Kyle Bilton <[email protected]> Co-authored-by: Kyle Bilton <[email protected]>
Adds a `translateKafkaV2Input` which contains the new `KafkaSource` implementation, and `translateKafkaInput` which implements `FlinkKafkaConsumer`. Tested using `2.50.0-lyft202502211740175713dev`
![semgrep-code-lyft[bot]](https://avatars.githubusercontent.com/u/4269340?s=60&v=4){kind=small}
| go.uber.org/atomic v1.9.0 // indirect | ||
| go.uber.org/multierr v1.8.0 // indirect | ||
| go.uber.org/zap v1.21.0 // indirect | ||
| golang.org/x/net v0.8.0 // indirect |
There was a problem hiding this comment.
Risk: Affected versions of golang.org/x/net are vulnerable to Uncontrolled Resource Consumption. The HTTP/2 vulnerability occurs when a malicious client rapidly creates and resets requests, causing excessive server resource consumption. Despite the MaxConcurrentStreams setting limiting the total requests, resetting an ongoing request enables the attacker to initiate a new request concurrently.
Fix: Upgrade this library to at least version 0.17.0 at beam/playground/backend/go.mod:78.
Reference(s): GHSA-4374-p667-p6c8, CVE-2023-39325
💬 To ignore this, reply with:
• /fp <comment> for false positive
• /ar <comment> for acceptable risk
• /other <comment> for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by ssc-59363580-1c3f-4351-96ad-c59c611f9c64.
This PR adds watermark alignment to Kafka sources that use `FlinkKafkaInput`. When this is enabled, all sources assigned to a `watermark_group` will be "aligned" (i.e., we will pause reading from a given source until all watermarks are within `max_allowed_watermark_drift`). Without this, during backfills we read messages from topics at different rates, causing us to discard messages that we expect to be aligned. Note: this was successfully tested in backfill pipeline via a dev release (`2.50.0-lyft202502251740531643dev`). --------- Co-authored-by: Arda Kuyumcu <[email protected]>
Legal RiskThe following dependencies were released under a license that RecommendationWhile merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue. CC-BY-SA-4.0 GPL-2.0 LGPL-3.0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No +1 necessary, just wanted the team to have visibility into which branch to use