Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental transformations and sinks #520

Open
yruslan opened this issue Nov 18, 2024 · 0 comments
Open

Incremental transformations and sinks #520

yruslan opened this issue Nov 18, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented Nov 18, 2024 •
edited
Loading

Background

Incremental ingestion is already supported. Now we need to be able to do processing and send data to sinks incrementally as well.

Feature

Incremental transformations and sinks.

Example [Optional]

To be added...

Proposed Solution [Optional]

To be added...
@yruslan yruslan added the enhancement New feature or request label Nov 18, 2024
yruslan added a commit that referenced this issue Nov 20, 2024
@yruslan
#520 Write batchid information for raw tables in the metastore.
df02404
yruslan added a commit that referenced this issue Nov 20, 2024
@yruslan
#520 Write batchid information for raw tables in the metastore.
a753387
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Fix handling of incremental ingestion for 'raw' format of metast…
07d1cc8 
…ore tables.
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Fix handling of incremental ingestion for 'raw' format of metast…
fe94be8 
…ore tables. (#521)
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Prepare interfaces for incremental transformer processing.
e2034c1
yruslan added a commit that referenced this issue Nov 22, 2024
@yruslan
#520 Prepare interfaces for incremental transformer processing.
6f944bf
yruslan added a commit that referenced this issue Nov 22, 2024
@yruslan
#520 Make the default batchid field for tables having 'raw' format to…
d90bd7d 
… match raw file ingestion.
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Implement offset management for incremental processing.
e5755e7
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Allow gracefully adding 'pramen_batchid' field to metastore tabl…
b8b85b1 
…es consistently with the old behavior.
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Add unit tests for the incremental processing.
273e615
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Add unit tests for the incremental processing.
53acd43
yruslan added a commit that referenced this issue Nov 27, 2024
@yruslan
#520 Add chain-commit for incremental jobs with transient job depende…
aeac5df 
…ncies.
yruslan added a commit that referenced this issue Nov 28, 2024
@yruslan
#520 Add chain-commit transaction for incremental jobs with transient…
254dafa 
… job dependencies.
yruslan added a commit that referenced this issue Nov 28, 2024
@yruslan
#520 Move some common offset management code to OffsetManagerUtils.
c381cf5
yruslan added a commit that referenced this issue Nov 29, 2024
@yruslan
#520 Remove redundant flag from the metastore reader.
c9924f4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yruslan