Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP Iceberg integration #47

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open

WIP Iceberg integration #47

wants to merge 25 commits into from

Commits on Sep 9, 2024

  1. Writer test

    @margon8
    margon8 committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    f176d39 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Reader

    @margon8
    margon8 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    5dcb3a4 View commit details
    Browse the repository at this point in the history

  2. Integrating Icerberg as default table format

    @margon8
    margon8 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    5a2645a View commit details
    Browse the repository at this point in the history

  3. Working delta+iceberg side by side

    @margon8
    margon8 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    a167fa9 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Update Streaming Writer Test

    @braislchao
    braislchao committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    e7e976c View commit details
    Browse the repository at this point in the history

  2. Iceberg Reader Test with Batch

    @braislchao
    braislchao committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    7e630da View commit details
    Browse the repository at this point in the history

  3. Generic Iceberg Reader

    @braislchao
    braislchao committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    48e3cba View commit details
    Browse the repository at this point in the history

  4. Add Iceberg Stream Reader + tests. Catalog modifications pending.

    @braislchao
    braislchao committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    85f9305 View commit details
    Browse the repository at this point in the history

  5. Clean Generic Reader

    @braislchao
    braislchao committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    20dc7b8 View commit details
    Browse the repository at this point in the history

  6. Add Iceberg Stream append writer test

    @braislchao
    braislchao committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    0997930 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Clean Writer Test

    @braislchao
    braislchao committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    74bd63a View commit details
    Browse the repository at this point in the history

  2. Clean Metabolic functions

    @braislchao
    braislchao committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    9e5da27 View commit details
    Browse the repository at this point in the history

  3. Add Delta tests to check batch read compatibility

    @braislchao
    braislchao committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    7c40774 View commit details
    Browse the repository at this point in the history

  4. Implement generic Stream Reader + tests for Iceberg and Delta compati… 

    …bility
    @braislchao
    braislchao committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    d7bd30a View commit details
    Browse the repository at this point in the history

  5. Reformat GenericReaderTest

    @braislchao
    braislchao committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    876bdeb View commit details
    Browse the repository at this point in the history

  6. Simplify GenericReader

    @braislchao
    braislchao committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    5667f12 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Update Spark Config to enable Iceberg

    @braislchao
    braislchao committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    af8213d View commit details
    Browse the repository at this point in the history

  2. Add aws compatibility to sbt

    @braislchao
    braislchao committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    690481d View commit details
    Browse the repository at this point in the history

  3. Update writer tests with spark AWS configs

    @braislchao
    braislchao committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    fd84c10 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Local generic reader

    @braislchao
    braislchao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9f006a0 View commit details
    Browse the repository at this point in the history

  2. Initial glue reader test

    @braislchao
    braislchao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    60a4d79 View commit details
    Browse the repository at this point in the history

  3. Refactor tests to new use cases 

    Initial Glue config writer test
    @braislchao
    braislchao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    186824c View commit details
    Browse the repository at this point in the history

  4. Change test logging behaviour

    @braislchao
    braislchao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    380bd32 View commit details
    Browse the repository at this point in the history

  5. SecretNanagerTest token is not working

    @braislchao
    braislchao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ca41d07 View commit details
    Browse the repository at this point in the history

  6. Fix region var

    @braislchao
    braislchao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    90d1e1b View commit details
    Browse the repository at this point in the history