Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Musicbrainz reduce memory used by processing by chunks #165

Open
wants to merge 47 commits into
base: main
Choose a base branch
from
Open

Musicbrainz reduce memory used by processing by chunks #165

wants to merge 47 commits into from

Commits on Aug 19, 2024

  1. refactor: more ignored keys

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    aca1887 View commit details
    Browse the repository at this point in the history

  2. refactor: avoid new allocation

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    110263b View commit details
    Browse the repository at this point in the history

  3. doc: explain if statement

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    f9d2091 View commit details
    Browse the repository at this point in the history

  4. refactor: extract and convert in chunks

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    f72eb1e View commit details
    Browse the repository at this point in the history

  5. fix: write header

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    48adb26 View commit details
    Browse the repository at this point in the history

  6. refactor: if not first level, then don't extract name

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    87d9fe5 View commit details
    Browse the repository at this point in the history

  7. refactor: refresh values list

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    6659d28 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Update convert_to_csv.py

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    e79de48 View commit details
    Browse the repository at this point in the history

  2. Revert "Update convert_to_csv.py" 

    This reverts commit e79de48.
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    6c565be View commit details
    Browse the repository at this point in the history

  3. refactor: read jsonl in chunks

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    088c40d View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Merge branch 'main' into musicbrainz-reduce-memory-used

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    9ff292f View commit details
    Browse the repository at this point in the history

  2. test: add print tests

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    2cbea9b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    7ae3914 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. fix: delete buggy code

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    04b3c41 View commit details
    Browse the repository at this point in the history

  2. style: delete print test statements

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    8203c71 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. fix: memory bug and header writting

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    6e51f96 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    e23ed58 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into musicbrainz-reduce-memory-used

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    740d365 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into musicbrainz-reduce-memory-used

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    c0d1c78 View commit details
    Browse the repository at this point in the history

  5. test: delete unused test files

    @candlecao
    candlecao committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    6c6f920 View commit details
    Browse the repository at this point in the history

  6. Update genre.csv

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    99ff276 View commit details
    Browse the repository at this point in the history

  7. fix: header problem by making a temp csv

    @candlecao
    candlecao committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    e757df8 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Update mapping.json 

    change the type of recording from Q49017950 to Q482994
    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    805c464 View commit details
    Browse the repository at this point in the history

  2. mapping: full columns mapping

    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    81f34e7 View commit details
    Browse the repository at this point in the history

  3. mapping: fill the empty mappings for updated CSVs

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    92bfaf1 View commit details
    Browse the repository at this point in the history

  4. refactor: ignore iso codes since they are duplicated info

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    581e86a View commit details
    Browse the repository at this point in the history

  5. refactor: recognize special math character

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    6422088 View commit details
    Browse the repository at this point in the history

  6. doc: add made-up URLs

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    89371de View commit details
    Browse the repository at this point in the history

  7. optimize: simplify input of convert_to_csv.py

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    e93147e View commit details
    Browse the repository at this point in the history

  8. fix: syntax error

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    8dc8814 View commit details
    Browse the repository at this point in the history

  9. fix: entity type is the last part of file path

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    397e95e View commit details
    Browse the repository at this point in the history

  10. fix: output pathname

    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    0759de7 View commit details
    Browse the repository at this point in the history

  11. doc: update manual based on changes that removed commandline arguments

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    10f62c2 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    738a7ed View commit details
    Browse the repository at this point in the history

  13. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    ceffbf1 View commit details
    Browse the repository at this point in the history

  14. style: delete unused test code

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    66133e8 View commit details
    Browse the repository at this point in the history

  15. doc: update docstring input to the correct number

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    7cb07d0 View commit details
    Browse the repository at this point in the history

  16. refactor: use writer instead of f.write()

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    3f61a23 View commit details
    Browse the repository at this point in the history

  17. Revert "Update mapping.json" 

    This reverts commit 805c464.
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    5a1783b View commit details
    Browse the repository at this point in the history

  18. doc: style update according to GPT

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    d47751e View commit details
    Browse the repository at this point in the history

  19. doc: add specification for chunk_size

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    3bad21e View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. refactor: change list to set

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    6231da5 View commit details
    Browse the repository at this point in the history

  2. fix: filename bug, readlines bug, opening wrong file bug

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    29dc943 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. fix: auto escape by using \t not comma

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    28243a5 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. feat: add quotechar to distinguish ", and , separator

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    37201ab View commit details
    Browse the repository at this point in the history

  2. refactor: temp should be a tsv

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    72b92e6 View commit details
    Browse the repository at this point in the history

  3. fix: correct a few temp.csv error

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    b64da8a View commit details
    Browse the repository at this point in the history