Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Musicbrainz reduce memory used by processing by chunks #165

Merged
merged 67 commits into from
Oct 25, 2024
Merged

Musicbrainz reduce memory used by processing by chunks #165

merged 67 commits into from
Oct 25, 2024

Commits on Aug 19, 2024

  1. refactor: more ignored keys

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    aca1887 View commit details
    Browse the repository at this point in the history

  2. refactor: avoid new allocation

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    110263b View commit details
    Browse the repository at this point in the history

  3. doc: explain if statement

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    f9d2091 View commit details
    Browse the repository at this point in the history

  4. refactor: extract and convert in chunks

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    f72eb1e View commit details
    Browse the repository at this point in the history

  5. fix: write header

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    48adb26 View commit details
    Browse the repository at this point in the history

  6. refactor: if not first level, then don't extract name

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    87d9fe5 View commit details
    Browse the repository at this point in the history

  7. refactor: refresh values list

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    6659d28 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Update convert_to_csv.py

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    e79de48 View commit details
    Browse the repository at this point in the history

  2. Revert "Update convert_to_csv.py" 

    This reverts commit e79de48.
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    6c565be View commit details
    Browse the repository at this point in the history

  3. refactor: read jsonl in chunks

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    088c40d View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Merge branch 'main' into musicbrainz-reduce-memory-used

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    9ff292f View commit details
    Browse the repository at this point in the history

  2. test: add print tests

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    2cbea9b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    7ae3914 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. fix: delete buggy code

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    04b3c41 View commit details
    Browse the repository at this point in the history

  2. style: delete print test statements

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    8203c71 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. fix: memory bug and header writting

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    6e51f96 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    e23ed58 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into musicbrainz-reduce-memory-used

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    740d365 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into musicbrainz-reduce-memory-used

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    c0d1c78 View commit details
    Browse the repository at this point in the history

  5. test: delete unused test files

    @candlecao
    candlecao committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    6c6f920 View commit details
    Browse the repository at this point in the history

  6. Update genre.csv

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    99ff276 View commit details
    Browse the repository at this point in the history

  7. fix: header problem by making a temp csv

    @candlecao
    candlecao committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    e757df8 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Update mapping.json 

    change the type of recording from Q49017950 to Q482994
    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    805c464 View commit details
    Browse the repository at this point in the history

  2. mapping: full columns mapping

    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    81f34e7 View commit details
    Browse the repository at this point in the history

  3. mapping: fill the empty mappings for updated CSVs

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    92bfaf1 View commit details
    Browse the repository at this point in the history

  4. refactor: ignore iso codes since they are duplicated info

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    581e86a View commit details
    Browse the repository at this point in the history

  5. refactor: recognize special math character

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    6422088 View commit details
    Browse the repository at this point in the history

  6. doc: add made-up URLs

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    89371de View commit details
    Browse the repository at this point in the history

  7. optimize: simplify input of convert_to_csv.py

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    e93147e View commit details
    Browse the repository at this point in the history

  8. fix: syntax error

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    8dc8814 View commit details
    Browse the repository at this point in the history

  9. fix: entity type is the last part of file path

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    397e95e View commit details
    Browse the repository at this point in the history

  10. fix: output pathname

    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    0759de7 View commit details
    Browse the repository at this point in the history

  11. doc: update manual based on changes that removed commandline arguments

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    10f62c2 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @candlecao
    candlecao committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    738a7ed View commit details
    Browse the repository at this point in the history

  13. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    ceffbf1 View commit details
    Browse the repository at this point in the history

  14. style: delete unused test code

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    66133e8 View commit details
    Browse the repository at this point in the history

  15. doc: update docstring input to the correct number

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    7cb07d0 View commit details
    Browse the repository at this point in the history

  16. refactor: use writer instead of f.write()

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    3f61a23 View commit details
    Browse the repository at this point in the history

  17. Revert "Update mapping.json" 

    This reverts commit 805c464.
    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    5a1783b View commit details
    Browse the repository at this point in the history

  18. doc: style update according to GPT

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    d47751e View commit details
    Browse the repository at this point in the history

  19. doc: add specification for chunk_size

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    3bad21e View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. refactor: change list to set

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    6231da5 View commit details
    Browse the repository at this point in the history

  2. fix: filename bug, readlines bug, opening wrong file bug

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    29dc943 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. fix: auto escape by using \t not comma

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    28243a5 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. feat: add quotechar to distinguish ", and , separator

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    37201ab View commit details
    Browse the repository at this point in the history

  2. refactor: temp should be a tsv

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    72b92e6 View commit details
    Browse the repository at this point in the history

  3. fix: correct a few temp.csv error

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    b64da8a View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fix: file suffix bug

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    241dea3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'musicbrainz-reduce-memory-used' of https://github.com/D… 

    …DMAL/linkedmusic-datalake into musicbrainz-reduce-memory-used
    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    956f924 View commit details
    Browse the repository at this point in the history

  3. mappings: update by Junjun

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3146891 View commit details
    Browse the repository at this point in the history

  4. Update .gitignore

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    b58e618 View commit details
    Browse the repository at this point in the history

  5. test: only include genre.csv since it's not updated often and is slow… 

    … to update
    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d6e48bf View commit details
    Browse the repository at this point in the history

  6. test: remove old test files

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    23a6861 View commit details
    Browse the repository at this point in the history

  7. Update .gitignore

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    222e715 View commit details
    Browse the repository at this point in the history

  8. Update .gitignore

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    dcaf73e View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' into musicbrainz-reduce-memory-used

    @candlecao
    candlecao committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    63c8914 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. doc: update lock file based on pyproject.toml

    @candlecao
    candlecao committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    8d4ef18 View commit details
    Browse the repository at this point in the history

  2. refactor: visualize file processing

    @candlecao
    candlecao committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    ed26803 View commit details
    Browse the repository at this point in the history

  3. feat: add multi-file export option

    @candlecao
    candlecao committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    622cefe View commit details
    Browse the repository at this point in the history

  4. mapping: add reconciled columns

    @candlecao
    candlecao committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    8e99189 View commit details
    Browse the repository at this point in the history

  5. doc: add other reconciled columns

    @candlecao
    candlecao committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    84e4b1b View commit details
    Browse the repository at this point in the history

  6. doc: add specification about multi-file output

    @candlecao
    candlecao committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    798d448 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. update mapping.json

    @candlecao
    candlecao committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    13e8601 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. refactor: add type recognition to musicbrainz

    @Yueqiao12Zhang
    Yueqiao12Zhang committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    6c6401e View commit details
    Browse the repository at this point in the history

  2. ignore: add ignore separate ttls

    @candlecao
    candlecao committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    d380d46 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into musicbrainz-reduce-memory-used

    @candlecao
    candlecao committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    c4be2a5 View commit details
    Browse the repository at this point in the history

  4. test: remove test print

    @candlecao
    candlecao committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    9fe4df5 View commit details
    Browse the repository at this point in the history