Cannot open the file: MusicBrainz releases

[Yueqiao12Zhang]

The csv2rdf script takes too much memories (the kernel returns Killed: 9 after running the program for a long time), when working with the releases data dumps.

[dchiller]

Could you post the traceback here? Is this the memory issue we talked about in the lab meeting?

[Yueqiao12Zhang]

Yes. I will copy the error message from the lab Mac after the meeting.

[Yueqiao12Zhang]

CHUNK_SIZE = 10000

if __name__ == "__main__":

    # the file must be from MusicBrainz's JSON data dumps.
    chunk = []

    with open(inputpath, "r", encoding="utf-8") as f:
        print(1) # It does not reach this line.
        for line in f:
            line_data = json.loads(line)  # Parse each line as a JSON object
            chunk.append(line_data)  # Add the JSON object to the current chunk

            # When the chunk reaches the desired size, process it
            if len(chunk) == CHUNK_SIZE:
                print(2)
                extract(chunk, {})
                chunk.clear()  # Reset the chunk
                convert_dict_to_csv(values)

            values.clear()

        # Process any remaining data in the last chunk
        if chunk:
            extract(chunk, {})
            chunk.clear()
            convert_dict_to_csv(values)

It works fine with smaller files.

[Yueqiao12Zhang]

(local) DDMAL-004:csv local$ python3 convert_to_csv.py ../data/raw/extracted_jsonl/mbdump/release release
[1]+  Terminated: 15          python3 convert_to_csv.py ../data/raw/extracted_jsonl/mbdump/release release
Killed: 9

[Yueqiao12Zhang]

Python does not return an error message.

[dchiller]

And so you know the place where it is erroring because it doesn't reach the print statement?

[dchiller]

Where is the python file you are running?

[Yueqiao12Zhang]

And so you know the place where it is erroring because it doesn't reach the print statement?

Yes,

Where is the python file you are running?

/csv2rdf/csv2rdf_single_subject.py

[Yueqiao12Zhang]

Problem solved! Very dump mistake 😭

[fujinaga]

Problem solved! Very dump mistake 😭

Great! No issues, we all make silly coding errors.