Replaced json with orjson for large size loads

[nevoodoo]

Description

This is based off #658 which proposes to switch to orjson in place of the standard json library, for deserialising values that might be large. This switch should offer a significant cut in loading times, without introducing any code complexity.

Changes Made

• Replaced json.loads calls with orjson.loads where deemed appropriate. This replacement has not been done on low-impact calls that deserialise small objects.

Notes

• We should probably use orjson when introducing any new features/fixes that involving deserialising of large objects.

Closes

• Closes Use orjson in place of json #658

[illusional]

Awesome stuff! Note my 2 review comments before merging, but broadly supportive of this change. I was going to suggest looking at replacing it for Pydantic (I don't actually know which version we use), but looks like there is some ongoing discussion here: pydantic/pydantic#6388

So maybe best to leave that for now.

[illusional]

This is a fun linting one, soon to be resolved in metamist, but your default isort profile in VSCode is incorrect, in your VSCode settings.json you can add:

    "isort.args": [
        "--profile=black"
    ],

We intend to add this to metamist's pyproject.toml soon (to replicate https://github.com/populationgenomics/cpg-python-template-repo/blob/6e2da21e1012e6d2b15ba0c385da579a9b80b550/pyproject.toml#L7), but we were waiting for other big PRs to be finalised before introducing lots of merge conflicts.

[nevoodoo]

Reverted and fixed! I assume that replication to metamist will happen along with the new pre-commit hooks?

[nevoodoo]

Awesome stuff! Note my 2 review comments before merging, but broadly supportive of this change. I was going to suggest looking at replacing it for Pydantic (I don't actually know which version we use), but looks like there is some ongoing discussion here: pydantic/pydantic#6388

So maybe best to leave that for now.

I'm in this rabbit hole now since the discussion leads to mgspec which reports to be even quicker than orjson as per their benchmarks. I'll hold off on merging this PR until I've done some more tests to see what the better option is I think. I see Pydantic are also having discussions with the creator of mgspec so there's a good chance they may borrow from that library instead. Thoughts?

[illusional]

I don't feel well versed with the different python libraries, looks like the conversation is pretty fresh so happy to wait - (what version of Pydanctic do we use?)

[nevoodoo]

I don't feel well versed with the different python libraries, looks like the conversation is pretty fresh so happy to wait - (what version of Pydanctic do we use?)

Believe fastapi==0.85.1 and pydantic==1.10.13 from circa 2022. Current versions are fastapi==0.109.0 and pydantic==2.5.3

[illusional]

Awesome, I tried to bump our fastapi, but there's an issue with the openapi version and the generator we use #643 (I think this issue has more commentary).

Is this PR good to merge if we don't want to touch that side?

[nevoodoo]

Awesome, I tried to bump our fastapi, but there's an issue with the openapi version and the generator we use #643 (I think this issue has more commentary).

Is this PR good to merge if we don't want to touch that side?

I think this should be good to merge for now. mgspec is still relatively new so maybe it's something to be looked at again in the future :)