DRAFT: run EpiAutoGP direct from juliacall

[SamuelBrand1]

This is a draft/WIP PR for trying a maximalist approach to resolving #897 .

In this design, we can remove all the .jl files and rely on https://github.com/JuliaPy/pyjuliapkg for julia deps and run a minimal version of model directly using juliacall.

This means that the input/output is handled by python, and the actually forecasting code reduces the the string JULIA_HELPER_CODE. This could be a cleaner way of interfacing to NowcastAutoGP assuming that prod bugs don't surface in implementation checks.

NB: I haven't removed all the julia files in this draft yet. This is really a design example for @sbidari @dylanhmorris @damonbayer to consider.

Note that only surfaced pain point was some incompat with the dev dep group (see pyproject.toml)

EDIT: there is a second surfaced pain point about config on multithread jointly with python and julia.

[codecov]

Codecov Report

❌ Patch coverage is 30.24390% with 143 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.06%. Comparing base (cf71ee8) to head (f96ef26).

Files with missing lines	Patch %	Lines
pipelines/epiautogp/juliacall_runner.py	30.35%	117 Missing ⚠️
...s/tests/test_epiautogp_juliacall_runner_parquet.py	31.42%	24 Missing ⚠️
pipelines/epiautogp/forecast_epiautogp.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1068      +/-   ##
==========================================
- Coverage   59.50%   57.06%   -2.45%     
==========================================
  Files          32       34       +2     
  Lines        2109     2308     +199     
==========================================
+ Hits         1255     1317      +62     
- Misses        854      991     +137     

Flag	Coverage Δ
epiautogp	91.78% <ø> (ø)
hewr	76.98% <ø> (ø)
pipelines	51.95% <30.24%> (-2.41%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[SamuelBrand1]

Hmmmm multithreading sounds risky https://juliapy.github.io/PythonCall.jl/stable/juliacall/#py-multi-threading.

I'll leave that here, but I think this approach might be nice to look at but feels too risky. AutoGP relies heavily on multi-threaded.

[damonbayer]

What is the conflict? You would certainly want to be able to have all of these deps loaded for local development.

[SamuelBrand1]

cfa-dagster[dev] down the dep chain requires dagster-dg-core which requires tomlkit<0.13.3 whereas the juliacall in its dep chain has juliapkg>=0.1.21 which wants tomlkit>=0.13.3

[SamuelBrand1]

https://github.com/dagster-io/dagster/blob/6ff640471fa43fda9c2d0e38c00bc0fe7714e398/python_modules/libraries/dagster-dg-core/pyproject.toml#L21

[SamuelBrand1]

https://github.com/JuliaPy/pyjuliapkg/blob/0aa8af1049b5bcec99710da305bd029fb0b2b0a8/pyproject.toml#L9

[SamuelBrand1]

Although I'd quite like to know more about the "bug in this version" comment!

[damonbayer]

dagster-io/dagster#30513