feat: use cloud storage from amalthea

[olevski]

Adds cloud storage for the new amalthea operator and with the new csi rclone version.

/deploy renku=feat-jupyter-free-sessions amalthea-sessions=feat-add-cloud-storage renku-notebooks=feat-jupyter-free-sessions renku-gateway=feat-add-renku-refresh-token-for-data-service renku-ui=andrea/custom-launcher-jupyter-free extra-values=amalthea-sessions.deployCrd=false,amalthea-sessions.rcloneStorageClass=csi-rclone-387-secret-annotation #notest

[olevski]

This change is part of the following stack:

• feat: add new apispec for the new amalthea sessions #360
  • feat!: expand environments specification #338
    • feat: run notebooks in data service #375
      • feat: use cloud storage from amalthea #387 ◀
        • feat: perf improvements for amalthea sessions #411

_{Change managed by git-spice.}

[RenkuBot]

You can access the deployment of this PR at https://renku-ci-ds-387.dev.renku.ch

[Panaetius]

General Question: there's still #347 that does a bit of cleanup in cloud storage and probably will cause quite a bit of conflict with this, so I'm wondering what the best way would be for us to bring this together with your changes. There's also lots of changes @leafty is working on in regards to cloud storage, and 3 people working on the same bits of code at the same time is bound to cause issues. maybe we should sync?

[Panaetius]

question: historically, users could mount arbitrary, unsaved cloud storage and the UI would just post a body here, in which case it wouldn't have a storage_id. I don't think this is around in v2 but v1 might still use this. So I'm not entirely sure if this if statement will break things and we'd have to generate ad-hoc ids or if this is fine?

[olevski]

For this first implementation of the cloud storage in V2 lets keep things simple and not allow people to mount cloud storage that is unsaved. Then if needed we can add this functionality. This also needs some design because in theory we could allow this in many other places in the API where we currently expect something that exists in the DB. So I will make the storage ID required in the post request to launch a session. If a payload is provided then it just overrides the cloud storage saved in our DB.

[leafty]

Note: waiting for a rebase, so I can try to work from this branch.

[Panaetius]

When it comes to #347 I think we should split out the API breaking change and merge that at some other time.

but the API is currently broken and that it works is mostly an accident, any update to sanic or one of its components could break it. So yes, it's a breaking API change but at the same time it's more of a bugfix and unbreaking the API. I wouldn't hold off on this. And it is needed for that PR. Also SwissDataScienceCenter/renku-ui#3299 addressed those changes on the UI AND addresses the changes to storage, so there's a bit of a triangle dependency here.

[RenkuBot]

Tearing down the temporary RenkuLab deplyoment for this PR.