Add technical infrastructure documentation

[maxrjones]

Issue copied Joe's post at pangeo-data/pangeo#922 (comment):

pangeo.io/cloud.html - based on https://github.com/pangeo-data/pangeo/blob/master/docs/cloud.rst

The 2i2c hubs are winding down so it would be good if we had some pointers for folks to help them understand what kind of options and services they could use to achieve pangeo-like cloud deployments. It would be nice if this included everything from DIY to managed solutions.

It would also be good to think of cloud as more than a Jupyterhub environment. We should include "compute" and "data" here, explaining how Pangeo enables cloud native science.

[TomNicholas]

@scottyhq perhaps you might be able to advise here?

[scottyhq]

Firstly thanks for spearheading this website revamp @maxrjones , it looks great!

@scottyhq perhaps you might be able to advise here?

Sure, i could take a crack at this in the near future. Could borrow some material from this this synthesis from a workshop last symmer, but I think there is definitely more to add such as things like dask-cloudprovider, coiled, lithops, github codespaces/actions, etc. Of course keeping things up to date is hard (since that write-up last summer the planteary computer hub is gone and I'm sure other things have changed!)

help them understand what kind of options and services they could use to achieve pangeo-like cloud deployments

I know there was a far amount of relevant discussion on Pangeo Discourse too on these topics which should be considered. I think a main challenge will be to strike a balance between straightforward and comprehensive.

should include "compute" and "data" here, explaining how Pangeo enables cloud native science

For data maybe just link to https://guide.cloudnativegeo.org ?

[TomNicholas]

Of course keeping things up to date is hard

It is hard, and working out how to keep it up to date is a separate question, but for now let's have an explicit "Recommendations were last updated September 2024" so that it doesn't become misleading if it isn't kept perfectly up to date.

I know there was a far amount of relevant discussion on Pangeo Discourse too on these topics which should be considered.

We want to take that into account, but we also don't want to be bottlenecked by review-by-committee. I suggest we just draft something and link to the PR from any directly relavant pangeo discourse discussion threads, and state an intended timeline for merging. That way we put the onus on other people to suggest alterations if they aren't satisfied (and if they don't make suggestions we just merge) rather than the onus being on us to obtain their explicit approval. For specific products (e.g. Coiled) we would tag someone from the product team on the PR.

For data maybe just link to https://guide.cloudnativegeo.org/ ?

Good idea to de-duplicate!