Use jupyter gateway to support remote jupyter server or server in docker images

[BoPeng]

#258 proposed the use of separate Python environments (e.g. conda environments) but in theory we could allow the start of kernels in a docker image, which would provide better insulation of toolsets. It would be helpful to investigate how this could be done, and if so, what SoS can do to support it.

[BoPeng]

telamonian from gitter pointed out that

On the lighter weight end of things, I've used this before: https://github.com/korniichuk/rk. If you can ssh into a host, rk can install a local kernel that will run on the host. Con: it hasn't been updated in years. Pro: when I used it, it worked (though that was also years ago).

On the heavier side, there's this: https://github.com/jupyter/kernel_gateway. Pro: up to date and maintained. Con: seems quite complicated to use for just spawning kernels. Apparently there's no way to just run kernels without also starting up a docker?
Probably the simplest thing would be to just run the server itself in the other environment, if you can

actually it turns out that the basic use case of kernel_gateway is lovely simple. On the remote, you run:

upyter kernelgateway
That starts up the kernel gateway server and spit out a url. Then wherever you're going to run your server > you just run:

jupyter lab --gateway-url=

The server will then start with the message Kernels will be managed by the Gateway server running at: . I do wonder why those instructions aren't prominently displayed in the kernel_gateway docs...

[BoPeng]

It seems that the proper way to support this is to implement SoS support for --gateway-url. Basically, with a %use --gateway-url URL --kernel --language we should connect to a kernel from a gateway. As shown in the doc, it is easy to turn any docker image with docker server to a gateway server. The conda case is a bit more complicated but basically users can start a proxy from the conda environment...

[BoPeng]

https://github.com/jupyter/notebook/blob/f68e34f198ccd93639458d1e1310fffe8c6ec858/notebook/gateway/managers.py#L299 is the relevant code for starting kernels through gateway.

[BoPeng]

Not quite sure if it works for remote gateways but it appears that the sos kernel works with local gateways.

[stevehaertel]

Would very much appreciate the ability to specify a remote kernel class

https://jupyter-enterprise-gateway.readthedocs.io/en/latest/kernel-library.html#a-k-a-library-mode

[BoPeng]

@stevehaertel My problem is that there does not seem to be a "standard" way to use remote kernels. I mean, if you can show me

1. How the remote kernel is set up.
2. How local server is set up to use the remote kernel (outside of SoS).

I can see how sos-notebook can be configured to talk to the remote kernel.

Note that the %use magic supports remote kernel so the most problem would be %get and %put, which would not work if data exchange was performed over disk and the local and remote server do not share a file system. It is possible to stop using local disks but the cost is rather high for this rarely used feature (remote kernel).