Make ImGui an even more serious research & development contender by improving Jupyter Notebook integration?

[pthom]

Motivation

At the moment, the python jupyter notebook integration is based on a kind of hack:

1. First, the application is launched in an external window:

2. Then, a screenshot is copied to the notebook when the window is closed.

This solution is not ideal, since it requires the user to have a window server, and thus this is incompatible with online notebook hosts, such as Google Colab. Most jupyter notebook will routinely use ipywidgets instead.

However, widgets provided by ipywidgets do not make much sense when the intention is to have similar widgets in a notebook and in an external (python) application.

Having a fully browser based solution using ImGui might be a worth solution.

Possible solution

By using a combination of imgui-js (see demo), and netImgui, it might perhaps be possible to have a solution like this (although it is complex)

Upon running a notebook, the following sequence of events would take place:

1. a javascript canvas is created in the notebook html dom
2. a javascript instance of "netImgui client" is launched and listen to messages from python
3. a python instance of "netImgui server" is launched and sends messages to the js client

I don't known whether this is really feasible or desirable, but the idea seemed worth being mentioned.

However, this might be only a partial solution, since the widgets from ipywidgets remain alive and are able to trigger callbacks even long after the cell execution ended; and this solution does not cover this.

[makkus]

I was wondering a similar thing, so I hope it's ok that I add my comment/question in here.

My understanding is that the emscripten feature of hello imgui only works for a 'pure' C++' (no Python) imgui project, right? So, I was wondering whether it'd be possible to run a imgui_bundle Python application packaged to be run with pyodide and get a similar experience (more sluggish obviously, but at least similar looking to the emscripten examples). My purpose would be to have a stand-alone web-app, using the same code as the desktop version.

But maybe that could also be an option for Jupyter integration?

I agree that the solution you outline could work, but I also agree that it does sound complex. And fragile... Not sure at all whether a pyodide version would be any better in that regard, of course...

I don't have much experience with wasm yet, and I don't know much about how Jupyter handles its DOM, so I guess there's a good chance none of what I'm thinking about is possible at all.

One issue could be that the Pyodide runtime can't be shared across cells/widgets, in which case there'd be a lot of those running which probably wouldn't be very nice, memory (and performance) wise. But assuming there'd be a way, maybe this could cut out the client/server thing you describe, and the (single) Pyodide runtime environment would be controlled by the 'imgui' ipywidget instances to manipulate the notebook html dom? Maybe pickle could be used as a serialization format, dunno...

[pthom]

I was wondering a similar thing, so I hope it's ok that I add my comment/question in here.

Of course

My understanding is that the emscripten feature of hello imgui only works for a 'pure' C++' (no Python) imgui project, right?

Absolutely

So, I was wondering whether it'd be possible to run a imgui_bundle Python application packaged to be run with pyodide and get a similar experience (more sluggish obviously, but at least similar looking to the emscripten examples). My purpose would be to have a stand-alone web-app, using the same code as the desktop version.

Having support for pyodide would be interesting in itself (i.e. jupyter notebook or not). If you decide to explore this, keep me in the loop please.

But maybe that could also be an option for Jupyter integration?
I agree that the solution you outline could work, but I also agree that it does sound complex. And fragile... Not sure at all whether a pyodide version would be any better in that regard, of course...

Well, when running notebook there is an external "iPython Kernel" linked to the notebook, i.e. a python server that executes the cells code and displays their output. When runing python code in pyodide, the link with this server would be broken (since only javascript would be running) and somehow would need to be restored via another form of interprocess communication.

Concerning notebook integration, the first thing to explore would be to improve integration with iPython. I had a related discussion here

[pthom]

Another possibility: remote frame buffer. with https://jupyter-rfb.readthedocs.io/en/stable/index.html

[pthom]

Also look at https://github.com/ggerganov/imgui-ws