Bug: elabapi-python fails to initialize in Pyodide/JupyterLite due to _multiprocessing dependency

[vfreysz]

Description

The elabapi-python library cannot be used in web-based Python environments like Pyodide or JupyterLite. The initialization of elabapi_python.ApiClient fails because it attempts to instantiate multiprocessing.pool.ThreadPool.

The multiprocessing module is not available in Pyodide/WASM environments, which leads to a ModuleNotFoundError. This makes it impossible to use the library for client-side applications running in the browser.

Steps to Reproduce

The following minimal code, when run in a JupyterLite notebook or a Pyodide environment, will reproduce the error:

%pip install -q elabapi-python
import elabapi_python
from elabapi_python.rest import ApiException

# Initialize a configuration object
configuration = elabapi_python.Configuration()
configuration.host = "https://your.elab.instance/api/v2"
API_KEY = "your_api_key"

# The following line triggers the error
api_client = elabapi_python.ApiClient(configuration)

api_client.set_default_header(header_name='Authorization', header_value=API_KEY)

Expected Behavior

The ApiClient should initialize without errors, allowing the library to be used in a browser-based environment. A possible solution could be to avoid using ThreadPool or to have a fallback mechanism for environments where multiprocessing is not supported.

Actual Behavior

The code fails with the following traceback:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 11
      8 API_KEY = "your_api_key"
     10 # The following line triggers the error
---> 11 api_client = elabapi_python.ApiClient(configuration)
     13 api_client.set_default_header(header_name='Authorization', header_value=API_KEY)

File /lib/python3.12/site-packages/elabapi_python/api_client.py:68, in ApiClient.__init__(self, configuration, header_name, header_value, cookie)
     66 self.configuration = configuration
     67 # Use a thread pool for asynchronous requests
---> 68 self.pool = ThreadPool()
     69 self.rest_client = rest.RESTClientObject(configuration)
     70 self.default_headers = {}

File /lib/python3.12/multiprocessing/pool.py:1018, in ThreadPool.__init__(self, processes, initializer, initargs)
   1017 def __init__(self, processes=None, initializer=None, initargs=()):
-> 1018     Pool.__init__(self, processes, initializer, initargs)

File /lib/python3.12/multiprocessing/pool.py:212, in Pool.__init__(self, processes, initializer, initargs, maxtasksperchild, context)
    210 self._taskqueue = queue.SimpleQueue()
    211 self._cache = {}
--> 212 self._change_notifier = self._ctx.SimpleQueue()
    213 self._maxtasksperchild = maxtasksperchild
    214 self._wrap_exception = True

File /lib/python3.12/multiprocessing/context.py:114, in BaseContext.SimpleQueue(self)
    112 def SimpleQueue(self):
    113     '''Returns a queue object'''
--> 114     from .queues import SimpleQueue
    115     return SimpleQueue(ctx=self.get_context())

File /lib/python3.12/multiprocessing/queues.py:23
     19 import errno
     21 from queue import Empty, Full
---> 23 import _multiprocessing
     25 from . import connection
     26 from . import context

ModuleNotFoundError: No module named '_multiprocessing'

Environment

• Python via Pyodide (in JupyterLite)
• elabapi-python version installed via %pip install elabapi-python

[NicolasCARPi]

Hello,

You can try using the openapi generator instead: https://github.com/elabftw/elabapi-python/?tab=readme-ov-file#generate-the-library-using-code-generators

This was initially done to prevent this error. But then all current examples broke so we reverted it, because the api is different. It's still a valid option. You'll just need to figure out the correct calls because examples work with the swagger one.

I honestly wish there wasn't two different generators with two different set of problems -_-'...

[mhxion]

I believe I have gotten super close to making elAPI work on JupyterLite, but I get response status code 0, which is happening because the eLab instances I tested with (which includes demo.elabftw.net) blocks cross-origin policy. I don't have a test instance with cross-origin policy allowed. So, you will also likely run into this error regardless of which eLab client you use on JupyterLite because of CORS (unless your JupyterLab instance is on the same origin as your eLab instance).

You can try requests or httpx on JupyterLite with respective patches pyodide-http, pyodide-httpx which are known to work with Pyodide. elAPI under the hood is just using HTTPX. If someone has a CORS-disabled test instance, I am willing to give this another try ;)

[mhxion]

Success! Installed JupyterLite on my local machine, and tried sending requests with elAPI 2.3.3 on Chromium 138.0.7204.100.

The following NGINX headers need to be enabled via /etc/elabftw.yml for CORS to work:

- ALLOW_ORIGIN= http://127.0.0.1:8000 # Or, your server domain
- ALLOW_METHODS= GET, POST, PATCH, DELETE, OPTIONS, HEAD
- ALLOW_HEADERS=Content-Type, Authorization, Origin, X-Requested-With, Accept

Note

1. At the moment, pyodide-httpx doesn't work with elAPI 2.3.3 out-of-the-box. I had to make some minor modifications to pyodide-httpx. We will soon release a new version (2.4.0) of elAPI, which will. Better yet, I can make package pyodide-httpx into elAPI so elAPI works out-of-the box.
2. Firefox didn't play well with the client side requests on my machine. So the above didn't work on Firefox for me.

[mhxion]

I got some time yesterday to get back on this. Since my last comment, someone added a recipe for httpx to pyodid. So that means with the latest elAPI version 2.4.1, you can just do %pip install elapi on a JupyterLab Lite notebook, and it will just work out of the box! (You might also need %pip install elapi h2 if you want HTTP/2.0 support). You just need to make sure first that the NGINX headers in my earlier comment are enabled in your server.

Making elabapi-python work in the browser

The generated elabapi-python uses urllib3 for making the HTTP requests. urllib3 already has support for Pyodide, so we are good on that. But it seems this specific elabapi-python generation uses from multiprocessing.pool import ThreadPool quite heavily and Pyodide doesn't support multiprocessing or threading. Pyodide's workaround for multiprocessing also doesn't work.

So I tried generating the asyncio client that doesn't use multiprocessing at all with --additional-properties=library=asyncio.

openapi-generator generate -g python -i https://raw.githubusercontent.com/elabftw/elabftw/hypernext/apidoc/v2/openapi.yaml -o ./elabapi_python_async --additional-properties=library=asyncio

I named it elabapi-python-async, and its APIs remain the same as elabapi-python. I uploaded the .whl to a fork and installed the wheel with micropip on a local Jupyter Lite instance.

import micropip

await micropip.install("https://raw.githubusercontent.com/mhxion/elabapi-python/refs/heads/pyodide-support/dist/elabapi_python_async-0.1-py3-none-any.whl")

Since elabapi-python-async is not using microprocessing, the code that fails with elabapi-python runs fine.

But when I try to make the request, I get:

  File "/lib/python3.12/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/lib/python312.zip/asyncio/events.py", line 303, in getaddrinfo
    raise NotImplementedError

NotImplementedError

So elabapi-python-async uses aiohttp which uses asyncio. aiohttp does a have recipe for Pyodide (so I assume it is supported on the browser), but Pyodide's event loop cannot run the coroutine get_item, because it doesn't support the getaddrinfo that Python's asyncio does. In fact, there is an issue for it in Pyodide: pyodide/pyodide#5859. So this solution also does not work for now.

Alternatives: 2 other client generators that I found that don't use multiprocessing are: https://github.com/openapi-generators/openapi-python-client and https://github.com/MarcoMuellner/openapi-python-generator. openapi-python-client uses HTTPX just like elAPI and looks promising. But I didn't try it out, but it has a different API interface, so it is not going to be drop-in replacement for elabapi_python.

At this point the easy way out is likely going to be using requests or elapi ;) Hope my experiment here helps someone else trying this out.