Forcing Free Thread with Python 3.14

[kdschlosser]

as we know Python now offers the ability to run without the GIL being enabled. This is something that is exceedingly handy for large applications that are performance bound using the GIL. Multiprocessing can be a solution for some folks if you don't need to pass large amount of data between the processes. When you have an application where large amount of data would need to be passed between processes the cost of passing the data between the processes can become to high and cause the application to run slower than it would by simply handing everything in a single process. The solution for this is getting rid of the GIL so threads can actually run in parallel. Care needs to be given in order to do this but is nothing that is too off the wall crazy to put into place. Python offers all of the bits needed to ensure that data would not get corrupted when running without the GIL...

Where the problems start.... Instead of opting to let the user know if a module is able to be used without the GIL the developers of Python opted for a different approach. Me personally I do not like this approach because it allows modules to turn the GIL on without ever letting the user know that the GIL is being used. There is no way for a user to turn off the use of the GIL once Python is running even after unloading the module that turned it on. The developers did however provide a mechanism to be able to force the GIL to be on or off regardless of what a module wants but the way that it needs to be done is not something that is not usable for an application packaged using pyinstaller. The first way is by having an environment variable (PYTHON_GIL) set before launching the application. This is not something that is usable because it would cause issues with Python being installed onto the same computer. so that's not something that is able to be used. The second option is by passing -Xgil=0/1 as an argument when launching the application. This would not work because the user would have to explicitly know to add that when running the application.

Just because a module marks itself as not being thread safe doesn't mean it cannot be used without the GIL. This is an assumption that is being made in Python that should not be made. It is treating the people that write Python code as if the are dummies and have no idea on how to write Python code so it will work correctly without having the GIL available. Yes, care would need to be taken to ensure that the same thread always gets used when accessing portions of a non thread safe module but it is not that difficult to do.

How can we go about telling pyinstaller to start the interpreter using the -Xgil=0 argument so even if external modules being used in an application are not thread safe Python can still be run without the use of the GIL?

[rokm]

--python-option "X gil=0", as per https://pyinstaller.org/en/stable/usage.html#cmdoption-python-option and https://pyinstaller.org/en/stable/spec-files.html#specifying-python-interpreter-options

[kdschlosser]

OK so how does this actually need to be written if wanting to set the option for a generated spec file?...

This is what the docs say...

Specify a command-line option to pass to the Python interpreter at runtime. Currently supports “v” (equivalent to “–debug imports”), “u”, “W ”, “X ”, and “hash_seed=”. For details, see the section “Specifying Python Interpreter Options” in PyInstaller manual.

If I want to set this when running python directly either of these variations will work.

--Xgil=0

--X="gil=0"

so when using the --python-option argument how does it need to be done?

--python-option=--X gil=0

--python-option=--Xgil=0

--python-option=X gil=0

--python-option=Xgil=0

--python-option="--X gil=0"

--python-option="--Xgil=0"

--python-option="--X=gil=0"

--python-option="--X=\"gil=0\""

--python-option="X=\"gil=0\""

--python-option="X \"gil=0\""

If running pyinstaller programmatically how does it need to be done? I have found that there needs to be an equals sign between the switch and the parameter when running it programmatically. so when want to include a hidden import I have to pas '--hidden-import=module_name' when it is done programmatically.

Documentation is not clear on how to do it if there is an equals sign involved and if the preceding -- needs to also be included or not. and if double quotes need to be added and if they do and passing it programmatically do those double quotes need to be escaped a specific way...

[kdschlosser]

From reading the section about modifying the spec file it appears like this just enables a pass through so pyinstaller is able to evaluate command line options that have been passed to the packaged program when it is run. If that a correct assessment what happens if nothing is passed? would nothing get forwarded when running the interpreter?

[rokm]

import PyInstaller.__main__
PyInstaller.__main__.run([
    '--clean',
    '--noconfirm',
    '--python-option=X gil=0',
    'program.py'
])

or

import PyInstaller.__main__
PyInstaller.__main__.run([
    '--clean',
    '--noconfirm',
    '--python-option', 'X gil=0',
    'program.py'
])

[rokm]

From reading the section about modifying the spec file it appears like this just enables a pass through so pyinstaller is able to evaluate command line options that have been passed to the packaged program when it is run. If that a correct assessment what happens if nothing is passed? would nothing get forwarded when running the interpreter?

The options passed via --python-option are stored in the PKG archive (as OPTION TOC entries), so that bootloader can pass them on to the python interpreter configuration API when it sets up the embedded interpreter.

The command-line options passed to the application executable when it is launched have no role in this; they are completely reserved for application's own purposes.