Skip to content

test: pybind embed test framework (#199) #200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

alexallmont
Copy link
Contributor

  • Initial unit test is a placeholder to demonstrate pybind11::embed functionality, constructing a FreeTensor from a context in Python API and then checking the returned pointer in C++ API.
  • Outstanding FIXME in CMakeLists to review linkage; running more than import results in a double free or corruption (out) which I think is due to using external rougpy import rather than private embedding.

- Initial unit test is a placeholder to demonstrate pybind11::embed functionality, constructing a FreeTensor from a context in Python API and then checking the returned pointer in C++ API.
- Outstanding FIXME in CMakeLists to review linkage; running more than import results in a `double free or corruption (out)` which I think is due to using external rougpy import rather than private embedding.
@alexallmont alexallmont self-assigned this Mar 13, 2025
@alexallmont alexallmont linked an issue Mar 13, 2025 that may be closed by this pull request
- Experimental change to try running tests from PYBIND11_EMBEDDED_MODULE rather than `import roughpy`, i.e. not depend on finding an external .so file.
- This adds a new static library RoughPy_StaticLib which contains the bulk of what was in RoughPy_PyModule, but with PUBLIC rather than PRIVATE linkage. This currently raises compiler warnings which will need fixing if this approach is taken.
- The #defed out code at the end of `test_python_embed.cpp` still causes the same double-free issue as before, I assume because embedded pybind does none of the module caching that the interpreter does. Fix in progres.
- This reverts previous RoughPy_StaticLib change as the warnings caused by previous commit highlight legitimate linker issues.
- Instead a new cmake macro roughpy_module_sources adds all roughpy files to both RoughPy_PyModule and test_RoughPy_PyModule if tests enabled, then the latter project is completely isolated.
- The FIXME in tests is regarding poor reuse of one interpreter for all tests, wiping down globals between each. This avoids the `double free or corruption (out)` error but needs a neater solution.
- Trying to resolve `ModuleNotFoundError: No module named 'numpy'` on CI. Not sure if this is it but it's one of the few Win32 custom linkage fixes.
- This is in attempt to fix failing embedded python test for Win32 CI, as it's also set for pytest.
- Experimental change to try to fix import numpy error on Windows build; perhaps can be resolved by telling ctest exactly the PYTHONPATH at time of compilation
- The previous code demonstrated embedding the test in pybind statements. This instead now uses a python script and stores any test variables in globals so they can be retrieved in C++ later.
- Also the import of _roughpy_embed is left to each TEST_F if required.
- Rather than a singleton with a created-once check, use a global that is initialised in the single call to `SetUpTestSuite`
- Added a custom exception to scalars.h that stores the source and destination conversion types. These can then be reported along with captured leaf information in the roughpy conversion manager.
- Note point for review, these currently use `throw` rather than RPY_THROW partly because the latter macro doesn't seem to accept multiple args for new exception, and because I'm not sure if public python API exceptions should report C++ line numbers as in the macro.
- Added explicit checks for strings in conversion manager. The previous code was regarding strings as sequences which resulted in it recursing one level in and reporting "maximum nested depth reached in this context" which didn't clearly identify the bad type as cause.
- Includes unit test to confirm exceptions.
- Tidying for shorter name in line with existing tests.
- Add new CMake option ROUGHPY_BUILD_TEST_PYTHON_EMBED
- Option is enabled for dev preset.
- Option is enabled on CI unless Windows platform to avoid getting bogged down in paths issue as this test is more for local dev currently.
- Renamed `test_python_api` to `test_python_embed` for clearer meaning.
- The issue of C++ appearing in Python exceptions has been raised in a separate ticket #202
- The logic here became muddled in the switch to ROUGHPY_BUILD_TEST_PYTHON_EMBED when the original ROUGHPY_BUILD_TESTS condition was replaced, conflating the add_custom_command logic for both cases. This was causing a Windows CI `ModuleNotFoundError: No module named 'roughpy._roughpy'` error becasue the embed flag is disabled for Windows, in turn making it bypass the _roughpy copy operation.
- The original code has been restored and the embed test separately underneath, duplicating the original copy _roughpy logic for each platform.
@alexallmont alexallmont marked this pull request as ready for review April 22, 2025 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Embedded pybind11 tests
1 participant