Skip to content

Python support using converter nodes#45

Merged
greenc-FNAL merged 71 commits intoFramework-R-D:mainfrom
wlav:python-support
Dec 19, 2025
Merged

Python support using converter nodes#45
greenc-FNAL merged 71 commits intoFramework-R-D:mainfrom
wlav:python-support

Conversation

@wlav
Copy link
Contributor

@wlav wlav commented Sep 23, 2025

This PR adds Python support through input/output converter nodes to solve the combinatorical problem. This approach makes all template instantiations compile-time only, thus cutting down on startup and memory overhead compared to using a JIT. They do add a per-algorithm run-time overhead.

Current support is for builtin types and std::vector of builtin types. Full support of all types requires some code generation from the eventual IDL. Life-times are managed using "lifelines", tying input underlying to output views, as Phlex can not track through converter nodes.

This PR adds support for registration that mimics C++ in naming, while remaining Pythonic and using reflection to cut down on boilerplate code where possible. It also adds a basic Pythonic representation of the configuration, with the actual remaining in C++. Error reporting translates Python tracebacks into C++ std::runtime_error exceptions, for handling and printing bye the C++ main.

The main open question is on interpreter finalization, which this PR punts on. This awaits resolution for when Phlex supports tokens.

This PR is ready to merge: the final failing checks are on code coverage of error reporting paths. These are in fact exercised: the tool is wrong.

@knoepfel knoepfel marked this pull request as draft September 24, 2025 12:45
@codecov
Copy link

codecov bot commented Oct 27, 2025

Codecov Report

❌ Patch coverage is 50.09823% with 254 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
plugins/python/src/modulewrap.cpp 40.77% 144 Missing and 39 partials ⚠️
plugins/python/src/errorwrap.cpp 56.81% 13 Missing and 6 partials ⚠️
plugins/python/src/configwrap.cpp 76.62% 12 Missing and 6 partials ⚠️
plugins/python/src/pymodule.cpp 64.58% 6 Missing and 11 partials ⚠️
plugins/python/src/lifelinewrap.cpp 0.00% 14 Missing ⚠️
phlex/app/phlex.cpp 75.00% 2 Missing ⚠️
phlex/configuration.hpp 85.71% 0 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (50.09%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project status has failed because the head coverage (76.09%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

@@            Coverage Diff             @@
##             main      #45      +/-   ##
==========================================
- Coverage   82.17%   76.09%   -6.09%     
==========================================
  Files         120      124       +4     
  Lines        2177     2719     +542     
  Branches      348      469     +121     
==========================================
+ Hits         1789     2069     +280     
- Misses        250      453     +203     
- Partials      138      197      +59     
Flag Coverage Δ
unittests 76.09% <50.09%> (-6.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
plugins/python/src/wrap.hpp 100.00% <100.00%> (ø)
phlex/configuration.hpp 94.44% <85.71%> (-5.56%) ⬇️
phlex/app/phlex.cpp 66.66% <75.00%> (+0.62%) ⬆️
plugins/python/src/lifelinewrap.cpp 0.00% <0.00%> (ø)
plugins/python/src/pymodule.cpp 64.58% <64.58%> (ø)
plugins/python/src/configwrap.cpp 76.62% <76.62%> (ø)
plugins/python/src/errorwrap.cpp 56.81% <56.81%> (ø)
plugins/python/src/modulewrap.cpp 40.77% <40.77%> (ø)

... and 14 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 42f6774...976cdd4. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

beojan
beojan previously requested changes Dec 15, 2025
Copy link
Contributor

@beojan beojan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's one change that I think is a blocker for merging this, which is that the new kind member on the configuration should have a name that makes it clear that it is not for external use.

@wlav wlav changed the title Proof-of-concept Python support using converter nodes Python support using converter nodes Dec 18, 2025
…neration (Framework-R-D#211)

* Catch exceptions in main to ensure graceful exit and coverage data generation

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@knoepfel knoepfel requested a review from beojan December 19, 2025 01:11
@knoepfel knoepfel dismissed beojan’s stale review December 19, 2025 14:24

The blocking issue was to rename configuration::kind(...) to something that signified it was intended to be an internal function. Wim renamed it to configuration::prototype_internal_kind(...).

Copy link
Contributor

@greenc-FNAL greenc-FNAL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a WIP addressing some of the remaining test coverage issues, but I believe this PR is good to go as is.

@greenc-FNAL greenc-FNAL merged commit 882ea59 into Framework-R-D:main Dec 19, 2025
34 of 36 checks passed
@wlav
Copy link
Contributor Author

wlav commented Dec 19, 2025

I have a WIP addressing some of the remaining test coverage issues, but I believe this PR is good to go as is.

There's also the minor issue that all Python APIs have error reporting, but many errors are unlikely/impossible to set up (e.g. only fail if out-of-memory). I'm not saying that coverage is at that point yet. :) But certainly some error paths take quite some thinking/setup to trigger.

Edit: additionally, for C++, when you provide a template that's supposed to be used for all builtin type, coverage will not ding you for not testing all said builtin types. In the current Python implementation, since they're all explicitly instantiated, they do need to be individually tested. I'll get to that once I have a provider in Python, such that I can generate all the annotated functions.

@wlav wlav deleted the python-support branch January 8, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants