PyPy
Replies: 6 comments 5 replies
-
|
This looks like a failure in test collection. Maybe @cfbolz or @rlamy have seen some of this. It might be some problem with pytest. Did you see this with earlier versions of pytest? |
Beta Was this translation helpful? Give feedback.
-
|
PyTest is written in pure Python code and so shouldn't be able to crash the interpreter regardless of the implementation. PyTest is importing nanobind-based extensions here. In contrast to normal C Python extensions that are relatively static (pointer to a table of methods/classes), nanobind is more dynamic. It dynamically creates a custom Python object for every function binding, a custom metaclass, and then it uses I suspect that it's this sequence that causes the crash, either because of nanobind making assumptions that are too specific to CPython, or a bug in PyPy's implementation of the CPython API. But I don't have a good handle on how to debug this since it only happens in the cloud 😓. Did you notice such crashes as well @mattip ? If I remember correctly, you are running the nanobind test suite against PyPy HEAD. |
Beta Was this translation helpful? Give feedback.
-
|
The weekly testing runs look pretty clean except for one week where |
Beta Was this translation helpful? Give feedback.
-
|
You're missing the NumPy and Eigen linear algebra library tests (skipped, likely because of missing dependencies). However, that is fine I think. The "contact surface" between nanobind and PyPy is well-covered by the other test. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Wenzel! From the traceback it looks like it's another instance of our 'pytest assert rewriting mystery bug': #3959 Therefore this very likely has nothing to do with nanobind directly. I'm incredibly frustrated with this issue, so far none of the instances have been reproducible on anyones laptop, they happen only on CI. therefore I haven't made much progress on understanding what's going wrong, despite various attempts :-(. Something that I've been meaning to try is to change one of the involved CI scripts to run this PyPy build: https://buildbot.pypy.org/pypy/pypy-lldebug-nightly.tar.bz2 Another option that I've been thinking about is to get CI to produce a coredump somehow and upload that as an artifact. |
Beta Was this translation helpful? Give feedback.
-
|
Dear @cfbolz, oh wow, I did not realize this was such a big thing. We run 6 PyPy configurations (two versions of PyPy -- each on macOS, Linux, and Windows). The build matrix fails with P=~1/3. Most often it's the linux build that crashes in my experience. Probably you could simply clone our repository with the GHA build matrix and replicate the linux configuration 10 times to have a solid repro. |
Beta Was this translation helpful? Give feedback.
-
|
I see the nanobind CI run installs Eigen, but not NumPy, so I will add that to the binary-testing CI run. The binary-testing run has an extra "Setup annotations on Linux" step, should that be removed? Looking back at the failures in the workflow, I see:
Going back further, I see more segfaults, mostly on ubuntu, I couldn't find any on macos. I wonder why the binary-testing CI runs don't see any failures when here they occur pretty often. Maybe it is something else entirely like the setup-cmake version? |
Beta Was this translation helpful? Give feedback.
-
|
The first CI run of pypy/binary-testing#49, which resyncs the CI workflow with upstream, crashed. I commented there on two things that might have changes: using a different cmake github action and some different build macros. Thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
Great. that's exactly the kind of crash that we are talking about here (https://github.com/pypy/binary-testing/actions/runs/8245574414/job/22549834434). I think you can remove all the CPython entries from the matrix since they are not relevant. In turn, you could insert some "dummy dimensions" into the matrix to get it to test PyPy many times with the same configuration to have more shots at triggering the issue. In the nanobind repo, I usually push the button to re-run crashed PyPy test runs when I get an email, and this re-run usually succeeds and causes things to be green. So the timeline on GitHub actions is not representative of how often this fails on the nanobind repo. I answered a question about compilation flags in the binary-testing PR. |
Beta Was this translation helpful? Give feedback.
-
|
just a quick update, I've managed to eventually reproduce it locally. the bug is still totally baffling, but at least it's not only on github ci any more. will write an issue to track it and document details how to reproduce it soon. |
Beta Was this translation helpful? Give feedback.
-
|
here's the issue: #4925 |
Beta Was this translation helpful? Give feedback.
-
Dear PyPy team,
I maintain the nanobind C++ <-> Python interop layer that also ensures PyPy cpyext compatibility through some entries in its build matrix. For a long time now, we've had issues with PyPy crashing spuriously while loading the extensions used in its test suite. You can see an example of such a crash here: https://github.com/wjakob/nanobind/actions/runs/8234561874/job/22516568891
It's unfortunately not very much to go by. Worse, I haven't been able to replicate this crash with separate testing on an intel Linux machine running exactly the same version of PyPy. Hence I wanted to ask a few questions:
setup-pythonaction). I am wondering if there is a difference between the binary they provide and what I download from the PyPy webpage.My hope with this ticket was to start a discussion and figure out what's wrong.
Thanks,
Wenzel
Beta Was this translation helpful? Give feedback.
All reactions