test with new pip dependency resolver?

[brainwane]

I'm writing on behalf of pip's maintainer team. I found you via a list of widely used Python packages and figured you might want your users to check whether some upcoming changes in pip will break their workflows.

We have now released pip 20.2. This release includes the beta of the next-generation dependency resolver. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the --use-feature=2020-resolver flag, and please ask your users to do the same. Please see our guide on how to test and migrate, and how to report issues. Please report bugs using this survey.

The new dependency resolver is off by default because it is not yet ready for everyday use.

We plan to make pip's next quarterly release, 20.3, in October 2020. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

More information at our blog post.

Thanks!

[zzzeek]

hi there -

I'm puzzled by this ticket for two reasons at the moment.

one is, I have a lot of packages on pypi and this package is by no means the most popular one. Why am I not getting a github issue for those?

another is, is the pypa team literally writing github issues to (selected? random?) projects by hand? how would this possibly scale?

if the answer to the previous question is, "of course not, we wrote a script to iterate through pypi pacakges that have [some kind of characteristics?] and post github issues", then.....why not make that script actually install each package in a container and see if it breaks ?

put another way, if pypa is making a change to packages that is so likely to be breaking, shouldn't pypa simply test the top 10000 packages against these changes to make sure they still install ?

[brainwane]

Hi there! Hope you're doing well. Sorry to have puzzled you.

I wrote personal notes to 5-10 projects yesterday that I chose by looking at some lists of popular/widely used packages, and did not mean for this approach to scale. I'm trying out a few things to try to get the word out about the changes, and then once we have a few more bits of feedback, we'll be trying some more automated things.

I suspect that the problem won't be so much "this individual package won't install"; it'll be unexpected conflicts among those popular packages, perhaps dependent on environment and particular constraints files. We're trying to get some early feedback via the survey so that we can then fix bugs, set up more automated testing, etc.

Thanks for asking your followup question, and of course for your work on software other people depend on. Does this clarify things?

[zzzeek]

Yes thanks, as the maintainer of Mako and a few other projects I'm not in a position to have any non-trivial configuration that might find some issue within a more complex setup as this is a library with only a few simple dependencies and not a full blown application. if you want to find applications that are relatively easy to test and will stress test the system like crazy I would suggest looking at the openstack applications: https://github.com/openstack/ . I run CI for openstack nova, neutron, and keystone against SQLAlchemy for example.

[brainwane]

@zzzeek Thank you for the tips! I'll have our team take a look at the OpenStack applications!

And could I ask you to forward this announcement to the SQLAlchemy mailing list?

(I've also written more about our in-progress publicity work here in case you want to add any thoughts there.)