Originally reported by: itziakos (Bitbucket: itziakos, GitHub: itziakos)
The problem is demonstrated by the following example code
from setuptools.sandbox import setup_context
from pkg_resources import DistributionNotFound
with setup_context(' '):
raise DistributionNotFound('Memory more memory')
Inside the setup_context context manager, if an exception takes place, is captured and saved in pickled format using UnpickleableException.dumps classmethod. The dumps classmethod will first try to pickle the exception. In the example above the pickling attempt will fail because inside the setup_context the DistributionNotFound is not pickleable (note that the call to hide_modules will have removed the pkg_resources module from sys.modules). On a failed picking attempt the dumps method will wrap the DistributionNotFound inside an UnpickleableException and try again calling the dumps classmethod with the new exception. Unfortunately inside the setup_context an UnpickleableException is also unpickleable because setuptools is not in sys.modules. From this point the dumps classmethod will be called recursively trying to pickle the UnpickleableException, until the system runs out of memory.
I was able to replicate the memory problems using both python 2.7 and 3.4
Originally reported by: itziakos (Bitbucket: itziakos, GitHub: itziakos)
The problem is demonstrated by the following example code
Inside the
setup_contextcontext manager, if an exception takes place, is captured and saved in pickled format usingUnpickleableException.dumpsclassmethod. The dumps classmethod will first try to pickle the exception. In the example above the pickling attempt will fail because inside thesetup_contexttheDistributionNotFoundis not pickleable (note that the call tohide_moduleswill have removed thepkg_resourcesmodule fromsys.modules). On a failed picking attempt thedumpsmethod will wrap theDistributionNotFoundinside anUnpickleableExceptionand try again calling thedumpsclassmethod with the new exception. Unfortunately inside thesetup_contextanUnpickleableExceptionis also unpickleable becausesetuptoolsis not insys.modules. From this point thedumpsclassmethod will be called recursively trying to pickle theUnpickleableException, until the system runs out of memory.I was able to replicate the memory problems using both python 2.7 and 3.4