Namespace package __path__ manipulation breaks packages using __init__.py to define code

[dzbarsky]

🐞 bug report

Affected Rule

rules_python/python/pip_install/tools/wheel_installer/namespace_pkgs.py

Lines 90 to 99 in fe33a45

	with open(ns_pkg_init_filepath, "w") as ns_pkg_init_f:
	# See https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages
	ns_pkg_init_f.write(
	textwrap.dedent(
	"""\
	# __path__ manipulation added by bazelbuild/rules_python to support namespace pkgs.
	__path__ = __import__('pkgutil').extend_path(__path__, __name__)
	"""
	)
	)

Is this a regression?

Not sure, first time I have tried this.

Description

The code in the package's source init.py seems to get overwritten. I'm not fully sure why, it appears that

rules_python/python/pip_install/tools/wheel_installer/namespace_pkgs.py

Lines 87 to 88 in fe33a45

	if os.path.isfile(ns_pkg_init_filepath):
	raise ValueError("%s already contains an __init__.py file." % dir_path)

is trying to guard against it.

If possible, it would be nice to lay out a site_packages that is less hacky.

🔬 Minimal Reproduction

https://github.com/dzbarsky/rules_python_pylink_bug

bazel test //:flink_import_test

🔥 Exception or Error

ImportError: cannot import name 'add_version_doc' from 'pyflink' (/private/var/tmp/_bazel_zbarsky/45e7e567ddccd93ad1b41c75488b912d/sandbox/darwin-sandbox/13/execroot/__main__/bazel-out/darwin-fastbuild/bin/flink_import_test.runfiles/pip_apache_flink_libraries/site-packages/pyflink/__init__.py)

🌍 Your Environment

Operating System:

Output of bazel version:

  
Bazelisk version: development
Build label: 6.3.2
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Aug 8 15:58:06 2023 (1691510286)
Build timestamp: 1691510286
Build timestamp as int: 1691510286
  

Rules_python version:

  
0.25
  

Anything else relevant?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[dzbarsky]

This is still an active issues, please keep open

[groodt]

Potentially related: #2156

I'm not completely sure that this is caused by lack of site-packages, so I won't close this as a duplicate.

[engnatha]

I was able to get around this by setting enable_implicit_namespace_pkgs = True in the pip_parse target.

[engnatha]

To follow up on my comment above, while this does allow the import to work, it wipes any data directories that don't have any python files. There's some flink directories that contain only java archive files. Still discovering when precisely those are needed.