Pyright with venv still uses Python 2.7 on MacOS

[jdtsmith]

Describe the bug
The (unsupported) python2.7 interpreter bundled with MacOS is still chosen by pyright (despite the fixes in #829), but only when a venv is specified.

To Reproduce
Specify a "venvPath" and "venv" in config. Start `pyright --verbose" in that directory:

  Python version 2.7 from interpreter is unsupported

Expected behavior
pyright checks for python3 and finds it in the venv (even python in the venv points to python3).

Additional context
It appears the fallback python3 -> python is disabled in the presence of a venv, and the venv is not first activated when determining the interpreter.

[erictraut]

There should be additional details logged to the console. Can you provide those? Also, can you please post your config file?

Pyright never activates a venv. It uses the directory of the venv that you specify. The only time that pyright invokes the python interpreter is when no venv is specified and it needs to fall back to the import resolution paths provided by the default python3 interpreter (i.e. the interpreter that runs when you type "python3" from the shell).

[jdtsmith]

Sure here's more detail:

% cat pyproject.toml
[tool.pyright]
venvPath = "./.venv"
venv = "default"

% python3 --version
Python 3.9.7

rain% ls .venv 
.venv@

% ls .venv/
default/

% pyright --verbose
No configuration file found.
pyproject.toml file found at /Users/XXX/code/python/scraps.
Loading pyproject.toml file at /Users/XXX/code/python/scraps/pyproject.toml
Python version 2.7 from interpreter is unsupported
Assuming Python platform Darwin
No include entries specified; assuming /Users/XXX/code/python/scraps
Auto-excluding **/node_modules
Auto-excluding **/__pycache__
Auto-excluding .git
stubPath /Users/XXX/code/python/scraps/typings is not a valid directory.
Search paths for /Users/XXX/code/python/scraps
  /usr/local/Cellar/pyright/1.1.186/libexec/lib/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/XXX/code/python/scraps
  /Users/XXX/code/python/scraps/typings
  /usr/local/Cellar/pyright/1.1.186/libexec/lib/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Users/XXX/code/python/venv/default/lib/python3.9/site-packages
Searching for source files

[erictraut]

OK, the import resolution search paths look correct. They point to your local venv, not to the libraries within the Python 2.7 interpreter on your machine.

The part that's not correct is the pythonVersion. You haven't specified a pythonVersion in the config, and pyright is trying to determine which version you would like it to use for its analysis.

I mentioned above that the only time pyright ever invokes the python interpreter is when no venv is specified and it needs to run the default interpreter to retrieve the import resolution paths. I forgot about one other case where the default python interpreter is invoked. If you don't specify a pythonVersion, pyright will invoke the default Python interpreter to try to query its version. It does not try to activate the configured virtual environment to get this information because virtual environment managers come in various flavors, and there are many ways to activate different venvs on different platforms. It simply does a shell invoke of "python -c <script to determine version>".

@heejaechang, this is the code you wrote in fullAccessHost.ts, in the getPythonVersion method. It looks like it's simply invoking "python" rather than first attempting to use "python3" like we do in getPythonSearchPaths. Shouldn't we be consistent here and use "python3" in both cases? I think that would fix this case.

@jdtsmith, while we look for a solution, you can work around this issue by adding pythonVersion = "3.9" to your config file.

[jdtsmith]

That works, thanks. The original issue I had combined this problem with an incorrectly specified venvPath, causing the old interpreter's path to be used, and picking up an old numpy install inside of v2.7. Which led to some real confusion as you might expect!

venvPath /Users/XXX/code/python/scraps/venv is not a valid directory.
venv default subdirectory not found in venv path /Users/XXX/code/python/scraps/venv.
stubPath /Users/XXX/code/python/scraps/typings is not a valid directory.
Python version 2.7 from interpreter is unsupported
Search paths for /Users/XXX/code/python/scraps
  /usr/local/Cellar/pyright/1.1.186/libexec/lib/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/XXX/code/python/scraps
  /Users/XXX/code/python/scraps/typings
  /usr/local/Cellar/pyright/1.1.186/libexec/lib/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Library/Python/2.7/site-packages/pip-18.1-py2.7.egg
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
  /Users/XXX/Library/Python/2.7/lib/python/site-packages
  /Library/Python/2.7/site-packages
  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC

[heejaechang]

Oops. PR auto closed the issue. re-opened so that we can close it once fix is released.

[erictraut]

This is addressed in pyright 1.1.188, which I just published.

[jdtsmith]

Thanks. This does seem to fix the problem for straightforward calls to pyright from the command line, but when running from emacs/lsp-mode as node /usr/local/bin/pyright-langserver --stdio, I get similar behavior as before (selecting Python v2.7), both times with a pyproject.toml file that specifies nothing:

The following clients were selected based on priority: (server-id pyright, priority 3)
Pyright language server 1.1.188 starting
Server root directory: /usr/local/lib/node_modules/pyright/dist/
Creating watchers for following 13 folders:
...
No configuration file found.
pyproject.toml file found at /Users/XXX/code/python/scraps.
Setting pythonPath for service "/Users/XXX/code/python/scraps": "/usr/bin/python"
Loading pyproject.toml file at /Users/XXX/code/python/scraps/pyproject.toml
Python version 2.7 from interpreter is unsupported
Assuming Python platform Darwin
No include entries specified; assuming /Users/XXX/code/python/scraps
Auto-excluding **/node_modules
Auto-excluding **/__pycache__
Auto-excluding .git
stubPath /Users/XXX/code/python/scraps/typings is not a valid directory.
Python version 2.7 from interpreter is unsupported
Searching for source files
Auto-excluding /Users/XXX/code/python/scraps/.venv/default
Found 17 source files

[erictraut]

The only way I can explain that is if emacs/lsp-mode is sending a "python.pythonPath" settings entry to pyright. If it does, pyright will execute the interpreter at the provided path rather than simply executing python3 and letting the shell determine which interpreter to use. I recommend consulting with the maintainers of emacs/lsp-mode.

[jdtsmith]

Yes, actually the plugin lsp-pyright is responsible. Too many chefs in the pythonPath selection kitchen. I've filed emacs-lsp/lsp-pyright#65. Thanks.