Windows issue in conda-forge tests

[maresb]

Description of your problem

Hooray, a Windows issue!

On conda-forge build, the test succeeds in Linux and OSX, but fails in Windows.

Azure Pipeline logs

The relevant line is monolithic, containing lots of \n characters, but a simple print("""[paste here]""") cleans things up:

Error while running command:
['C:\\Miniconda\\conda-bld\\pyscaffoldext-markdown_1615157790774\\_test_env\\python.exe', '-m', 'tox', '-e', 'build']
{'stderr': -2, 'universal_newlines': True}

******************** Terminal ($? = 1) ********************
build create: C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build
build installdeps: build
build installed: build==0.3.0,packaging==20.9,pep517==0.9.1,pyparsing==2.4.7,toml==0.10.2
build run-test-pre: PYTHONHASHSEED='457'
build run-test: commands[0] | python -m build .
Unable to copy 'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\test_generated_extension0\\tmpex4iq1oa\\my_project\\.tox\\build\\Scripts\\venvlauncher.EXE'
Unable to copy 'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\test_generated_extension0\\tmpex4iq1oa\\my_project\\.tox\\build\\Scripts\\venvlauncher.exe'
Unable to copy 'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\test_generated_extension0\\tmpex4iq1oa\\my_project\\.tox\\build\\Scripts\\venvwlauncher.exe'
Traceback (most recent call last):
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\lib\site-packages\build\env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\venv\__init__.py", line 75, in create
    self._setup_pip(context)
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\venv\__init__.py", line 303, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Miniconda\conda-bld\pyscaffoldext-markdown_1615157790774\_test_env\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
ERROR: InvocationError for command 'C:\Users\VssAdministrator\AppData\Local\Temp\pytest-of-VssAdministrator\pytest-0\test_generated_extension0\tmpex4iq1oa\my_project\.tox\build\Scripts\python.EXE' -m build . (exited with code 1)
___________________________________ summary ___________________________________
ERROR:   build: commands failed

I don't know what's going on, and as a policy for my personal sanity, I don't do Windows. Any ideas?

[FlorianWilhelm]

Windows Errors! Don't we all love them so much? I don't know how many hours they cost us already in the development of PyScaffold.

The problem seems to be that venvlauncher.exe is missing and at least in 2019 there was a conda-related issue in Python itself. But it might also be related to tox where @abravalheri is an absolute expert.

[FlorianWilhelm]

Not sure I completely understand the Azure CI/CD but this run seems to have worked. Also is there any difference to PyScaffold itself, as this has even way more Windows unit tests?

[maresb]

Yay! I disabled pytest for pyscaffoldext-markdown in order to get the package to build. I figure we can diagnose this in parallel and then release a new build with tests enabled, although for rebuilds, conda-forge seems to only test against linux.

[abravalheri]

Sorry guys, just seeing it now...
I am not very familiar with the conda-forge pipelines, but there is something very weird... for example there is a bunch of warnings about undefined pytest marks that are clearly present in setup.cfg...
I wonder if the test runner is even seeing setup.cfg, that could explain some missing packages, maybe?

Instead of running pytest directly, could we run the tests as we do in the Cirrus CI via tox? It is kind of a blind guess, but it would help to isolate the problem (so we can compare directly the behaviour of Azure and Cirrus and try to find which configuration it differs).

Some other thing we should do also is update the building process, i.e. replace python -m pep517.build with python -m build. I will work on that now.

[maresb]

Ah yes! I was having a lot of trouble with tox because of some of the assumptions that the tests seem to make:

The conda-forge workflow is to install the package, copy the test files to a temp directory, and run the tests from there. At least in the pyscaffold tests, there seemed to be a lot of tests to ensure for example that putup was inside of .tox. But with this workflow, it's in site-packages instead.

It also means that I need to explicitly copy setup.cfg. Will provide references shortly...

[abravalheri]

I see... Yes in the codebase of PyScaffold we added some of those tests because at somepoint we were testing things against the wrong installation/other versions... So they work like metatests/sanity checks.

Please let me know when you find those, I think we could arrange some env vars and skip when running tests for conda-forge.

I have a silly question (sorry not familiar with conda): in order to see if the changes in the pyscaffold-dsproject solve the problem in the feedstock, do we need to keep tagging new versions?

[maresb]

Yes, I'm trying to dig up a list of those tests which were failing. (I got around them by adding not slow and not system so I wonder if we could simply add another pytest marker for skipincondaforge or something.) I will send links soon.

For reference, here's how one copies setup.cfg to the temporary test directory (I need to do this for pyscaffoldext-markdown)

[maresb]

Ok, this assertion fails because pyscaffold isn't installed in src. This function is called by these three tests.

Finally, assert ".tox" in putup

Those are all the references I wanted to find.

And your very good and not at all silly question, conda-forge renders from PyPI releases. So to first order, you'd need to publish to PyPI. But there are some mechanisms to test locally via Docker. But in that case I don't think we have access to Windows (but I could be wrong). I also think conda-forge only tests against Linux for new versions.

Let me see if I can find some feasible way to test conda-forge based on local source...

[abravalheri]

Thank you very much @maresb. Later today I will try to replicate the tests from conda in a windows machine... let's see what I can find wrong about it...

I wonder if adding python-build as a test dependency + the lastest change I did in the master a few minutes ago would solve the problem....

[abravalheri]

Regarding the tox thing, I will add this to the TODO list also.

[maresb]

I think I have things straightened out on the rc branch, so if you publish rc2 I should be able to build that on conda-forge. I'm still looking into other possibilities.

Regarding tox in general, unfortunately I'm not so familiar with it, but from what I gathered, when running tox it triggers its own build and runs pytest against that. But in conda-forge I want to run pytest against the conda-forge build, so for conda-forge I'm trying to run pytest by itself.

[abravalheri]

Hi @maresb, I just sat down to investigate these issues and saw the latest comment from Cris. Should we just do the:

- imports
- pip check
- putup --help

as suggested by him? I suppose these should speed up things and alleviate the load on their CI (PyScaffold tests are quite heavy... our Cirrus setup usually takes at least 30min).

For the extensions, I suppose we could just check if the extension flag is amount the output of putup --help, which would be very easy with grep, but then in Windows I have no idea how to make work...

[abravalheri]

If we have problems (e.g. git not being installed on the CI env or not being available), we can also try to postpone the following command (that is currently running the first time the pyscaffold.shell module is imported):

# Currently defined as a top-level var in `src/pyscaffold/shell.py`
git = get_git_cmd()

[maresb]

I just published a potential fix which includes the tests. I am now including the setup.cfg in the testing directory, so maybe the Windows problem goes away. In case it doesn't, then let's fall back to the putup --help solution.

[abravalheri]

Ok, meanwhile I will try to run the recipe locally as suggested by him. Too bad conda just takes forever for installing/creating environemtns...