Issue 35644: venv doesn't work on Windows when no venvlauncher executable present

Issue35644

Created on 2019-01-02 20:30 by Ray Donnelly, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (9)
msg332892 - (view) Author: Ray Donnelly (Ray Donnelly) * Date: 2019-01-02 20:30
Happy New Year!

I'm not sure if this is a misunderstanding on my part, a docs bug or a code bug.

At https://docs.python.org/3/library/venv.html we see:

"The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages"

and

"This will create the tutorial-env directory if it doesn’t exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files."

However, when testing with https://www.python.org/ftp/python/3.7.2/python-3.7.2-amd64.exe I see no Python interpreter (nor DLL) in my venv directory:

```
python.exe -m venv %TEMP%\venv
%TEMP%\venv\Scripts\activate.bat
dir %TEMP%\venv
```

gives:

```
 Directory of C:\Users\RDONNE~1\AppData\Local\Temp\venv

02/01/2019  19:38    <DIR>          .
02/01/2019  19:38    <DIR>          ..
02/01/2019  19:38    <DIR>          Include
02/01/2019  19:38    <DIR>          Lib
02/01/2019  19:38               121 pyvenv.cfg
02/01/2019  19:38    <DIR>          Scripts
               1 File(s)            121 bytes
               5 Dir(s)  912,281,780,224 bytes free
```

pyvenv.cfg contains:

```
home = C:\Users\rdonnelly\AppData\Local\Programs\Python\Python37
include-system-site-packages = false
version = 3.7.2
```

Further to this, after activating, I do not see the `venv` directory in `sys.path`:

```
python -c "import sys; print(sys.path)"
['',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\DLLs',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\lib',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']
```

From past experience, the old `virtualenv` project would copy the interpreter and DLL across.

Any help here would be appreciated!
msg332894 - (view) Author: Ray Donnelly (Ray Donnelly) * Date: 2019-01-02 20:37
I found the executable is in the `Scripts` directory, closing. The real issue I'm facing is on Anaconda Distribution's build of Python 3 which I'm updating to 3.7.2.

Closing,

Cheers!
msg332895 - (view) Author: Ray Donnelly (Ray Donnelly) * Date: 2019-01-02 21:04
Bit of an update to this, I'm re-opening it as there appears to be a regression from Python 3.7.1 to 3.7.2 for the case when there is no venvlauncher.exe present (i.e. when there are no python{w,}.exes in Lib\venv\scripts\nt). The old code of copying `sys.executable` is no longer run (on Windows only).

Currently Anaconda Distribution's venv is done this way. Should we change it to use the same method as official CPython Windows releases?

Here is a diff I think we need to apply for now, if you feel it is reasonable to make a PR then I'm happy to do so.

```
$ diff -urN Lib/venv/__init__.py.orig Lib/venv/__init__.py
--- Lib/venv/__init__.py.orig   2019-01-02 20:56:45.015131800 +0000
+++ Lib/venv/__init__.py        2019-01-02 20:56:55.330130800 +0000
@@ -188,9 +188,9 @@
         binpath = context.bin_path
         path = context.env_exe
         copier = self.symlink_or_copy
+        copier(context.executable, path)
         dirname = context.python_dir
         if os.name != 'nt':
-            copier(context.executable, path)
             if not os.path.islink(path):
                 os.chmod(path, 0o755)
             for suffix in ('python', 'python3'):
```
msg332897 - (view) Author: Ray Donnelly (Ray Donnelly) * Date: 2019-01-02 21:10
The commit that my patch modifies is: https://github.com/python/cpython/commit/1c3de541e64f75046b20cdd27bada1557e550bcd

Cheers.
msg332913 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-03 07:26
That patch will break the case where the launcher *is* present, which is why it was changed. If you're sure the launcher won't be present under Scripts/nt, then that patch is okay. You'll probably need to add code to also copy python37.dll and vcruntime140.dll, since those are needed by the regular python.exe but not the launcher.

However, why not use the launcher? All it does is find pyvenv.cfg, read its home variable and launches the python.exe it finds there. If you copy your build of venvlauncher over (or even our build) it will do the same thing.

That said, I'm a little concerned about the sys.path in your first post. I assumed nobody had hit problems since there haven't been any complaints (I've been using the Store version exclusively since 3.7.2 was released), but there seems to be something wrong there. Perhaps it was due to your build?
msg332914 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-03 07:29
The installed venv seems to be okay, so I'm going to assume it's something about your own build.

Now I realise I haven't ever tried virtualenv against 3.6 or later... no idea what state that's in.
msg332943 - (view) Author: Ray Donnelly (Ray Donnelly) * Date: 2019-01-03 19:25
Thanks Steve, the sys.path value from the first comment can be discarded, it was running the wrong Python!

The 'old' mechanism (which my patch reverts to) does copy all the necessary DLLs already. I released builds with this patch now and venv works fine (tested with pyperformance which uses venv).

However, we are more than happy to switch to the venvlauncher method as not deviating from upstream unnecessarily is always a good thing!

Do you have any pointers about how to build venvlauncher? I'll try to schedule some time for that.
msg332952 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-03 23:15
It should just build directly from venv[w]launcher.vcxproj, though you'll need to rename venv[w]launcher.exe to python[w].exe (otherwise they'd conflict in the build directory).
msg338557 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-21 16:34
Doesn't seem to be anything to fix upstream here.
History
Date User Action Args
2022-04-11 14:59:09adminsetgithub: 79825
2019-03-21 16:34:17steve.dowersetstatus: open -> closed
resolution: not a bug
messages: + msg338557
2019-01-03 23:15:53steve.dowersetmessages: + msg332952
2019-01-03 19:25:06Ray Donnellysetmessages: + msg332943
2019-01-03 07:29:13steve.dowersetmessages: + msg332914
2019-01-03 07:26:26steve.dowersetmessages: + msg332913
2019-01-02 21:26:11Ray Donnellysettitle: venv doesn't do what it claims to do (apears not to work at all?) -> venv doesn't work on Windows when no venvlauncher executable present
2019-01-02 21:10:31Ray Donnellysetmessages: + msg332897
2019-01-02 21:04:51Ray Donnellysetstatus: closed -> open

messages: + msg332895

2019-01-02 20:37:59Ray Donnellysetstatus: open -> closed

messages: + msg332894
stage: resolved

2019-01-02 20:30:12Ray Donnellycreate