Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script containing a "shebang" would not start under Windows #100107

Closed
peter88213 opened this issue Dec 8, 2022 · 11 comments
Closed

Script containing a "shebang" would not start under Windows #100107

peter88213 opened this issue Dec 8, 2022 · 11 comments
Labels
3.11 only security fixes OS-windows type-bug An unexpected behavior, bug, or error

Comments

@peter88213
Copy link

peter88213 commented Dec 8, 2022

Bug report

This Python script hello.pyw does not start on my PC when I double click it in Windows Explorer.

#!/usr/bin/env python3
from tkinter import messagebox
messagebox.showinfo(message='hello, world')

Instead, a console window pops up and disappears instantly.

Now I modify the shebang as follows:

#!/usr/bin/env python
from tkinter import messagebox
messagebox.showinfo(message='hello, world')

On double-clicking hello.pyw, a console opens, then the message box appears.

Now I remove the shebang:

from tkinter import messagebox
messagebox.showinfo(message='hello, world')

On double-clicking hello.pyw, the message box appears as expected.

This misbehavior was observed with the introduction of Python 3.11. It was not like this until version 3.10.8.

Your environment

  • CPython versions tested on: 3.11.1
  • Operating system and architecture: Windows 10 22H2 x64

I have implemented quite a few open source programs with Python, all of which have a shebang for working across platforms. On my download pages I had to generally discourage the use with Python 3.11.

Linked PRs

@peter88213 peter88213 added the type-bug An unexpected behavior, bug, or error label Dec 8, 2022
@AlexWaygood AlexWaygood added OS-windows 3.11 only security fixes labels Dec 8, 2022
@eryksun
Copy link
Contributor

eryksun commented Dec 8, 2022

The new launcher's "/usr/bin/env" virtual command searches PATH for the given command name, even if the command is "python" or "python3". The old launcher reserves these names.

If the command isn't found in PATH, the launcher ignores the shebang and runs the script normally. In this case, the GUI launcher, "pyw.exe", will execute the script using the default GUI binary (e.g. "pythonw.exe" from a registered installation).

Instead, a console window pops up and disappears instantly.

The launcher probably found the app execution alias "%LocalAppData%\Microsoft\WindowsApps\python3.exe". If the app version of Python 3 isn't installed, then by default Windows creates "python.exe" and "python3.exe" aliases for an "app installer" app that opens the store to install Python 3. If this app is passed any command-line arguments, it prints an informative message1 and exits. Since the app installer doesn't inherit a console from "pyw.exe", it allocates one to print the message. This console window gets destroyed as soon as the app exits2.

If you don't have the app distribution of Python installed, I recommend disabling the "python.exe" and "python3.exe" app execution aliases.

On double-clicking hello.pyw, a console opens, then the message box appears.

For the virtual command "/usr/bin/env python", the launcher searches PATH for "python.exe". Normally this binary is a console application, which means that the system automatically allocates a console for the process if it doesn't inherit one from the parent process.

The old launcher reserves names that start with "python", even for the "/usr/bin/env" virtual command. In this case, "pyw.exe" runs a registered version of the GUI (non-console) binary, "pythonw.exe".

Footnotes

  1. "Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases."

  2. If the user's default console is Windows Terminal, it can be configured to keep a tab open after a console session has terminated. This allows the user to read and copy text from the terminal window after a console application has exited or crashed, even when the application is executed by a GUI process such as Explorer.

@peter88213
Copy link
Author

Thank you for the explanation. So then it's a feature, not a bug, that my programs don't start after updating to 3.11 and I have to dive somewhere into the depths of the Windows configuration?

I would probably do that, but I can't expect my non-technical audience to do that, with the best will in the world.

If this isn't fixed, I'll have to modify my setup scripts to remove the "shebang" in Windows.

For now, I'll go back to Python 3.9 and continue to advise against version 3.11.

@eryksun
Copy link
Contributor

eryksun commented Dec 8, 2022

So then it's a feature, not a bug, that my programs don't start after updating to 3.11 and I have to dive somewhere into the depths of the Windows configuration?

Yes, it's a feature, but one that's not currently implemented to be novice friendly considering the default configuration of Windows. It would be nice if the launcher detected and ignored the "python.exe" and "python3.exe" aliases for the app installer. I don't recall if the API has something to simplify this, but even if it doesn't, the launcher could just read the data in the reparse point to check whether the target app is "Microsoft.DesktopAppInstaller". If it finds the app installer, ignore the shebang and run the default Python.

I'll have to modify my setup scripts to remove the "shebang" in Windows.

Can you use "/usr/bin/python3" instead of "/usr/bin/env python3"? The former is restricted to registered installations and commands. It does not search PATH for "python3.exe".

@peter88213
Copy link
Author

peter88213 commented Dec 8, 2022

Can you use "/usr/bin/python3" instead of "/usr/bin/env python3"?

Thank you for the advice. Unfortunately, I can't test it for now, since I just uninstalled Python 3.11.1, just as I did some weeks ago with 3.11, that already cost me a lot of time until I even found the source of the problem at the "shebang".

Yes, it's a feature, but one that's not currently implemented to be novice friendly considering the default configuration of Windows.

I guess that's true. However, "novice" does not quite fit the non-tech user group of my open source software. I don't expect them to learn Python or anything. I want to provide them with useful applications that will work without problems even after they might switch to Linux some day. However, for many of them it may already be a hurdle that they have to install a Python interpreter for this at all.

@peter88213
Copy link
Author

Can you use "/usr/bin/python3" instead of "/usr/bin/env python3"?

This seems to do the trick. I have patched 24 projects so far. Thank you so much for your support.

However, I give consideration to the fact that there might be a zillion Python scripts out there that suddenly "stop working" after updating to 3.11 because of the shebang. Not a good prospect.

@eryksun
Copy link
Contributor

eryksun commented Dec 8, 2022

there might be a zillion Python scripts out there that suddenly "stop working" after updating to 3.11 because of the shebang.

I agree. I'd prefer for the launcher to ignore the result of searching PATH if it finds an app execution alias for "Microsoft.DesktopAppInstaller". I think it's the responsibility of the launcher to work around this common problem.

As to launching a GUI script with an attached console, that's not a showstopper, but it's also not correct behavior. The GUI script launcher could special case "python[X[.Y]]" to search for "pythonw[X[.Y]]". For your case, this would also avoid the pitfall of finding a "Microsoft.DesktopAppInstaller" alias. Windows doesn't create app installer aliases for "pythonw.exe" and "pythonw3.exe".


More generally, the GUI script launcher could always use the process creation flag DETACHED_PROCESS. This prevents the system from automatically allocating a console. Also, if the console script launcher is executed without a console, it should use the DETACHED_PROCESS flag when executing the command.

If a console application is launched that really requires a console, the spawned process may silently fail. A properly designed interpreter shouldn't crash if it has no console, but a script might die on an unhandled exception if it expects sys.std* to never be None, or if it expects to be able to open "CONIN$" or "CONOUT$" or use the console API. That would be peculiar behavior for a ".pyw" script, but not surprising for a ".py" script.

@flagrama
Copy link
Contributor

Can you use "/usr/bin/python3" instead of "/usr/bin/env python3"?

This a problem if you distribute your script to Mac users. This will use the default Python 3.9 install instead of any up-to-date installed versions. I believe BSDs have their python executables in /usr/local/bin/ as well. Because of the Windows Store version of Python and the stubs in the PATH there is no good way to distribute a script that runs cross-platform since the correct way is with the #!/usr/bin/env python3 shebang. It works on effectively every operating system except Windows without the py launcher and Windows 10/11 with py launcher.

A project I've worked on had no issues with just directing people to install the py launcher until the Windows Store version came around and broke things. Telling literally ever individual user to disable the app execution alias is draining on support volunteers. They're going to go with the alternate shebang right now but breaking BSD support and limiting the ability to use new language features is not ideal, and those are only the known issues. They're hoping this gets resolved before the problems end up as larger issues, but it doesn't really seem like this is actively trying to be resolved.

Copy link
Member

the launcher could just read the data in the reparse point to check whether the target app is "Microsoft.DesktopAppInstaller". If it finds the app installer, ignore the shebang and run the default Python.

This might be the only feasible option. I wish there were better APIs for managing these aliases, but Windows refuse to add them, and I'm starting to suspect they've quietly deprecated them entirely (in favour of what? I have no idea...).

Needs someone to contribute it. And I'd only want it to happen in the context of the environment search - nobody else should have to pay the perf penalty.

@flagrama
Copy link
Contributor

flagrama commented Jan 18, 2024

I don't have much experience with Windows development, but as a proof of concept I wrote this: https://github.com/python/cpython/compare/main...flagrama:cpython:py-launcher-shebang-appexeclink?expand=1

This seems to work how I would expect. If no Python is installed, I get an error about not being able to find a Python version. If I have just the store version installed it launches fine, as well if I have just a version from python.org installed. I don't know which one it uses when both are installed, but it still launches my script with a #!/usr/bin/env python3 shebang without me having to turn off app execution aliases.

The code was written and tested in a fresh Windows 11 VM. The website version of python was 3.12.1 installed as a custom installation, the only optional features enabled were pip and py launcher and advanced options was only the Associate files with Python option.

Copy link
Member

That's a great start, but the problem with that is we should use an actual alias to a real install, which your proposed change will also reject.

Inside the inner condition we need to read the contents of the appexeclink and check which package the link belongs to. It will differ between the App Installer redirector and a PSF released Python install. Only the App Installer one should cause us to reject.

@flagrama
Copy link
Contributor

This mess is what I managed: https://github.com/python/cpython/compare/main...flagrama:cpython:py-launcher-shebang-appexeclink?expand=1

Not pretty, probably inefficient, and no proper error handling, but it does what it needs to.

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 25, 2024
…Microsoft Store (pythonGH-114358)

(cherry picked from commit d5c21c1)

Co-authored-by: Vincent Cunningham <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 25, 2024
…Microsoft Store (pythonGH-114358)

(cherry picked from commit d5c21c1)

Co-authored-by: Vincent Cunningham <[email protected]>
Jan 25, 2024
…oft Store (GH-114358)

(cherry picked from commit d5c21c1)

Co-authored-by: Vincent Cunningham <[email protected]>
Jan 25, 2024
…oft Store (GH-114358)

(cherry picked from commit d5c21c1)

Co-authored-by: Vincent Cunningham <[email protected]>
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes OS-windows type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

5 participants