python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

xond · 2023-04-16T17:04:28Z

how to replicate the issue

Python 3.11.3/3.11.2/3.11.1/3.11.0 (python-3.11.3-amd64.exe etc)
Microsoft Windows [Version 10.0.19044.2130] (Active code page: 936)

tmp.py

#coding:utf-8
print("中文")

python -m pdb tmp.py

Traceback (most recent call last):
  File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 1774, in main
    pdb._run(target)
  File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 1652, in _run
    self.run(target.code)
             ^^^^^^^^^^^
  File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 167, in code
    return f"exec(compile({fp.read()!r}, {self!r}, 'exec'))"
                           ^^^^^^^^^
UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 28: illegal multibyte sequence
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\users\xond\appdata\local\programs\python\python311\lib\pdb.py(167)code()
-> return f"exec(compile({fp.read()!r}, {self!r}, 'exec'))"

Linked PRs

gaogaotiantian · 2023-04-16T18:30:28Z

This is an encoding issue - we can simply use io.read_code to avoid it, as we are always expecting the read code with pdb. PR is done with the regression test.

`pdb` should use `io.open_code` to open code to avoid encoding issue.

…H-103581) `pdb` should use `io.open_code` to open code to avoid encoding issue. (cherry picked from commit 31acfd7) Co-authored-by: Tian Gao <[email protected]>

) (#103867) `pdb` should use `io.open_code` to open code to avoid encoding issue. (cherry picked from commit 31acfd7) Co-authored-by: Tian Gao <[email protected]>

…103581) `pdb` should use `io.open_code` to open code to avoid encoding issue.

hauntsaninja · 2023-04-26T20:53:28Z

Thanks again!

Apr 27, 2023 · 2023-04-27T19:31:20Z

This is a case that always should've gone through io.open_code, and people may be caught out by having code execution bypass that path.

We should backport to 3.8+

Apr 27, 2023 · 2023-04-27T19:33:52Z

I take that back - 3.8-3.10 were already fixed, it was a change in 3.11 that regressed this. So no backports required.

hauntsaninja · 2023-04-27T19:43:38Z

Thanks for checking, sorry I could have more explicitly mentioned this! :-)

bedevere-bot mentioned this issue Apr 16, 2023

gh-103578: Fix pdb reading code with non-utf8 encoding #103581

Merged

hauntsaninja pushed a commit that referenced this issue Apr 26, 2023

gh-103578: Fix pdb reading code with non-utf8 encoding (#103581)

31acfd7

`pdb` should use `io.open_code` to open code to avoid encoding issue.

bedevere-bot mentioned this issue Apr 26, 2023

[3.11] gh-103578: Fix pdb reading code with non-utf8 encoding (GH-103581) #103867

Merged

itamaro pushed a commit to itamaro/cpython that referenced this issue Apr 26, 2023

pythongh-103578: Fix pdb reading code with non-utf8 encoding (python#…

8c02f95

…103581) `pdb` should use `io.open_code` to open code to avoid encoding issue.

hauntsaninja closed this as completed Apr 26, 2023

3.10 only security fixes 3.9 only security fixes 3.8 (EOL) end of life labels Apr 27, 2023

3.11 only security fixes 3.12 bugs and security fixes and removed 3.10 only security fixes 3.9 only security fixes 3.8 (EOL) end of life labels Apr 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

xond commented Apr 16, 2023 •

edited by bedevere-bot

Loading

gaogaotiantian commented Apr 16, 2023

hauntsaninja commented Apr 26, 2023

Apr 27, 2023

Apr 27, 2023

hauntsaninja commented Apr 27, 2023

python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

Comments

xond commented Apr 16, 2023 • edited by bedevere-bot Loading

Linked PRs

gaogaotiantian commented Apr 16, 2023

hauntsaninja commented Apr 26, 2023

Apr 27, 2023

Apr 27, 2023

hauntsaninja commented Apr 27, 2023

xond commented Apr 16, 2023 •

edited by bedevere-bot

Loading