Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiprocessing not propagating -Xfrozen_modules=off #106723

Closed
ergoithz opened this issue Jul 13, 2023 · 0 comments · Fixed by #106724
Closed

Multiprocessing not propagating -Xfrozen_modules=off #106723

ergoithz opened this issue Jul 13, 2023 · 0 comments · Fixed by #106724
Assignees
@gpshead
Labels
3.11 only security fixes 3.12 bugs and security fixes topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@ergoithz
Copy link
Contributor

ergoithz commented Jul 13, 2023 •
edited by bedevere-bot
Loading

Bug report

After python3.11 changes around frozen imports, when using multiprocessing contexts other than fork, the newly added -Xfrozen_modules=off isn't passed to spawned process interpreters.

Simple snippet demonstrating the issue:

"""

$ python -Xfrozen_modules=off test.py
main: {'frozen_modules': 'off'}
forkserver: {}
spawn: {}
fork: {'frozen_modules': 'off'}

"""

import sys
import multiprocessing

def xoptions():
    return sys._xoptions

def main():
    print('main:', xoptions())
    for ctx in ('forkserver', 'spawn', 'fork'):
        with multiprocessing.get_context(ctx).Pool(1) as pool:
            print(f'{ctx}:', pool.apply(xoptions))

if __name__ == '__main__':
    main()

The issue seems to be subprocess._args_from_interpreter_flags not honoring frozen_modules key from sys._xoptions.

$ python -Xfrozen_modules=off -c 'import subprocess;print(subprocess._args_from_interpreter_flags())'
[]

Your environment

python 3.11.4

Linked PRs
@ergoithz ergoithz added the type-bug An unexpected behavior, bug, or error label Jul 13, 2023
This was referenced Jul 13, 2023
Merged
Closed
Closed
@gpshead gpshead self-assigned this Jul 28, 2023
@gpshead gpshead added 3.11 only security fixes 3.12 bugs and security fixes labels Jul 28, 2023
gpshead added a commit that referenced this issue Jul 28, 2023
@ergoithz @kumaraditya303 @gpshead
gh-106723: forward -Xfrozen_modules option to spawned process interpr…
3dcac78 
…eters (#106724)

Co-authored-by: Kumar Aditya <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
@bedevere-bot bedevere-bot mentioned this issue Jul 28, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 28, 2023
@ergoithz @kumaraditya303
@gpshead @miss-islington
pythongh-106723: forward -Xfrozen_modules option to spawned process i…
a2d8412 
…nterpreters (pythonGH-106724)

(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <[email protected]>
Co-authored-by: Kumar Aditya <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
@bedevere-bot bedevere-bot mentioned this issue Jul 28, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 28, 2023
@ergoithz @kumaraditya303
@gpshead @miss-islington
pythongh-106723: forward -Xfrozen_modules option to spawned process i…
69130d9 
…nterpreters (pythonGH-106724)

(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <[email protected]>
Co-authored-by: Kumar Aditya <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
kumaraditya303 added a commit that referenced this issue Jul 28, 2023
@miss-islington @ergoithz
@kumaraditya303 @gpshead
[3.11] gh-106723: forward -Xfrozen_modules option to spawned process …
8492aae 
…interpreters (GH-106724) (#107368)

gh-106723: forward -Xfrozen_modules option to spawned process interpreters (GH-106724)
(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <[email protected]>
Co-authored-by: Kumar Aditya <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
kumaraditya303 added a commit that referenced this issue Jul 28, 2023
@miss-islington @ergoithz
@kumaraditya303 @gpshead
[3.12] gh-106723: forward -Xfrozen_modules option to spawned process …
4f72a9a 
…interpreters (GH-106724) (#107367)

gh-106723: forward -Xfrozen_modules option to spawned process interpreters (GH-106724)
(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <[email protected]>
Co-authored-by: Kumar Aditya <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gpshead gpshead

Labels
3.11 only security fixes 3.12 bugs and security fixes topic-multiprocessing type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-106723: forward -Xfrozen_modules option to spawned interpreters ergoithz/cpython
3 participants
@gpshead @ergoithz @AlexWaygood