PEP 734: Multiple Interpreters in the Stdlib

eric.snow · 2023-12-21T17:48:40.670Z

Thanks for the feedback!

Alyssa Coghlan:

the expected behaviour for Python implementations that don’t offer subinterpreters needs to be defined (could be as simple as “there is no interpreters module”, but an explicit query flag in sys.implementation could also make sense, as then the error message on importing the interpreters module could be more explicit about the problem)

Alyssa Coghlan:

can the subinterpreter specific ExecFailure be dropped and ExceptionGroup used instead? Even if it is only ever a group of one in the direct subinterpreter invocation use case, it still feels like the same flavour of error container to me (although it may get confusing when attempting to catch foreign exception types)

That’s an interesting idea, but I’ll have to think about it more. The conceptual relationship between the two exception types isn’t standing out to me, other than that they both wrap other exceptions. The main difference is, I think, critical though: ExceptionGroup wraps actual exception objects, while ExecFailure wraps a snapshot of an exception (a “remote” one, at that). That said, being able to use except * to catch the wrapped exception is something that had not even crossed my mind. It’s an intriguing and alluring idea.

Ultimately, I want the API to feel as natural to users as possible (i.e. principle of least surprise). In this specific case, what will feel most natural (and be most useful)? I think what I’ve proposed fills that need (but also leaves room for a more elegant solution in the future).

(expand for further detail)

The behavior of comparable existing features/modules doesn’t provide sufficient insight; I’m not sure there’s any precedent for the combination of (1) wrapping one exception in another, (2) the wrapped exception was uncaught in some “remote” execution, and (3) it is a snapshot of the exception (including, importantly, its traceback) rather than the actual exception object.

Again, I did consider precedent (though not ExceptionGroup):

multiprocessing: has “remote” uncaught exceptions, but doesn’t propagate them synchronously
threads: don’t propagate exceptions either, but there are several ways to add an exception hook to deal with uncaught exceptions
asyncio: doesn’t seem like a good fit (but I haven’t looked closely enough, I suppose)
subprocess: propagates remote errors but not uncaught exceptions (and certainly not with tracebacks)
exec(): uncaught exceptions are propagated synchronously but are neither remote nor snapshots
calls: similar to exec()
__cause__/__context__: expected to be set to exception objects, not snapshots

Also note that I’m trying to focus on what is actually useful for users here. I tried all sorts of variations, but, in the end, I realized that the one thing that mattered was showing the traceback for uncaught exceptions. (That realization came from actually trying to use subinterpreters in real code.) In effect, if the propagated ExecFailure is uncaught then the default sys.excepthook will show the remote exception as it would have shown it if it were “local” (i.e. traceback and str(exc)).

As I iterated through the design, at first I considered reconstituting the remote exception locally. However, that was too tricky, hid the remote origin of the exception, and introduced a conceptual ambiguity around whether to wrap the exception or not. I also looked into propagating a full snapshot that captured all the information of the exception (including attributes) and could be used as a substitute for the remote snapshot. Before I got in too far it started to feel like overkill and was tricky enough to make me reconsider.

I also looked at this from a different angle: making use of ExecFailure.__cause__ to wrap the remote exception. However, that required either an actual exception object or changing __cause__ to work with exception surrogates/snapshots. It was at that point I realized all I really cared about was the error display for uncaught exceptions.

To show the traceback we don’t need more than a wrapper exception (ExecFailure) and a minimal snapshot. If the user needs to handle the remote exception then they can do it in the “remote” code (or perhaps via some kind of excepthook).

FWIW, I’ve also thought about introducing a builtin RemoteError type, to represent an exception originating in any kind of “remote” execution and formalize what I’m trying to do with interpreters.ExecFailure. (ExecFailure would be a subclass, or I’d drop it.) As a builtin, RemoteError could be special-cased by sys.excepthook()/traceback.TracebackException.

I tabled the idea since it would require extra design/specification, and I found I didn’t really need it for PEP 734 (which I’m trying to keep minimal). It also felt like overkill for what would currently be a single use case.

Alyssa Coghlan:

I think Antoine is correct that the cross-interpreter Queue design needs further thought. One advantage of the “channel” terminology was that the type restrictions were more acceptable without the semantic implications of the “Queue” terminology established by the threading and multiprocessing modules

The main reason I went with a Queue-like type is familiarity for users. FWIW, I’m still ruminating over the potential consequences of using pickle by default, particularly as it relates to “sharing” mutable objects.

Alyssa Coghlan:

Specification section should appear before the Rationale section (as the latter assumes reader familiarity with the former)

Mostly, I was following PEP 1, as well as what I thought was precedent. However, switching the order definitely makes much more sense.

Alyssa Coghlan:

the create APIs need to be discussed in the Rationale section (i.e. as you noted above, they create both the underlying instance and the running interpreter’s proxy to it, they don’t just instantiate the proxy as implied by a direct constructor call)

Alyssa Coghlan:

I like your proposed exec()/call()/call_in_thread() method rename

That’s good to hear! It’s still my preference but I’m also still mulling it over.

MegaIng · 2023-12-21T18:04:08.744Z

I would suggest two different transfer methods/formats, one CopyQueue that can transport anything copyable (with potential optimizations for immutable things that can be shared) and one ShareQueue or a similar interface that can only transfer things that can be shared. Notably, these two queue objects would themselves not be copyable, but only shareable: It wouldn’t make sense to write these out to disk, so they shouldn’t support they pickle interface and therefore not be transportable via the CopyQueue. (Naming is hard: I am not really happy with either of these two).

These separation would make it clear to the user what is happening and reduce surprises. Note that for something like call, it might make sense to specify that the arguments can be either shareable or copyable so that you can directly pass over the queues. But that would IMO not be as clean as forcing users to make the choice consciously to share objects

steve.dower · 2023-12-21T18:04:31.085Z

Eric Snow:

The main reason I went with a Queue-like type is familiarity for users. FWIW, I’m still ruminating over the potential consequences of using pickle by default, particularly as it relates to “sharing” mutable objects.

I think the idea of LowLevelQueue makes a lot of sense here, leaving the Queue name available for a richer version to come later.

You can probably write a recipe for using pickle by default with LowLevelQueue, but I think it’d be a waste to make that the “standard” protocol. We’ll want to be able to define a new __proxy__ or similar (to mirror __reduce__/__getargs__) that knows how to construct an object on the other side assuming the original one still exists (and hence, any native memory/etc. is also available), and that can be part of Queue.

But I think we’re still at the stage where we want 3rd party packages to design the Queue object, and users can use those on each end of the channel. So the PEP right now probably only wants to say we intend to add Queue, but not yet, and add a deliberately simple one as a building block now.

guido · 2023-12-21T19:05:36.804Z

I think there will always be a notion of shareable objects — though that’s a poor name, it’s really about things that have value semantics. And the interpreters module can have a Queue that only allows values. Over time the definition of “value” can be adjusted.

Others can define more general queues in different namespaces, using pickle, proxies, or whatever else people invent.

ncoghlan · 2023-12-22T00:18:39.137Z

Eric Snow:

The main difference is, I think, critical though: ExceptionGroup wraps actual exception objects, while ExecFailure wraps a snapshot of an exception (a “remote” one, at that). That said, being able to use except * to catch the wrapped exception is something that had not even crossed my mind. It’s an intriguing and alluring idea.

Yeah, that’s where my mind was going, too. However, after further thought, I realised it won’t work in this case since the exception types won’t reliably match across interpreters (although they might, more on that below).

Instead, I think you’re right that the better precedent is the lightweight TracebackException objects defined in the traceback module that allow reporting of full tracebacks without keeping entire frame stacks around. This is especially applicable with a shift to supporting pickle-able objects by default when calling sub interpreters, since the result could be composed in the executing interpreter and then passed back via pickle.

Alternatively, a more general solution that doesn’t make assumptions about imported modules in the calling interpreter would replace object references with their string representations (and always skip exporting locals info)

Even with the exception existing solely in the interpreters module, I think further clarification is needed on the translation of the snapshot between the two interpreters:

what happens when an exception in the snapshot comes from a module that the caller hasn’t imported?
can the exception processing implicitly trigger code execution in the caller? (that question also applies to regular object returns & queue usage)
what default options are used when creating the TracebackException object in the called interpreter? Can those options be configured?

FWIW, I’ve also thought about introducing a builtin RemoteError type, to represent an exception originating in any kind of “remote” execution and formalize what I’m trying to do with interpreters.ExecFailure. (ExecFailure would be a subclass, or I’d drop it.) As a builtin, RemoteError could be special-cased by sys.excepthook()/traceback.TracebackException.

I tabled the idea since it would require extra design/specification, and I found I didn’t really need it for PEP 734 (which I’m trying to keep minimal). It also felt like overkill for what would currently be a single use case.

I agree that a built-in “ForeignException” type would need to consider more use cases (e.g. true FFI, where the called target isn’t written in Python), so doing something more specific to this use case makes sense.

I think I’d prefer a bikeshed painted that kind of colour from a naming perspective though, and have the interpreters module docs refer consistently to local and “foreign” objects rather than local and remote ones (as unlike multiprocessing, the foreign objects are still in the same process on the same machine, as they are for a cross-language FFI, they’re just not part of the local Python object graph).

a-reich · 2023-12-22T03:34:54.819Z

I’ll add my reaction as a user that I’m pretty excited by this PEP, and the potential concurrency patterns it enables! (I’ve started playing with the implementation a bit too).

I’m especially interested now that the proposal supports sharing buffer views. I come from the data analysis/ML area, and in my experience when using concurrency libraries transferring arrays/dataframes between processes had a major impact on performance. But for these kinds of objects, most of the “size” is backed by binary buffers, and as Antoine noted this aligns well with pickle’s out-of-band buffers which offer an existing general protocol for this. (Unfortunately, none the popular extension libraries seem possible to run in subinterpreters yet.)

My two cents on some of the choices debated:

+1 to having a low level method like the current exec_sync as well as the higher level ones like call(callable, args=None, kwargs=None, /) → object or call_threaded.
While I understand the reasons for preferring a more minimal initial API, e.g. that we don’t know what will work well until more user experience, IMO to get the adoption that will provide that experience it will be important to have an option for sending general objects via pickle or something (ideally customizable) like it. If most people wanting to use it will need to write/find a wrapper for handling other types, and those wrappers will mostly work similarly, why not provide that code already to get them started. But maybe it makes sense to have that in a PyPI package until later, I don’t know.

thomas · 2024-03-25T15:26:50.302Z

The SC has been evaluating and discussing PEP 734 since it was sent our way last month (PEP 734 -- Multiple Interpreters in the Stdlib · Issue #234 · python/steering-council · GitHub), and we have some concerns. This is partly with my RM hat on, having seen the kinds of oversights we’ve had to fix in 3.12 late in the release cycle of 3.12.0, and as backported fixes. It’s understandable, given the kind of changes that are required, their scope and the limited experience (and limited number of eyeballs) involved, but it does worry me. On top of that we have a completely new API that we have basically no experience with, even though it’s somewhat similar to other solutions we’ve seen (like concurrent.futures).

How invasive is the PEP 734 implementation in the interpreter? To what extent could it be released as a PyPI package first, before we consider it for inclusion in the standard library? If it can’t entirely be a separate package, could we make a private module with the minimum number of hooks for such a PyPI package to work? Having it be a separate package would make it much easier to evolve the API and, for example, fix problematic semantics, since you can release at will and users can always pin to an older version of the package while they make their code work with the newer version.

Also, the SC would like to invite you to come talk to us about the PEP, either at the regularly scheduled office hour, or if that doesn’t work for you, we can schedule something separately.

eric.snow · 2024-03-25T18:38:18.478Z

Thomas Wouters:

The SC has been evaluating and discussing PEP 734 since it was sent our way last month (PEP 734 – Multiple Interpreters in the Stdlib · Issue #234 · python/steering-council · GitHub ), and we have some concerns.

Thanks for the update. I appreciate your thoughtfulness.

Thomas Wouters:

This is partly with my RM hat on, having seen the kinds of oversights we’ve had to fix in 3.12 late in the release cycle of 3.12.0, and as backported fixes. It’s understandable, given the kind of changes that are required, their scope and the limited experience (and limited number of eyeballs) involved, but it does worry me.

Most of those fixes related to subinterpreters generally, especially isolated interpreters. The effect of PEP 734 is primarily in how it makes the existing feature more available and thus exposes existing flaws. What is your concern relative to this PEP?

Thomas Wouters:

On top of that we have a completely new API that we have basically no experience with, even though it’s somewhat similar to other solutions we’ve seen (like concurrent.futures).

At its core, PEP 734 has only a few parts:

expose the existing C-API (which has been around a long time) via a lightly-wrapping abstraction
provide a simple, safe way for one interpreter to run code in another (similarly a light implementation on top of existing C-API)
provide a basic mechanism for safely sending data from one interpreter to another

(There’s also a small amount of “sugar” on top of that.)

I’d consider the first part to be uncontroversial. The second part is focused and simple.

The third part is fairly important because subinterpreters aren’t nearly as useful without a way to communicate between them. That became clear almost immediately in the early PEP 554 discussions. PEP 734 presents an API that is almost identical to the existing queue.Queue, so there shouldn’t be any real surprises for users.

From my perspective, the PEP 734 API presents very little risk. My guiding principle from the beginning has been to present a minimal API on which we could build. Based on the many discussions this PEP has had, and on the practical experience with it that I and others have had with the implemented API, I’m confident it is a good starting point.

To me, that’s the key thing here: we want a solid foundation in the stdlib on which people may start using multiple interpreters in their programs. We can build from there as appropriate.

Thomas Wouters:

How invasive is the PEP 734 implementation in the interpreter?

There are many things I’ve implemented that were mostly inspired by the needs of PEP 734 or because of my experiences with it, but most (or nearly all?) of them are valuable on their own. They’ve certainly helped us identify flaws in CPython over the last five years.

When it comes to invasiveness, I’ve been careful to keep a lot of the relevant code isolated to certain files. Nearly all the C-API I’ve added is strictly limited to the internal API (Include/internal/pycore_*.h).

If you were to ask what we would rip out if PEP 734 were rejected, I’m not sure that I’d put anything on that list. Basically it is valuable in testing subinterpreters, at the least.

To put it a different way, there are only a handful of things in the repo that are currently only used by the PEP 734 implementation (and they are almost all consolidated in specific files). That includes only one piece of runtime state: the internal “cross-interpreter data registry”, which I’d already like to replace with a new type slot (via a separate PEP) rather than adding a public API.

Thomas Wouters:

To what extent could it be released as a PyPI package first, before we consider it for inclusion in the standard library?
…
Having it be a separate package would make it much easier to evolve the API and, for example, fix problematic semantics, since you can release at will and users can always pin to an older version of the package while they make their code work with the newer version.

Early on I worked hard to implement PEP 734 in a way that I could publish it on PyPI. In fact, my plan has been to publish such a package for use with 3.12. The same could be done for 3.13, though I think it’s better suited for the stdlib.

My main concern with publishing a PyPI module is that there will be less exposure to the feature and less accessibility. I’m certainly biased here, but I’m convinced the multiple interpreters feature is a great benefit to users and want it in their hands with the least friction possible. Again, I’m confident that the minimal API provided by PEP 734 is the right place to start.

Thomas Wouters:

If it can’t entirely be a separate package, could we make a private module with the minimum number of hooks for such a PyPI package to work?

That minimal module is basically what PEP 734 specifies. We could certainly move the PEP 734 implementation to a PyPI package, either my building on the necessary existing internal C-API or via what is currently the _xxsubinterpreters module. (That said, I don’t see the value in doing so over adding the new stdlib module.)

Thomas Wouters:

Also, the SC would like to invite you to come talk to us about the PEP

I’ll meet at the office hours this week.

ncoghlan · 2024-03-28T02:38:42.773Z

Eric Snow:

I’m confident that the minimal API provided by PEP 734 is the right place to start.

I think this is true for the proposed semantics, but I’m not 100% convinced it is true for the exact naming choices.

Specifically, given the PEP (and presumably documentation) terminology focuses on “shareable objects” (vs arbitrary objects), the name of the syncobj flag on interpreters.Queue objects seems odd. It feels like share_data would better describe the behaviour being requested.

For the notion of operating on PyPI for a release cycle, the natural API split point would presumably be to have the _interpreters module in the standard library, and put interpreters on PyPI (including InterpreterPoolExecutor).

The main advantage I see to this more conservative approach is that it would allow some of the open questions in the PEP to be deferred until the module’s promotion to the standard library (whether that comes next release or later), specifically:

default behaviour of Queue objects when the interpreter adding the object to the queue goes away before the object is retrieved. The current default feels prone to “errors passing silently without being explicitly silenced” to me, akin to the original handling of async tasks that never get scheduled.
exact API design for InterpreterPoolExecutor (in particular, how the new interpreters are configured, and how the pools support execution of configuration code in a way that ensures every interpreter in each pool is configured exactly once. This will presumably be similar to ThreadPoolExecutor and ProcessPoolExecutor, but I’m not sure the existing initializer API will quite be sufficient for InterpreterPoolExecutor)
whether some improved ergonomics are feasible for cross-interpreter exception handling (e.g naming a parameterless context manager as a dotted-string when calling Interpreter.exec, so the actual execution in the other interpreter runs inside a with statement using that context manager)
building out a list of other not-yet-shareable object types where it would be genuinely helpful to be able to share them
ensuring the data buffer sharing works as expected with other data buffer exporters (NumPy, etc)

While the initial level of adoption will definitely be lower than for a generally available standard library module, I think the adoption you’ll see will be from exactly those you most want at this stage of feature publication: folks that aren’t happy with the trade-offs between threads and processes, and are actively looking for something that strikes the middle ground of combining low-overhead data sharing with strong default data separation.

barry · 2024-04-10T21:36:13.942Z

Hi Eric,

Thank you again for coming to the Steering Council’s office hours to discuss PEP 734. We all found it very helpful to talk about the PEP with you in real time.

After much discussion, the Steering Council thinks that the best way forward with this module is to maintain it separately for now, release it on PyPI, and let it mature there for a while before including it in the stdlib. There are several reasons leading us to this decision.

We think the API needs more real-world usage before it can be deemed stable. It may indeed be the best API available, but without some maturation on PyPI, we can’t really know for sure. With the module being independently developed and released, its API can evolve much more quickly than it can once the module is in the stdlib. It will also not have to adhere to the strict backward compatibility and deprecation rules of the stdlib. The Steering Council thinks this is a good policy in general for new stdlib packages, and plans to propose this as “standard operation procedure” for most new packages.

You expressed a concern about the maintenance burden of a PyPI release, but we think that’s solvable. You should be able to recruit co-maintainers, either from the current cohort of core developers, or from users of the subinterpreters package. We’re also confident that we can get help with setting up any automation needed for testing and releases. It shouldn’t be much more of a burden developing and releasing it independently for now than it would be in the stdlib.

We are going to mark the PEP as Deferred, and we can always reevaluate stdlib inclusion for a future version of Python.

Cheers,
-Barry (on behalf of the Steering Council)

steve.dower · 2024-04-11T12:24:08.045Z

My understanding of this PEP is that would mean Eric can add the API as private but exposed (e.g. a _subinterpreters module) and then use a PyPI package to expose it? That allows some freedom to change it in a subsequent release, but the PyPI package probably wouldn’t amount to much more than from _subinterpreters import * (and InterpreterPoolExecutor, which I agree could live outside of the stdlib).

In other words, it’s “provisional” except we don’t mark stuff as provisional anymore so it’s just “private”.

The whole concept is based around exporting internal functionality, so it’s not like the API can be separated from the interpreter. Behaviour changes have to occur within the runtime, not within the module, and the interesting development is all at a level higher than proposed here (again, except InterpreterPoolExecutor). Right now, ctypes would be needed to access our internal C APIs to get the same behaviour, and I don’t think there’s really any way to do the synchronisation needed to actually make that work.

I’m sure Eric presented this analogy, but this module here is essentially exposing os.fork so that 3rd parties can develop multiprocessing outside of the stdlib. If the SC is okay with exposing os._fork for now, then I’m sure that’ll be workable, but if the SC is going to be surprised at the new internal APIs showing up with no stdlib users, it’d be good to clear that up sooner rather than later.

eric.snow · 2024-04-11T17:56:11.870Z

Thanks for the taking the time to consider the PEP and for being clear about the position of the Steering Council. While I don’t agree with the decision and would have liked more direct discussion with the Steering Council, I’m on board and ready to move forward.

FWIW, I do see the point you (and Alyssa) have made about uncertainty with the design choices (e.g. names), regardless of the scale of the proposal. I also agree that community feedback from an implementation of the PEP on PyPI has a good chance of identifying potential improvements.

(Personally, I consider the advantage of exposure in the stdlib to outweigh the risk of having to tweak the API in later versions, given the small proposed surface area. That said, I’ll readily admit that my perspective is skewed toward the value I anticipate the new module to add for Python users, making it hard for me to tell if I’m weighing things fairly.)

Here’s my plan for 3.13:

wrap up the various various 3.13 fixes I have in flight (or planned)
rename the _xxsubinterpreters module to _interpreters
work on a PyPI package (3.12/3.13) that uses _interpreters (still keeping it minimal)
look for collaborators

There are alternatives for the second point: expose a bunch of necessary internal C-API in the public API, or use the internal C-API directly (i.e. with Py_BUILD_CORE). However, I’d much prefer using the low-level _interpreters module as it make a lot of things simpler.

eric.snow · 2024-04-11T18:38:49.304Z

Also, I have some feedback for the Steering Council on how things have played out with this PEP, involving level-of-interaction and clarity on the decision. I’m not interested in complaining/venting, but rather want to identify some key good/bad parts of my experience here, in the hope that it helps the Steering Council and the community. (I’d also like to see what we can do to better support the Steering Council, who are volunteers like the rest of us, but have a distinct (and challenging) role as gatekeepers.)

Where would be the best place to start such a discussion?

barry · 2024-04-11T23:38:50.973Z

That seems like a good plan to me ^[1]. Keep in mind too that the PEP is deferred so you can definitely come back and ask for a re-evaluation for a future Python release.

wearing my core dev hat, not necessarily SC member hat ↩︎

barry · 2024-04-11T23:41:39.443Z

Speaking with my SC member hat on, I certainly would welcome feedback. Maybe start with another SC office hours session? If you wanted to write down your thoughts first, an email to the steering council?

rtobar · 2024-09-26T04:28:45.609Z

Hi all,

Yesterday I opened this discussion about the state of PEP 734 in the “Python Help” category in this forum. It’s been a day getting a few views, but no replies, and I fear I might have posted the message in the wrong category – maybe I should have written it here directly instead. This is my first time opening a discussion in this forum, so I wasn’t sure what the rules were.

In any case, I thought mentioning it here might attract the correct audience. Maybe this cross-posting is frowned-upon though, so please excuse me in advance if that’s the case. Thanks!

eric.snow · 2024-09-26T22:10:47.553Z

No, you did the right thing. I just haven’t gotten to your post yet. It will probably be today. Sorry for the wait.

TkTech · 2024-11-11T05:57:34.085Z

When using InterpreterPoolExecutor, the started sub-processors do not seem to fully inherit the sys.path of the parent, which does work for all other types of executors. I ran into this when using sub-interpreters inside of a pytest-based tests, as the started jobs did not get the test directory in their sys.path. This is under 3.13 using the latest backport.

I’m not entirely convinced this is a bug as it’s a logical outcome, but it’s certainly unexpected when compared to the other executors.

eric.snow · 2024-11-11T22:05:10.698Z

I’d say this is worth a bug report. Please open an issue and feel free to CC me (@ericsnowcurrently) on it.

ZeroIntensity · 2024-11-13T22:12:26.387Z

For reference: InterpreterPoolExecutor workers do not inherit modifications made to sys.path before starting. · Issue #126714 · python/cpython · GitHub