Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for PEP 489 multi-phase module initialisation #34

Open
philthompson10 opened this issue Aug 20, 2024 · 0 comments
Open

Add support for PEP 489 multi-phase module initialisation #34

philthompson10 opened this issue Aug 20, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@philthompson10
Copy link
Contributor

philthompson10 commented Aug 20, 2024

Python v3.5 saw the implementation of multi-phase module initialisation described in PEP 489. This supports the reloading of extension modules and the release of all allocated memory when an interpreter is finalised (particularly useful when Python is embedded in a C/C++ application). As these are relatively minor use cases, and that the traditional initialisation methods are still supported, it hasn't been a priority to add the support to the sip module or to SIP generated modules. However the recent changes to Python to support multiple sub-interpreters and the disabling of the GIL mean that the support should now be added.

Note that although extension modules will be able to be reloaded, the libraries they wrap cannot and they may contain their own mutable global data. Modules need to be written with this in mind, ie. that the state of a wrapped library may be influenced by the actions of a previously loaded copy of the module.

The main requirements for multi-phase module initialisation are:

  • all global data must be immutable
  • all mutable data must be stored in the module state
  • all module-defined types must be created on the heap.

The support will be implemented as ABI v14. An extension module is written to target a particular ABI. For example, v12 (which uses a custom type for wrapping C/C++ enums) is targeted by PyQt5, and v13 (which uses the standard Python enum type for wrapping C/C++ enums) is targeted by PyQt6.

In more detail:

  • At the moment ABIs use the same data structures for the definition and implementation of wrapped types and instances. This causes unnecessary problems in maintaining backwards compatibility. v14 will split these using immutable data structures, similar to PyMethodDef and PyType_Spec, and mutable data structures that are completely opaque and specific to the ABI version.
  • PEP 573 added support for accessing the module state from most contexts. This was implemented in Python v3.9 but only added to the limited API in Python v3.10. This support will be required for ABI v14 and so Python v3.9 will not be supported. Note that it is still not possible to access the module state from slot methods (eg. nb_add). It is assumed that this will not be a problem.
  • All wrapped methods will use METH_FASTCALL.
  • Contemporary techniques will be used to create wrapped types and instances. (The current implementation might still be using techniques that were required when supporting Python v1.5.)
  • The functionality of v12 and v13 will be merged and specific functionality (eg. the nature of the support for C/C++ enums) will be configured at build time. ABIs v12 and v13 will be deprecated and eventually removed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant