C API Working Group and plan to get a Python C API compatible with alternative Python implementations?

Core Development C API
1

The C API Working Group produced a PEP (PEP 733 – An Evaluation of Python’s Public C API | peps.python.org) entitled “An Evaluation of Python’s Public C API”.

Part of this PEP is dedicated to discuss issues related to the CPython C API. The issues are grouped in different categories. The category API Specification and Abstraction is especially important for long-term potential Python performance improvements.

I won’t try to summarize the PEP and associated issues in GitHub - capi-workgroup/problems: Discussions about problems with the current C Api, but basically this is about the fact the CPython C API is incompatible with a lot of strategies to improve performance of interpreters (strategies used for different dynamic languages).

Without a proper solution to these issues, I’m a bit pessimistic about the possibilities that:

  • alternative Python implementations like PyPy and GraalPy become more usable and useful in practice for a lot of Python users.
  • the Faster CPython project leads to major performance improvements, especially when Python extensions are used (i.e. for basically all Python codes related to science and data).

I try to write a text on this subject but here my question is more specific.

I observed the huge work done by the C API Working Group to produce PEP 733.

I know that HPy could be an answer to these problems. I aware of GitHub - markshannon/New-C-API-for-Python: Design and discussion for the new C-API for Python. Unfortunately, these projects are no longer very active.

What are the next steps for the C API Working Group?

Is there a long-term plan to get these issues fixed in few years? Is there a plan to make the swift from a Python ecosystem using CPython for nearly anything to a Python ecosystem compatible with advanced and faster Python implementations?

2

(Speaking for myself without having checked with the rest of the WG.)

A “few” years is short-term for Python. We’re looking more at the 10+ year scale for genuine fixes, and have been focusing for now on stemming the growth in random directions of the current API.

You can see some of the more forward-looking discussion right here in this same category - just click on the “C API” link at the top of this page. There’s also our repository where we discuss specific designs (if not clear, the WG is mostly reactive to proposals, and while some of the members are also actively proposing improvements, anyone is allowed to bring them - the WG exists to answer the question of “who should review it”), and the issue trackers on other repositories in that GitHub org have various proposals and discussions for long-term designs.

That said, as our priority is to not break existing code without a good reason, and also to not increase our maintenance burden without the prospect of later reducing it (for example, by deprecating/removing something whenever we add something else), you aren’t going to notice a lot of changes in the code base.

I think I can summarise our current vision/guiding goal/north star right now as:

  • incrementally lead native extension modules to use stable and reliable APIs[1] that do not prevent innovation on the core interpreter loop

It turns out there’s plenty to worry about here! But there are a few things that this overarching goal does exclude or deprioritise:

  • we aren’t treating native extension performance as the top concern, but Python performance and extension stability (this doesn’t mean we try to slow down native code, but the choices are usually like “we can give extension modules a 1% speedup but never modify the interpreter again”, and we have decided that the interpreter wins)
  • we aren’t focusing on embedders (I personally am, but the assumption is that most embedders are going to be spending most time in native modules anyway, and so they get the benefits there)
  • we aren’t actively improving the core runtime ourselves (other groups are)
  • we are more concerned with (C or non-C) extension modules being loaded into CPython than with CPython modules being loaded into non-CPython runtimes (but we try not to do API designs that will only work for CPython or from C)

And probably more examples of things that people might be hoping to see but aren’t getting.

I’ll let other WG members chime in as they want. Hopefully that goes some way to answering your question, but if you have any specific queries, this is as good a place as any to bring them up.

  1. I’d have said “Stable ABI” or “Limited API” if I’d meant either of those specific things. I’m just using “stable” as an adjective here. ↩︎

5 Likes
3

There is a slow on-going effort to hide implementation details. Replace code accessing directly structure members with new getter functions. Add new functions which don’t expose implementation details. It should be easier to implement these functions on PyPy and GraalPython, and it should make it possible (in the long term at least) to evolve Python internals to develop new optimizations.

There were multiple atttemps to replace the C API with a new one, but it seems like we have to support the C API in the long term (to support the great diversity of existing C extensions). So we can only make tiny changes following the slow PEP 387 deprecation process to evolve the C API.

Maybe things will change and prove me wrong, but so far changes are incremental and slow. Steadily the C API is evolving to become better :wink:

A problem with adding new functions is that projects don’t want to use them until they are available for the oldest Python version that they support. I wrote the pythoncapi-compat project to make new functions available on older Python versions, so it becomes possible to use new functions as soon as possible. For example, it provides Py_HashBuffer() function for Python 3.13 and older, wheras the function was only added to Python 3.14. The pythoncapi-compat project also supports PyPy.

There is a work-in-progress to add more public C API for Cython (example). Currently, Cython has to consume the internal C API which some practical issues. This topic is being discussed for a few years, but it’s complicated so it’s not solved yet.

Good progress was made to make the limited C API more complete and more usable in practice.

On my side, I would like to replace direct access to PyThreadState structure with getter and setter functions, but I didn’t start to work on this topic yet. Projects like greenlet and Cython access directly to PyThreadState.

You might need to elaborate on which problem should be solved here. From my point of view, PyPy already solved the problem: it supports the C API with cpyext. The main issue is that the emulation is slow and that’s why HPy was created.

HPy has a nicer API and ABI, and is faster on PyPy, but it takes time to port existing C extensions. Python continues to support its C API in the long term.

2 Likes
4

First, I’d like to thank you @vstinner and @steve.dower for your answers. Is it remarkable that a small question from a random guy get (so quickly) so interesting and informed answers. This is very much appreciated.

I’m going to add few remarks from my point of view of user of the scientific Python stack (and dev of few packages in this field, and Python teacher).

What would change my life in terms of Python usage would be to be able for applications needing some packages of the scientific Python stack to use PyPy and GraalPy (nearly) as easily as I use CPython today and to get the benefit in terms of speed of these alternative implementations even when extensions are used.

Currently,

  • alternative Python implementations are very slow as soon as extensions are used (even a bit, like one poor Numpy float in pure Python code)
  • Most projects do not upload wheels for PyPy and GraalPy (famous examples: Scipy and Pandas)

A proper solution to get a Python stack adapted for modern interpreters is clearly something like HPy. I write “something like HPy” since I don’t know if it should be HPy or an alternative project inspired by HPy. But I think the “universal wheels” of HPy are a key aspect that should be taken into account. Also gradual porting and better debugging. HPy has a lot of nice qualities.

Of course, the question of when this could be possible is important. It has strong practical implications if one can get that in something like 3 years or in something like 10 years. In 10 years, it is probable that Mojo will be properly open-source and quite mature. I take this example (Mojo) but I guess you see what I mean. Will it still be reasonable in few years to use Python for new scientific software projects needing performance? If Python users needing extensions have no other choice than using interpreters using or emulating the memory layout used in old CPython and ref counting, it might not be the case.

So it seems to me that we should try to find a fast path towards a Python stack adapted for modern interpreters.

The process that you describe seems really reasonable from the point of view of CPython (and thus the whole Python ecosystem) but it is also important that some projects (in particular Numpy) can go faster.

Of course, the changes in the CPython C API going in the direction of a C API adapted for modern interpreters seem very useful, especially if they can be used for older Python versions with pythoncapi-compat.

However, I’d like to point out that the current situation and plan of the C API working group introduce an uncertainty which does not help to start now an adaptation of some packages.

Let’s take the point of view of Numpy (which is really the key project to go towards a Python stack adapted for modern interpreters):

  • Should we invest on HPy?
  • Should we wait for another new CPython C API officially supported by the C API WG?
  • Should we just slowly follow CPython using pythoncapi-compat?

It is even worth than that! Numpy 1 has been ported to HPy (at least nearly, I don’t know if all tests passed and there is also the question of the Numpy C API) but now the project is stalled for different reasons, but I guess in particular because of uncertainties about the direction to be taken regarding the Python C API. One could find some founding for such important and impactful tasks, but it is difficult if the direction is not clear.

It seems to me that to get “universal Numpy wheels” in few years, the C API WG should officially support HPy or something like HPy, i.e. a layer above the CPython C API that could be used now to port C extensions while supporting a reasonable range of Python versions. I realize that it is a very big thing and that it would need a specific PEP.

I’m going to finish by warning that all this is only a modest point of view of a Python user without experience of the CPython C API for anything serious!

2 Likes
5

I think you’re making a very important point here. Even I as a CPython core developer wouldn’t be able to answer these questions, and that’s a problem.

2 Likes
6

Great questions, and some very important context none of us have mentioned yet is that libraries that want to move faster can move faster, but CPython has to change at the rate of the slowest moving ones. If everyone were as responsive as numpy, we could dramatically accelerate any improvements, but alas, they are not.

What you’re asking about here is which level of abstraction should libraries adopt. There are some great choices out there, but they all come with tradeoffs, mostly (in this space) between speed and maintenance. To offer a more complete list (in order):

  • internal C API (highest performance, most maintenance, least portable)
  • CPython C API
  • pythoncapi-compat
  • HPy[1]
  • pybind11, nanobind, PyO3[2]
  • Cython[3]
  • limited C API/stable ABI
  • Python code (lowest performance, least maintenance, most portable)

Each individual library will also have their own constraints on which options are viable, but ultimately, this is how you choose which one to use.

If you choose the CPython C API, you’ll be able to achieve better performance (though we’re working to close that gap by providing better APIs for code generators), but you’ll spend more time chasing our changes. If you choose, say, nanobind[4], then once you’ve adapted to it you’ll get to use new CPython APIs automatically, potentially at a slight cost to performance, and should (eventually) be able to recompile for other runtimes and get peak integration there, too. HPy is in between these two levels, but already offers the recompiling (and no doubt some level of new API adoption, though I haven’t checked that out).

And of course, you can totally mix and match. Most projects do, by combining Python code and some form of native code, but if there’s value in using Cython for a lot, HPy for a few parts and Python for the rest then there’s no reason you can’t do it.

What we as the C API WG can’t do is to force people into this. As I said, we have to work at the pace of the slowest adopters, not the fastest. But we can and do encourage fast adopters to use something other than our oldest APIs.

Very minor point of clarification:

another new CPython C API officially supported by the C API WG

I’d say the C API WG would endorse a new CPython C API, which is one legitimate interpretation of “supported by”. But the other interpretations (cared for, maintained by, etc.) are by the entire maintainer team (core developers). The role of the WG is to review changes/proposals and ensure consistency across the codebase, not necessarily to be the only ones who work on the API.

  1. Ranked separately from the other bindings due to manual reference counting and also the relative thinness of the bridge. ↩︎

  2. I’ve no doubt missed some, please don’t get mad. These are representative examples not a census. ↩︎

  3. Ranked separately from the other bindings because it’s typically used for complete implementations, not just bridging languages/interfaces. ↩︎

  4. Which seems to me to be the “highest” reasonable choice for numpy. ↩︎

2 Likes
7

I don’t want to answer for @paugier here, but I think that part (the level of abstraction) is already understood by interested third-parties. The problem is more “which alternative is future-proof” and “which timeline can I be confident on for the CPython’s C API evolution”.

For example, HPy is a tremendous idea but it doesn’t seem to receive much maintenance or development anymore, and without any official endorsement/support by the CPython team or the PSF it may simply die in the next few years. Nobody wants to start rebasing their library on a dying API.

For now the implicit message seems to be “wait for us to fix things at the CPython C API level”, there’s no clear timeline or roadmap, which leads people to just… wait indeed (same for the stable ABI vs. free-threading, incidentally).

8

I think the other aspect that has been mentioned, but not really picked up on, is “which alternative(s) offer the best hope for good performance across other Python implementations?”

That’s not necessarily something the CPython core developers can comment on, but it is a reasonable question for a developer wanting to write an extension to ask. And it’s something that is relevant to the CPython C API WG, in the sense that portability vs performance is often a trade-off, and if portability isn’t a priority for the WG, then using the C API may be a poor choice for developers who value that.