Allow to pass custom CFLAGS and LDFLAGS to the compilation of builtin extension modules

[pablogsal]

This gives redistributors more control over the compilation flags of the extension modules. This allows among other things to pass custom RPATH/RUNPATH values that point to the correct PLATDIR

Check #131781 for the discussion and rationale of why we decided to allow this

Linked PRs

• gh-131842: Allow to pass custom CFLAGS and LDFLAGS to the compilation of builtin extension modules #131847

[ned-deily]

@erlend-aasland, isn't this already supported for individual modules with the *_CFLAGS and *_LIBS configure options? I don't know if we still have or did have a global options for all extension modules.

[erlend-aasland]

Yes, this is already supported (and mostly documented IIRC).

[pablogsal]

What we are missing is a global option for ALL extension modules. Also, LIBS != LDFLAGS

Additionally, the individual flags aren't available for all modules, just a select few.

[erlend-aasland]

Right, we do not have anything like that IIRC. Thanks for clarifying.

[pablogsal]

Small note x-posted from #131781 (comment):

I checked and @ned-deily was right: as confusing as it is the *_LIBS are treated as LDFLAGS but only apply to individual (and not all) extension modules. Also they are only available for a select collection of them.

The PR still applies as we are adding a global addition for all of them

[erlend-aasland]

Correct, configure is not consequent regarding LIBS and LDFLAGS. That's worth a cleanup.

[ned-deily]

as confusing as it is the *_LIBS are treated as LDFLAGS but only apply to individual (and not all) extension modules

One further point of confusion (to which I may have added in my comment above): AFAIK, the intent is that those *_CFLAGS and *_LIBS configure options are for each third-party library we use (not each extension module) and their values apply to every extension module that uses that library; there are some third-party libs that are linked to more than one stdlib extension module. If there are any third-party libs that aren't covered, that's a bug.

[ned-deily]

Thinking about this topic some more, especially after we've cleared up some initial confusion, it seems to me a key insight that the changes for 3.11+ made was to focus on the third-party libraries used by the various stdlib extension modules independently from the extension modules themselves, something we didn't really have previously.

In general, we now have at least two ways to tailor compile and link details for each third-party library or package of libraries: one, using pkg-config and the .PC files supplied by most third-party libraries; and two, the ability to override any individual or missing pkg-config values for each lib through the *_CFLAGS and *_LIBS variables to ./configure. That would seem to cover any requirements for specifying --rpath info for relocatable third-party libraries, one of the motivating examples for this proposal.

If so, is there anything missing? There aren't that many third-party libs that we use but I suppose you could argue that it may be tedious to add *_CFLAGS and *_LIBS arguments to one's ./configure command if most or all of the third-party libs to be used have been built to use an rpath and they are all to be installed in the same prefix (and even though this should just be a once-and-done configuration exercise). So perhaps for this use case, it would be useful to provide global defaults for _CFLAGS and _LDFLAGS for the third-party library configuration logic (but not for all stdlib extension modules). But there also might be cases where one or more third-party libs have different install prefixes for whatever reason (different distributor, picked up from the operating system, etc) and, if so, we should be careful to allow *_CFLAGS and/or *_LIBS config options specified to override any such global third-party defaults. I don't know how useful such a global option would be - I don't recall third-party distributors asking for it - but it might be good to ask (say, on Discuss).

Beyond that can we identify any use cases where there is a need to supply additional information to stdlib extension modules builds that isn't related to third-party libs and isn't already covered by existing build options? If so, those would be more in the spirit of the original PR. If so and we deem both useful, maybe we need both kinds of options: ones that supply overridable defaults for (stdlib users of) the third-party libs and one that supplies options for all stdlib extension modules (whatever those options might be). In any case, it would be good to specify motivating use cases.

As background, I've spent more than a dozen years on and off in this neck of Python's build system and every time I delve into it my brain ends up hurting. My observation is that every couple of years someone comes along with an issue in this area with the idea that it can be fixed by adding or deleting some new CFLAGS, CPPFLAGS, and/or LDFLAGS variables and, as a result, we accumulated a lot of cruft in configure and the makefile. I know I've been guilty of this myself. That's why I was very pleased to see the thoughtful design and work that @tiran and @erlend-aasland and others put into this a few years back by adopting newer, more standard configuration options, removing some of the cruft, and focusing on how we configure using the third-party libraries. I think we should be really careful that any changes we make in this area do not compromise this design. (I also think that, whatever if anything we decide to do here, that the topic of adding support for additional third-party libraries should be covered in the devguide and that documentation for the use of the existing options should be made more visible to builders of Python).

Other thoughts?

[pablogsal]

The global flags for extension modules are still needed because:

1. While the per-library approach works well for known dependencies, it creates a maintenance burden when modules evolve. Without global flags, each time a module adds new dependencies, redistributors must remember to update specific flags for that module.

2. Global flags provide a safety net that ensures consistent compilation and linking options across all extension modules, preventing situations where a module is compiled without necessary flags because someone forgot to add them.

3. For environments where common settings (like rpaths) need to be applied across many modules, repeating these settings for each individual library is inefficient and error-prone.

4. The current approach requires redistributors to track and update multiple configuration points when making systemic changes, increasing the risk of inconsistencies.

The per-library configuration remains valuable for specific needs, but global extension module flags would complement this by providing defaults that ensure consistent behavior while reducing maintenance overhead.​​​​​​​​​​​​​​​​

[pablogsal]

@erlend-aasland what are your thoughts?