Silent ctypes.struct layout change from native to Windows when bitfields are used

[BCSharp]

Bug report

Bug description:

Python 3.14 implements a special field _layout_ in ctypes.Structure that allows the programmer to explicitly choose the layout rules for bitfields used by either GCC/Clang or MSVC. The default layout is appropiate for the OS platform (MSVC on Windows, GCC otherwise) which is expected behaviour. It is also possible to use field _pack_ to change the layout to match the compiler. This is currently only implemented for the MSVC layout, not for GCC.

The problem is that using _pack_ on non-Windows systems silently switches the layout rules from GCC to MSVC, which is an unexpected layout by default on non-Windows systems. I propose that this case be handled as an error (NotImplementedError) rather than the silent layout switch. The error message may contain a hint that if the MSVC layout is indeed what is intended, one need to select it explicitly by setting field _layout_.

From the Zen of Python:

Explicit is better than implicit.
Readability counts.
Special cases aren't special enough to break the rules.
Errors should never pass silently.

Tested on 3.14a6

CPython versions tested on:

3.14

Operating systems tested on:

macOS

[ZeroIntensity]

cc @encukou (you messed with bitfields recently, right?)

If this is valid, we should probably mark this as a blocker.

[BCSharp]

@ZeroIntensity I'm glad that you see it as a bug and not a feature request. I discovered it while implementing the same functionality in IronPython, and just couldn't bring myself to follow CPython in this case (normally IronPython follows what CPython does, if possible). It would be great if CPython changes this so that there are no uneecessary differences between IronPython and CPython.

[ZeroIntensity]

Yeah, I doubt it's intended behavior. Assuming that's correct (I wasn't part of designing or reviewing _layout_/_pack_, so I'm going to yield to someone else), it would be good to get this fixed before the beta freeze locks it in.

[encukou]

Hi! I have limited access to the computer & I'm writing from memory. Please verify my claims :)

Yes, 3.14 has an overhaul of struct layout; look at 3.13 for the previous status quo.
As far as I remember, _pack_ has always switched to the MSVC layout; changing that would be a breaking change too.

In both 3.13 and the current 3.14, the meaning of _pack_ is “MSVC #pragma pack”.
GCC/clangs's __attribute__((packed)) is not implemented/supported at all.
We could make that more explicit in the docs. (Unfortunately, after such time, people depend on the actual behaviour; it doesn't matter that there could be a better design that would match the docs.)
Would such a docs change (“_pack_ is “MSVC #pragma pack”) allow you to bring yourself to follow CPython? (If so, could you send a PR?)

---

We could deprecate _pack_ without _layout_ on non-Windows, but I fear that it would break existing users unnecssarily.

Of course, if __attribute__((packed)) is implemented soon (which is not on anyone's roadmap AFAIK), it might be nice to switch the default for “Linux with _pack_” to it.
But my current vague long-term plan is to make _layout_ a callable that constructs the field list, and gradually move away from configuration strings to something like _layout_ = gcc_sysv(packed=1). And if _pack_ will eventually be (soft-)deprecated, then I want to avoid “churn” for existing users: let them switch at once to a well-designed replacement, whenever it's ready.

[BCSharp]

To clarify (or add to the confusion) #pragma pack is also supported by GCC/Clang and is something different than __attribute__((packed)). I understand that _pack_ deals only with the former and not the latter, and that GCC packing is not yet supported in any form. It also makes sense that eventually a new scheme will be needed to support __attribute__((packed)), since it can be applied per field. For backward compatibility I see _pack_ being supported as an alternative to the new scheme, but then it will be even more confusing if it always means MSVC #pragma pack and never GCC #pragma pack.

If the silent layout change was always in Python, I see it as it was always a bug that went unreported. This is not surprising since till 3.14 there far more serious problems with handling bitfields (overlapping fields, spurious padding bits inserted that did not match the compiler) so I doubt this is widely used. Even if making it an error "breaks" some existing code that just happens to work correctly, there is a trivial remedy by adding _layout_ = 'msvc' (or whatever the codestring for MSVC will be in the final release).

At the very least, if not an error, please make it a warning that the layout was changed and that in the future this will become GCC pragma pack, so the user better be explicit.