OpenSSL 3 support

[svenschwermer]

Using libkmsp11.so (version 1.3) with OpenSSL 3 doesn't appear to work. I'm getting aborts and core dumps (see attached)

Process 288243 (openssl) of user 1000 dumped core.
Module /tmp/libkmsp11.so without build-id.
Module /tmp/libkmsp11.so
Stack trace of thread 288243:
#0  0x00007c9e722ac83c n/a (libc.so.6 + 0x8e83c)
#1  0x00007c9e7225c668 raise (libc.so.6 + 0x3e668)
#2  0x00007c9e722444b8 abort (libc.so.6 + 0x264b8)
#3  0x00007c9e71bec753 n/a (/tmp/libkmsp11.so + 0x424753)
#4  0x00007c9e71bef6d4 n/a (/tmp/libkmsp11.so + 0x4276d4)
#5  0x00007c9e71bead70 n/a (/tmp/libkmsp11.so + 0x422d70)
#6  0x00007c9e71b2bd80 n/a (/tmp/libkmsp11.so + 0x363d80)
#7  0x00007c9e71b347ef n/a (/tmp/libkmsp11.so + 0x36c7ef)
#8  0x00007c9e72bcc3f1 n/a (pkcs11.so + 0xc3f1)
#9  0x00007c9e72583785 n/a (libcrypto.so.3 + 0x183785)
#10 0x00007c9e725838d7 n/a (libcrypto.so.3 + 0x1838d7)
#11 0x00007c9e725cf7ac OPENSSL_LH_doall (libcrypto.so.3 + 0x1cf7ac)
#12 0x00007c9e7258185a n/a (libcrypto.so.3 + 0x18185a)
#13 0x00007c9e725803fa n/a (libcrypto.so.3 + 0x1803fa)
#14 0x00007c9e7262a2f0 OPENSSL_sk_pop_free (libcrypto.so.3 + 0x22a2f0)
#15 0x00007c9e725d882e OPENSSL_cleanup (libcrypto.so.3 + 0x1d882e)
#16 0x00007c9e7225ecc6 n/a (libc.so.6 + 0x40cc6)
#17 0x00007c9e7225ee10 exit (libc.so.6 + 0x40e10)
#18 0x00005b1dcb7f92ca n/a (openssl + 0x3d2ca)
#19 0x00007c9e72245cd0 n/a (libc.so.6 + 0x27cd0)
#20 0x00007c9e72245d8a __libc_start_main (libc.so.6 + 0x27d8a)
#21 0x00005b1dcb7f9795 n/a (openssl + 0x3d795)
ELF object binary architecture: AMD x86-64

I also couldn't build the latest master against OpenSSL 3. I couldn't find any documentation that OpenSSL 1.x is required 🤷

[tdbhacks]

Apologies for the late reply!

Right, if I remember correctly the library is built against 1.1.0 (see dependency).

OpenSSL 3 introduces a bunch of changes (including the transition from "engine" to "provider"), so the build errors you have seen seem reasonable, unfortunately. Keeping this open as a feature request for future consideration. We should probably also mention this somewhere in our docs, as you noted.

[es-fabricemarie]

I'm getting similar core dumps, making it completely unusable on latest Fedora 40.

I tried to compile the latest master branch against openssl3 using :
bazel build --config openssl //kmsp11/main:libkmsp11.so but it complained of conflicts between BoringSSL-openssl1 compat headers and Openssl3.

Then I tried to recompile the latest master branch without modifications and the build succeeded.

However when I try to sign something, I get this stacktrace:

#0  0x00007fe7c81ca0d2 in pkcs11_getattr_alloc (ctx=ctx@entry=0x3f591e987bb78c92, session=4802441702199765720, 
    object=object@entry=7857815905065540909, type=type@entry=288, value=value@entry=0x7ffc7285e5e0, 
    size=size@entry=0x7ffc7285e5d8) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_attr.c:62
#1  0x00007fe7c81ca8b0 in pkcs11_getattr_bn (ctx=ctx@entry=0x3f591e987bb78c92, session=<optimized out>, 
    object=object@entry=7857815905065540909, type=type@entry=288, bn=bn@entry=0x7ffc7285e640)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_attr.c:92
#2  0x00007fe7c81d07df in pkcs11_get_rsa (key=0x70bda0)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:197
#3  0x00007fe7c81d0b50 in pkcs11_get_evp_key_rsa (key=0x70bda0)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:265
#4  0x00007fe7c81cea12 in pkcs11_get_key (key0=key0@entry=0x70bda0, object_class=<optimized out>)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_key.c:456
#5  0x00007fe7c81ceaaa in pkcs11_rsa (key=0x70bda0) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:34
#6  pkcs11_get_key_size (key=0x70bda0) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:332
#7  pkcs11_private_encrypt (padding=1, key=0x70bda0, to=0x72ae50 "\232\326o", 
    from=0x741940 "010\r\006\t`\206H\001e\003\004\002\001\005", flen=51)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:91
#8  pkcs11_rsa_priv_enc_method (flen=51, from=0x741940 "010\r\006\t`\206H\001e\003\004\002\001\005", 
    to=0x72ae50 "\232\326o", rsa=<optimized out>, padding=1)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:384
#9  0x00007fe7c7dbfd86 in RSA_sign (type=<optimized out>, 
    m=m@entry=0x7ffc7285ebb0 "\372\b\334r\022\b\232\357\320̈́\232dW1,\267\304B軅\342\373\230\214,Z\201\266A\n", 
    m_len=m_len@entry=32, sigret=sigret@entry=0x72ae50 "\232\326o", siglen=siglen@entry=0x7ffc7285eb44, 
    rsa=rsa@entry=0x6f7820) at crypto/rsa/rsa_sign.c:307
#10 0x00007fe7c7dc2a91 in pkey_rsa_sign (ctx=0x6ff9c0, sig=0x72ae50 "\232\326o", siglen=0x7ffc7285ec50, 
    tbs=0x7ffc7285ebb0 "\372\b\334r\022\b\232\357\320̈́\232dW1,\267\304B軅\342\373\230\214,Z\201\266A\n", tbslen=32)
    at crypto/rsa/rsa_pmeth.c:178
#11 0x00007fe7c7d4f91b in EVP_DigestSignFinal (ctx=<optimized out>, sigret=0x72ae50 "\232\326o", siglen=0x7ffc7285ec50)
    at crypto/evp/m_sigver.c:677
#12 0x00007fe7c7da0e04 in PKCS7_SIGNER_INFO_sign (si=si@entry=0x70d780) at crypto/pkcs7/pk7_doit.c:934
#13 0x00007fe7c7da2025 in do_pkcs7_signed_attrib (mctx=<optimized out>, si=0x70d780) at crypto/pkcs7/pk7_doit.c:711
#14 PKCS7_dataFinal (p7=p7@entry=0x6f02f0, bio=bio@entry=0x5e87c0) at crypto/pkcs7/pk7_doit.c:833
#15 0x0000000000403103 in IDC_set (p7=p7@entry=0x6f02f0, si=si@entry=0x70d780, image=<optimized out>) at idc.c:216
#16 0x0000000000402947 in main (argc=<optimized out>, argv=<optimized out>) at sbsign.c:274

I works fine on Fedora 39:

• openssl-pkcs11-0.4.12-4.fc39.x86_64
• openssl-libs-3.1.1-4.fc39.x86_64

But fails as described above with Fedora 40:

• openssl-libs-3.2.1-2.fc40.x86_64
• openssl-pkcs11-0.4.12-8.fc40.x86_64

Apparently the engines in OpenSSL3 are still supposed to work, and the migration to providers instead is not necessary right this minute.

Note: the version of openssl on the machine should not change anything, as this is a pkcs11 library that could be called by anything (not necessarily OpenSSL). In my case kmspkcs11 is called by p11kit which is called by openssl engine.