Make SHA256 compression pluggable

[furszy]

Tackling the long-standing request #702.

Right now we ship our own SHA256 implementation, a standard baseline version that does not take advantage of any hardware-optimized instruction, and it cannot be accessed by the embedding application - it is for internal usage only.

This means embedding applications often have to implement or include a different version for their use cases, wasting space on constrained environments, and in performance-sensitive setups it forces them to use a slower path than what the platform provides. Many projects already rely on tuned SHA-NI / ARMv8 / or other hardware-optimized code, so always using the baseline implementation we ship within the library is not ideal.

This PR gives our users a way to bring their own SHA256 compression, either at build time or at runtime, while keeping the default behavior untouched for everyone else.

Compile-time: External SHA256 module

The built-in transform is moved out of hash_impl.hinto a small sha module with its own public header (secp256k1_sha.h). The idea is to stop treating the compression step as a private, hidden detail, and instead make it part of a public interface that users can swap out if they want to.
Both build systems gain optional support for pointing libsecp to an external implementation:

• Autotools: --with-external-sha256=<path>
• CMake: SECP256K1_EXTERNAL_SHA256_PATH

Note:
Due to our current limitation of disabling C++ builds (see this), the provided external implementation must be written in C. (A way to overcome this would be to compile and link this externally instead of doing it within the library).

Runtime: Context callback

For setups where we detect the available SHA256 compression function at runtime and cannot re-compile the sources, or when the function is not written in bare C89, there’s a new API:

secp256k1_context_set_sha256_transform_callback(ctx, fn_transform)

This lets users plug-in their hardware-optimized implementation into the secp256k1_context, which is required in all functions that compute sha256 hashes.
During the initial setting, as a sanity check, this mechanism verifies that the provided compression function is equivalent to the default one.

As a quick example, the changes required to implement this in Bitcoin-Core at runtime are very straightforward: furszy/bitcoin-core@f68bef0

Implementing this at compile time in Bitcoin Core is possible, but it would require changing the C version we build against (which is not recommended). For example, we cannot link the SHA-NI implementation we have in Core (even if rewritten in C89) because it depends on the arm_neon library, which requires C99.

[fjahr]

I guess this is more of a draft for initial review but I will spell it out anyway: It's currently missing some CI coverage for building with an external library as well as some docs. Curious what people think in terms of docs for something like this: Should there be extensive guidance in which context this is safe to use or would users be left on their own to try it out? Or are we assuming all users that go to this length are competent enough to judge if it's a good idea to bring their own sha256 or use the systems one?

[furszy]

Thanks for the feedback fjahr!

I guess this is more of a draft for initial review but I will spell it out anyway: It's currently missing some CI coverage for building with an external library as well as some docs.

Yeah, can create a CI job testing the compile-time pluggable compression function very easily. Thanks for reminding that.

And about the missing docs; yeah. I didn't add it because we currently don't have much documentation outside configure.ac / CMakeLists.txt. Happy to create a file for it.

Curious what people think in terms of docs for something like this: Should there be extensive guidance in which context this is safe to use or would users be left on their own to try it out? Or are we assuming all users that go to this length are competent enough to judge if it's a good idea to bring their own sha256 or use the systems one?

It’s not that we’re letting people plug in whatever they want. Both introduced features have guardrails:

1. Compile-time: we run all current tests against the provided implementation, plus a runtime self-test.
2. Runtime: besides the runtime self-test, have introduced an "equivalence check" that hashes known inputs and compares them against the library’s internal implementation before accepting the external function.

So you can bring your own compression function, but it still has to prove it behaves exactly like ours bit-for-bit.

Also, the target user here is someone who's actually written a hardware-optimized SHA256. It’s not like they stumbled into this by accident.

[real-or-random]

Argh, that's a breaking change. I didn't see that coming.

But I think it can be avoided. If the user wants to pass a custom nonce function, they anyway have no way of calling our internal function (whether it uses a custom SHA256 transform or not).

So we can keep the struct here and add the context arg only to our built-in nonce function. Our ECDSA signing function (and Schnorr signing, and ECDH, etc.) will need to special-case the built-in nonce function because it has a different function signature. Not elegant but it will do the job.

Maybe we can come up with something nicer if we (ever) add a modern ECDSA module and deprecate the ECDSA stuff in the main secp256k1.h file. edit: Probably no because this affects not just ECDSA.

[real-or-random]

Do we need this function given that there's already secp256k1_selftest_sha256(void), which is run when a context is created? The existing one seems a bit simpler than your function and it's already there, so using that will make the diff smaller (but it will need to be modified to obtain an optional context). And perhaps the existing one is a bit slower than yours. I'd say either function is fine but keeping both seems overkill.

(The existing selftest is currently in selftest.h. That's already wrong; it should have been put in a selftest_impl.h. But your approach is better: the SHA256 test function should be exposed from the internal hash module and then called from selftest.)

edit: Okay, I see now. It's annoying than I thought. We expose secp256k1_selftest(void) which doesn't get a context object... The reason it's exposed is that you can use it when using the static context.

So I believe it makes sense to check the SHA256 at two places:

• In secp256k1_selftest(void) because this will catch bugs in the built-in SHA256. This is particularly important with this PR given that it makes possible overriding the built-in implementation it at build time.
• When setting a custom SHA256 transform.

This means we may perform two checks in the worst case, but that won't be the end of the world. You get the overhead only at library initialization time. But the question remains: How should the test look like? And I still think what I said above is true: It's okay to have a single secp256k1_sha256_selftest function in the hash module, and this can be called from the selftest module and from secp256k1_context_set_sha256_transform_callback.

[real-or-random]

A potential problem with storing the callback in the secp256k1_sha256 struct is that you could have this series of events:

• User sets callback to cb1.
• User calls API function that calls secp256k1_sha256_initialize, which outputs some opaque object that contains the secp256k1_sha256 struct.
• User sets callback go cb2.
• Users passes the opaque object to another API function, which calls secp256k1_sha256_write, which calls cb1. UB.

(I believe) this cannot happen in the current code because there's no pair of API functions that behaves in this way, but it's a potential footgun for the future.

My suggestion is just passing the context object to every internal function that needs the SHA256 callback. (Another angle: State is annoying. We have state already in the context, so let's try to keep it there.)

[real-or-random]

I think it will be nice to define this in the internal hash module, just like secp256k1_ecmult_gen_context is defined in the ecmult_gen module.

[real-or-random]

I was wondering whether it makes sense to allow for a real return value here to make it possible for the callback to indicate some kind of failure (e.g., couldn't access external hardware, or malloc failed). We could then fall back to our implementation or simply call the error callback in order to crash.

My current conclusion is no: I can't imagine this being useful in practice. (If you call external hardware, this will be super slow anyway. If you use malloc for SHA256, you're doing it wrong.) Even if there's some failure, the callback will still have the possibility to crash the process.

[real-or-random]

What's the motivation for the rounds arg? (Do you think we'll ever call this with any other value than 1?)

[real-or-random]

What's the motivation for exposing the SHA256 compression?

Unless I'm missing something, then if anything, I think we'd want to expose the high-level SHA256 function. I'm not sure how useful it is, and we tried to keep the library focused on ECC. On the other hand, we'll have it anyway in the code base, and it's required for Schnorr sigs (and we anyway expose a tagged hash for Schnorr sigs). So if you ask me, it's fine to expose SHA256 as long as we add a comment that says that users shouldn't expect a highly performant implementation.

But perhaps it's better to debate the exposing in a separate PR.

[real-or-random]

Naming: I wonder whether it makes sense to avoid the word callback here. Sure, it's a callback, but whenever we said callback in the past 10 years of this library, it was about error handling. We even have type secp256k1_callback (which probably should have been called secp256k1_error_callback but it's too late now.)

Maybe we can just call it "function " and "function pointer", like we do in the bring-your-own-nonce-function interfaces.

[real-or-random]

It will be nice to have a typedef for the function type in the public header.

[real-or-random]

Concept ACK, great to see a PR for this!

[real-or-random]

Regarding the build stuff for the build-time override: We already have the possibility to have build-time overrides for the error callbacks, e.g., see AC_ARG_ENABLE(external_default_callbacks, below.

When it comes to the implementation of how the new override is configured and performed, I think it's similar to the existing overrides for the callback. (Except that these don't allow configuring the header to be included, which makes them rather useless... #1461)

So this will need a rework anyway [1]. This belongs to a different PR, of course, but I think the SHA override here is a good opportunity to think about a better design of a compile-time override, so we may want to iterate on this a bit more than expected. No matter how it's designed, I think we should, in the long run, have only one (non-deprecated) mechanism for a build-time override of a function.

[1] I don't think what I did for the error callbacks great, and I guess it would be okay to change it (I doubt that many users rely on this). Or even better, we could deprecate it and simply provide ways to provide build-time overrides for standard library symbols like printf, stderr, and abort. See for example the approach taken by mbedTLS (in the past ?) which is built with embedded systems in mind: https://github.com/Mbed-TLS/mbedtls/blob/550a18d4d6b29e62b6824201d8a49b8224e61c97/tf-psa-crypto%2Fdrivers%2Fbuiltin%2Finclude%2Fmbedtls%2Fplatform.h They provide macros for overriding standard library functions. So you have full flexibility as a user: if you've implemented a function that aborts, but it's not called abort() (perhaps because that is a reserved name) then you can link it to the library also with a different name.

[real-or-random]

nit: I think this one should also be an enable-style configure option instead of a with-style configure options; with is for compiling with external packages e.g., with-libxyz. And what the user provides here is not a package.