perf: pre-compute required transcript hashes per security policy

[MrMistic]

Goal

Skip transcript hashing for hash algorithms that the connection's security policy can never require.

Why

Before cipher suite negotiation completes, s2n-tls hedges by hashing every transcript update with all hash algorithms (MD5, SHA-1, MD5+SHA-1, SHA-224, SHA-256, SHA-384, SHA-512). For most policies, several of these can never be needed. For example, a TLS 1.3-only policy can only ever use SHA-256 or SHA-384 (RFC 8446 §7.1), so the other hashes are wasted work on every ClientHello.

How

At s2n_security_policies_init(), pre-compute for each security policy the set of hash algorithms that could be needed before negotiation:

• MD5, SHA-1, and MD5+SHA-1 only if the policy allows < TLS 1.2 (legacy PRF and default signature schemes)
• Each cipher suite's PRF hash (SHA-256 or SHA-384)
• Each TLS 1.2-eligible signature scheme's hash, since the TLS 1.2 CertificateVerify hash is always chosen from the policy's signature preferences. TLS 1.3-only schemes are skipped because TLS 1.3 CertificateVerify signs the cipher suite's PRF hash.

Connections copy the pre-computed bitmap at config-set time (s2n_connection_set_config / s2n_connection_set_security_policy). The existing post-ServerHello narrowing in s2n_conn_update_required_handshake_hashes is unchanged and still reduces to the single negotiated hash.

Callouts

• The bitmap is stored in the mutable security_policy_selection[] table since s2n_security_policy structs are const. Policies not in the table (e.g. test-constructed) are computed on the fly.
• Policies missing cipher or signature preferences fall back to requiring all hashes (only possible for test-constructed policies; the public API validates both).
• TLS 1.3-only policies narrow to just SHA-256 + SHA-384. Policies allowing TLS 1.2 with restricted signature preferences also benefit. Legacy policies with broad signature preferences keep most hashes, as required for correctness.

Testing

• Full handshake, connection, PRF, and security policy suites pass (~541,000 assertions)
• Client auth paths specifically exercised: s2n_client_cert_verify_test, s2n_client_auth_handshake_test, s2n_mutual_auth_test
• Transcript-hash-sensitive edge cases: SSLv3 PRF, extended master secret, PSK binders, session resumption, HelloRetryRequest transcript recreation (~13,300 assertions)

Related

N/A

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[kaukabrizvi]

Could you please use the PR template for the description: https://github.com/aws/s2n-tls/blob/main/.github/PULL_REQUEST_TEMPLATE.md

[MrMistic]

Could you please use the PR template for the description: https://github.com/aws/s2n-tls/blob/main/.github/PULL_REQUEST_TEMPLATE.md

yep, fixed

[kaukabrizvi]

Is connection reuse in scope for this optimization? s2n_connection_wipe preserves conn->config but resets required_hash_algs back to all 7, so a wiped-and-reused connection would re-hedge all 7 hashes unless set_config is called again.