fix: isolate reference implementation RNGs

[exocognosis]

Summary

Isolate every randomness path used by the production libbitcoinpqc targets and make the entropy contract explicit at the reference implementation boundary.

This change:

• removes the shared randombytes() adapter, its global entropy state, its buffer-cycling behavior, and its /dev/urandom fallback
• adds an explicit seeded ML-DSA key generation entry point
• adds an explicit optrand SLH-DSA signing entry point
• passes caller entropy and deterministically derived signing inputs directly into the vendored implementations
• excludes the vendored unseeded key generation and signing entry points from production library targets
• documents the complete RNG inventory, including vendored standalone files that are not linked into libbitcoinpqc
• adds a regression test that traps accidental calls to the legacy randombytes() hook

Root cause

The initial entropy refactor centralized the two reference implementations behind src/randombytes_custom.c. That adapter allowed the public API to inject caller-provided bytes, but it also retained hidden process-global state and silently read /dev/urandom whenever that state was absent.

This created several problems:

• the implementation contradicted the documented promise that libbitcoinpqc never sources randomness internally
• a lower-level or incorrectly ordered call could become nondeterministic without the caller requesting it
• failure to open or fully read /dev/urandom produced zero-filled output while the void randombytes() interface could not report the failure
• concurrent operations shared mutable entropy pointers and offsets
• requests larger than the injected buffer silently cycled and reused bytes
• the actual algorithm-specific entropy requirements remained obscured behind a generic hook

The existing user entropy documentation described the intended public behavior, but the build did not structurally enforce that behavior.

Proposed solution

The CMake library targets now define LIBBITCOINPQC_EXPLICIT_ENTROPY. This mode compiles only reference implementation entry points whose randomness inputs are explicit.

Operation	Explicit input
secp256k1 key generation	first 32 caller bytes used as the secret scalar
secp256k1 signing	deterministic libsecp256k1 path with aux_rand32 = NULL
ML-DSA key generation	first 32 caller bytes passed to crypto_sign_seed_keypair()
ML-DSA signing	first 32 bytes of SHAKE-256 over `sk
SLH-DSA key generation	first 48 caller bytes passed to crypto_sign_seed_keypair()
SLH-DSA signing	first 16 bytes of the existing deterministic derivation passed as optrand

The default standalone behavior of the vendored Dilithium and SPHINCS+ trees remains available when LIBBITCOINPQC_EXPLICIT_ENTROPY is not defined. Their original randombytes.c and NIST KAT RNG files remain in the source tree for upstream Makefiles and test-vector tooling, but they are not part of the production CMake targets.

The public bitcoin_pqc_* API is unchanged. Existing ML-DSA and SLH-DSA key and signature golden vectors remain unchanged.

Regression coverage

The new rng_isolation test provides its own trap implementation of randombytes() and exercises key generation, signing, and verification for secp256k1 Schnorr, ML-DSA-44, and SLH-DSA-SHA2-128s. The test fails if any public algorithm path calls the legacy hook.

The entropy documentation now records:

• exact bytes consumed by each algorithm
• deterministic signing behavior
• which vendored RNG files remain and why
• the boundary between the production library and independently compiled reference code
• the separate 32-byte secp256k1 and 128-byte PQC minimum input policies

Impact

• libbitcoinpqc no longer contains a hidden system entropy source.
• Production key generation and signing no longer depend on shared mutable RNG state.
• Callers retain full control over key generation entropy.
• Deterministic public behavior and existing golden vectors are preserved.
• The production library no longer exports unseeded vendored signing and key generation helpers that were not part of the installed public API.
• Standalone vendored reference builds retain their original unseeded interfaces.

Verification

• Release CMake build with examples and tests enabled
• CTest release suite: 4 passed, 0 failed
• AddressSanitizer and UndefinedBehaviorSanitizer suite: 4 passed, 0 failed
• Existing ML-DSA and SLH-DSA golden key and signature vectors passed
• SLH-DSA direct reference cross-check matched the existing golden signature
• Default standalone Dilithium and SPHINCS+ signing sources compiled without explicit-entropy mode
• Static and shared library symbol audits found no randombytes, /dev/urandom, getrandom, or arc4random dependency
• Release install layout contained the expected static library, shared library, and public headers
• git diff --check

Validation notes

• The library targets and new isolation test compile with -Wall -Wextra -Werror.
• The full warnings-as-errors test build reaches an existing unused mem_eq_hex helper in tests/secp256k1_schnorr_test.c. That file is outside this change.
• The full Nix gate was unavailable locally because nix and just are not installed. The host CMake path used by the non-Nix CI job passed.

Fixes #12

[cryptoquick]

While I wasn't really wanting to modify the reference implementations here, I appreciate this work, and I will also be working to reimplement libbitcoinpqc entirely in Systems Lean once I have that work finished.

Much appreciated.

[exocognosis]

No problem.