Rework Buildtime Random Stack Cookie Values to Improve Incremental Build Times and Ensure Binary Reproducibility

[TaylorBeebe]

Description

If the stack cookie value is randomized in the AutoGen.h file each build, the build system will determine the module/library must be rebuilt causing effectively a clean build every time. This also makes binary reproducibility impossible.

This PR updates the early build scripts to create 32 and 64-bit JSON files in the build output directory which each contain 100 randomized stack cookie values for each bitwidth. If the JSON files are already present, then they are not recreated which allows them to be stored and moved to other builds for binary reproducibility. Because they are in the build directory, a clean build will cause the values to be regenerated.

The logic which creates AutoGen.h will read these JSON files and use a hash of the module GUID (the hash seed is fixed in Basetools) to index into the array of stack cookie values for the module bitwidth. This model is necessary because there isn't thread-consistent data so we cannot use a locking mechanism to ensure only one thread is writing to the stack cookie files at a time. With this model, the build threads only need to read from the files.

• Impacts functionality?
  • Functionality - Does the change ultimately impact how firmware functions?
  • Examples: Add a new library, publish a new PPI, update an algorithm, ...
• Impacts security?
  • Security - Does the change have a direct security impact on an application,
    flow, or firmware?
  • Examples: Crypto algorithm change, buffer overflow fix, parameter
    validation improvement, ...
• Breaking change?
  • Breaking change - Will anyone consuming this change experience a break
    in build or boot behavior?
  • Examples: Add a new library class, move a module to a different repo, call
    a function in a new library class in a pre-existing module, ...
• Includes tests?
  • Tests - Does the change include any explicit test code?
  • Examples: Unit tests, integration tests, robot tests, ...
• Includes documentation?
  • Documentation - Does the change contain explicit documentation additions
    outside direct code modifications (and comments)?
  • Examples: Update readme file, add feature readme file, link to documentation
    on an a separate Web page, ...

How This Was Tested

Tested by building and confirming the reduced build time for debug builds after the initial build.

Integration Instructions

N/A

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 1.23%. Comparing base (c8f9883) to head (8c587da).

Additional details and impacted files

@@               Coverage Diff               @@
##           release/202311     #773   +/-   ##
===============================================
  Coverage            1.23%    1.23%           
===============================================
  Files                1302     1302           
  Lines              332084   332084           
  Branches             6683     6683           
===============================================
  Hits                 4117     4117           
  Misses             327891   327891           
  Partials               76       76           

Flag	Coverage Δ
MdeModulePkg	0.69% <ø> (ø)
MdePkg	5.37% <ø> (ø)
NetworkPkg	0.00% <ø> (ø)
PolicyServicePkg	30.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.