Incremental Build

Overview

In a build process, there are multiple intermediate files generated. If the build tool re-use the intermediate files which is generated in previous build, we say it's a incremental build. BaseTools provides 3 options for incremental build. They are timestamp based incremental build, hash based incremental build and cache based incremental build. This wiki will discuss these kinds of incremental build.

Design

For a platform build process, there are 5 phases. They are pre-build, autogen, make, genfds, postbuild. Basetools build.py implement the autogen, make and genfds phase, and leave the pre-build and post-build for platform implementation. Here I'll focus on BaseTools implemented phases.

Timestamp based incremental build

Timestamp based incremental build re-uses the output of AutoGen phase and the output in Make phase. For Make phase, timestamp based re-use is supported by make program naturally. The quality and efficiency of incremental build in Make phase are completely depends on make program. For AutoGen phase, the purpose of this method is to try to skip the step of creating autogen files.

There are 2 levels of checkpoint for AutoGen phase incremental build, platform level, module level.

Platform level checkpoint files are:

• fdf file and all its include fdf files
• dsc file and its include dsc file
• all dec file
• build options (build write build options into a file).
• PcdTokenNumber (build tool write PcdTokenNumber into a file).
• build_rule.txt
• tools_def.txt

Module level checkpoint files are:

• module inf file
• files under inf [source] section
• library inf files
• module's all header file.

Platform level checkpoints are evaluated in WorkspaceAutoGen._InitWorker function. That is to calculate out the latest modification time of those checkpoint files, and save that time (platform timestamp) in WorkspaceAutoGen object.

Module level checkpoints are evaluated in Module.CreateMakeFile function. That is to write all the checkpoint files as a list into "AutoGenTimeStamp" file.

As we mentioned above, the purpose of this method is to skip creating autogen.c, autogen.h and makefile, the CanSkip() function will be called in ModuleAutogen.CreateCodeFile and ModuleAutoGen.CreateMakeFile functions. The CanSkip() logic is as follow:

1. If "AutoGenTimeStamp" does not exist, return False
2. If the platform timestamp is bigger than "AutoGenTimeStamp" modification time, return False.
3. Get timestamp of each file list in "AutoGenTimeStamp" file, if there is one timestamp bigger than "AutoGenTimeStamp" modification time, return False
4. Otherwise return True.
5. Since in one build process, all the checkpoints file do not change, we use a global variable gSikpAutoGenCache to cache the result of CanSkip().

Timestamp based incremental build is the build-in feature in build tool.

Hash based incremental build

Hash based incremental build can skip the step of generating autogen files and the step of executing make together. ModuleAutoGen.CanSkipbyHash() function is used to check if a module's build result can be re-used. It's called before ModuleAutoGen.CreateCodeFile() and ModuleAutoGen.CreateMakeFile() functions in AutoGen phase. If CanSkipbyHash return True, build will skip this module's autogen process and make process as well. If CanSkipbyHash return False, timestamp based incremental build process take effect.

Hash value generation

There are 3 levels of hash.

Platform level hash

Platform level hash value is evaluated in WorkspaceAutoGen._InitWorker function, after the platform checkpoints for timestamp based incremental build being evaluated.

Platform level hash value is calculated by md5 algorithm with the file content of the file lists:

• fdf file and all its include fdf files
• dsc file and its include dsc file
• build options (build write build options into a file).
• PcdTokenNumber (build tool write PcdTokenNumber into a file).
• build_rule.txt
• tools_def.txt

The hash value is stored in build folder and a global variable.

Package level hash

Package level hash value is evaluated in WorkspaceAutoGen._InitWorker function, in the loop of Arches. Each package of each arch has a hash value.

Package level hash value is calculated by md5 algorithm with the file content of the file lists:

• dec file
• All files under package include paths.

The hash value is stored in build folder and a global variable.

Module level hash

Module level hash value is evaluated in ModuleAutoGen.CanSkipByHash function.

Module level hash value is calculated by md5 algorithm with the file content of the file lists:

• Platform Hash value
• Dependent package hash value
• All its library hash value
• Module itself INF file content
• File content of all files listed in Source section of INF file.

The hash value is stored in build folder and a global variable.

Hash value usage

Platform level hash value and package level hash value are a part of Module hash value. Module hash value will be calculated in very build process and compared with corresponding hash value created and stored in previous build. If two hash value is the same, then build will ignore this module's autogen action and make action; otherwise build tool will autogen and make and update the hash value for a module.

Hash based incremental build is enabled by --hash build option. Platform level hash value and package level hash value are calculate only once in on build process.

Cache based incremental build

Based on the hash value, cache based incremental build enable build tool to re-use the binary files out of Build folder. Those binary files and a hash value file make up the build cache. The build cache can be provided by user himself/herself or by others like CI infrastructure.

Cached based incremental build have 2 additional functions in Basetools. CopyModuleToCache and AttemptModuleCacheCopy. The corresponding build option --hash --binary-destination= is used to generate build cache. The build option --hash --binary-source= is used to use binary from build cache.

Producing build cache action happens after Make phase. Build tool copy the following files to build cache folder.

• Module hash file
• Module output files
• Module .depx file
• Module .gdb file
• Module OffsetBin file

Consuming build cache action happens in Module.CanSkipbyHash function. Build tool check if the module hash value(generated here) is equal to the value in module hash file in the build cache folder. If same, build tool will copy the binary files from cache folder to build folder.

Testing

Timestamp based incremental build

Hash based incremental build

Cache based incremental build

Conclusion

Thought on current status