Consider using git-submodules instead of external/fetch_sources.py

[rossburton]

In Linux distributions it's common to have strict rules around sources, for example it's not uncommon for packages to be built in an environment without networking so that it can be proven that the list of sources is accurate for license compliance, source mirroring is complete, etc.

The use of external/fetch_sources.py and external/vulkan-validationlayer/src/scripts/update_deps.py makes this almost impossible, because it's very difficult to download the right sources in advance for mirroring/archiving, and then put them in the right place for a build that works in a non-networking environment.

Have you considered using git submodules as an alternative to these scripts? They're integrated into git, can automatically update when needed, and a single clone of this repository can fetch all of the dependencies at once.

[AlexGalazin-IMG]

submodules have a notoriously bad rep and actually none of the Khronos repos (as far as I'm aware) use them.

[rossburton]

Their reputation is undeserved IMHO, and random scripts that go and download things have a worse reputation. Building this repository means running two different scripts.

[AlexGalazin-IMG]

The fetch_sources.py script was initially developed for downloading libpng and zlib sources and then was gradually extended to support other type of dependencies.

It deals not only will full git repositories but also with individual files as well as other types of dependencies.

This script fetches dependencies at specific revisions that are known to work well with the CTS code.

For git repos it supports the clone operation through either ssh or https depending on the user preference/requirement. This is an important feature when used for development that depends on non-public versions of dependencies.

Validation layers and hence external/vulkan-validationlayer/src/scripts/update_deps.py are completely optional.

Perhaps, you could explain your use case in more details. As per the current README you need to do

git clone
python external/fetch_sources.py

You'll then have all necessary sources to build the CTS without any network available,

[rossburton]

You'll then have all necessary sources to build the CTS without any network available,

Not quite true. The VVL fetcher then does another fetch at configure time.

We build entire linux distributions and a key part of several features is the ability to do fully offline builds: your build machine can be entirely disconnected from the internet and builds can be done from a prefetched source mirror. This underpins several features such as the ability to produce SBOMs of all the software used, source mirrors, etc.

As such we want a complete list of the sources used in advance. My current solution is a script that parses the main and VVL's fetching scripts separately (as they're different tools entirely( to generate a list of all of the repositories used, but this is suboptimal to say the least.

[AlexGalazin-IMG]

Validation layers are not required for the CTS to run, they are completely optional.

[rossburton]

Interesting. I'm a distro monkey, not a graphics guru: what's the impact of VVL not being present, and how do I do that?

[AlexGalazin-IMG]

Validation layers are used in the CTS development to ensure that the tests themselves do not violate the Vulkan valid usage requirements https://registry.khronos.org/vulkan/specs/latest/html/vkspec.html#fundamentals-validusage. Be default, they are disabled https://github.com/KhronosGroup/VK-GL-CTS/blob/main/framework/common/tcuCommandLine.cpp#L248.

To skip VVL configuration and build you need to use this cmake option -DSELECTED_BUILD_TARGETS=<list of targets> (https://github.com/KhronosGroup/VK-GL-CTS/blob/main/README.md#selecting-a-subset-of-targets-to-build-by-default)