[feature] Better build caching support in Conan 2.0 (as compared to 1.x)

[johschmitz]

A topic that was always quite a weak point with Conan 1.x is the topic of build caching support. Obviously since Conan is not a full blown build system one may claim that Conan is also not fully responsible for this topic. Nevertheless it does have a very strong impact on the file system layout during build time and therefore should be held responsible for at least not preventing build caching or making it very difficult to implement/configure.
The minimum I would expect/wish for is a description and some recommendations in the documentation how to use Conan together with ccache locally.
A more advanced requirement would be to better support shared build caches like sccache or similar alternatives.

• I've read the CONTRIBUTING guide.

[memsharded]

I'd also like to see something a bit more explicit about this point for 2.0.
But the first point is try to define how a succesful proof of concept looks like, define the basic requirments, UI, etc.
For example:

• What build systems need to be supported initially (i'd say only CMake?)
• How does the basic usage of ccache looks like in that build system, without Conan. Define a basic flow without Conan
• What are the challenges, bottlenecks, poor UI to solve with current Conan 2.0 (the released beta for example)
• How does the UI for Conan looks like to define opt-in into the ccache usage. Define the scope of the feature.

As we are already approaching 2.0 GA (already in beta), it doesn't look like that such a new feature will be in 2.0 scope but in 2.X. Looking forward to getting more detailed feedback from users.

[johschmitz]

I will link this old tickets #1597, #7693 as a starting point.
Will need to look more into this and experiment to see how to combine build caching with Conan. I was actually hoping to find an opinionated solution within the Conan ecosystem which works well enough. As of now it still seems to be secret sauce knowledge though..

[pwqbot]

Hi @memsharded!

I'm currently working with remote ccache and conan2 and have encountered an issue related to the calculation of cache hash using include paths. Some conan packages seem to generate different include paths on separate machines, even when everything else remains the same.

For example, the include path for OpenSSL 3.1.0 might vary like this: ~/.conan2/p/opens738d1e90ec137 vs ~/.conan2/p/opensb892b04917b0. Moreover, every time I reinstall the package, the path changes. However, other libraries, such as zlib, consistently generate the same include paths.

I am unsure how conan calculates these paths. If the package version and toolchain are the same, having consistent include paths would greatly simplify the integration with remote ccache. Could you please provide some insight into this issue or suggest possible ways to ensure consistent include paths across different machines and installations?

Thank you for your assistance!

[memsharded]

Hi @pwqbot

Thanks for your report.
I have been having a look, this is the related code:

@staticmethod
    def _short_hash_path(h):
        """:param h: Unicode text to reduce"""
        h = h.encode("utf-8")
        md = hashlib.sha256()
        md.update(h)
        sha_bytes = md.hexdigest()
        # len based on: https://github.com/conan-io/conan/pull/9595#issuecomment-918976451
        # Reduce length in 3 characters 16 - 3 = 13
        return sha_bytes[0:13]

    @staticmethod
    def _get_path(ref: RecipeReference):
        return ref.name[:5] + DataCache._short_hash_path(ref.repr_notime())

As you can see, this should be a perfect mathematical, constant hash of the recipe/package reference. That means that the folder should always be exactly the same one for the same recipe (same revision) and the same package (same package revision)

Could you please double check that you are installing exactly the same recipe and package revision in both machines? A log of the conan install in the 2 different machines could help, it should contain everything there, including the revisions and the different folder names.

[pwqbot]

Install in the same machine twice will get different result.
conanfile.py

from conan import ConanFile
from conan.errors import ConanInvalidConfiguration


class test_ccache(ConanFile):
    name = "contek_adapter"
    version = "0.0.0"
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"

    def requirements(self):
        self.requires("openssl/3.1.0")

first time.

openssl/3.1.0: package(): Packaged 3 files: c_rehash, openssl, LICENSE
openssl/3.1.0: package(): Packaged 1 '.txt' file: LICENSE.txt
openssl/3.1.0: package(): Packaged 135 '.h' files
openssl/3.1.0: package(): Packaged 2 '.a' files: libcrypto.a, libssl.a
openssl/3.1.0: package(): Packaged 1 '.cmake' file: conan-official-openssl-variables.cmake
openssl/3.1.0: package(): Packaged 2 '.dylib' files: legacy.dylib, fips.dylib
openssl/3.1.0: Created package revision 7c0c796248b58e869035b45d1aa35e34
openssl/3.1.0: Package '144c355fd5d237d22e14c77819fe31f3933f7e4c' created
openssl/3.1.0: Full package reference: openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#7c0c796248b58e869035b45d1aa35e34
openssl/3.1.0: Package folder /Users/benben/.conan2/p/opens7252b9b44ae00/p
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: openssl/3.1.0, zlib/1.2.13
WARN:     'cpp_info.build_modules' used in: openssl/3.1.0
WARN:     'env_info' used in: openssl/3.1.0

======== Finalizing install (deploy, generators) ========
conanfile.py (contek_adapter/0.0.0): Writing generators to /Users/benben/tmp/build
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeToolchain' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: conan_toolchain.cmake
conanfile.py (contek_adapter/0.0.0): Preset 'conan-release' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-release'
conanfile.py (contek_adapter/0.0.0): If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/Users/benben/tmp/build/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: CMakePresets.json
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeDeps' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): Generating aggregated env files
conanfile.py (contek_adapter/0.0.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully

second time.

openssl/3.1.0: package(): Packaged 3 files: c_rehash, openssl, LICENSE
openssl/3.1.0: package(): Packaged 1 '.txt' file: LICENSE.txt
openssl/3.1.0: package(): Packaged 135 '.h' files
openssl/3.1.0: package(): Packaged 2 '.a' files: libcrypto.a, libssl.a
openssl/3.1.0: package(): Packaged 1 '.cmake' file: conan-official-openssl-variables.cmake
openssl/3.1.0: package(): Packaged 2 '.dylib' files: legacy.dylib, fips.dylib
openssl/3.1.0: Created package revision a480b3336aa4e36f9e4011ab7435d4dc
openssl/3.1.0: Package '144c355fd5d237d22e14c77819fe31f3933f7e4c' created
openssl/3.1.0: Full package reference: openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#a480b3336aa4e36f9e4011ab7435d4dc
openssl/3.1.0: Package folder /Users/benben/.conan2/p/opens1a90f1ceb4970/p
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: openssl/3.1.0, zlib/1.2.13
WARN:     'cpp_info.build_modules' used in: openssl/3.1.0
WARN:     'env_info' used in: openssl/3.1.0

======== Finalizing install (deploy, generators) ========
conanfile.py (contek_adapter/0.0.0): Writing generators to /Users/benben/tmp/build
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeToolchain' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: conan_toolchain.cmake
conanfile.py (contek_adapter/0.0.0): Preset 'conan-release' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-release'
conanfile.py (contek_adapter/0.0.0): If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/Users/benben/tmp/build/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: CMakePresets.json
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeDeps' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): Generating aggregated env files
conanfile.py (contek_adapter/0.0.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully

the reference is
openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#7c0c796248b58e869035b45d1aa35e34 vs
openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#a480b3336aa4e36f9e4011ab7435d4dc

[memsharded]

Thanks for the feedback.
I see what is happening: you are building openssl from source twice. The build of openssl is not deterministic/hermetic/reproducible. Each time it will create a different binary, with a different checksum, and that produces a new revision, so it will be in a different folder (Just in case, achieving deterministic builds is super challenging, and outside of Conan scope, nothing that Conan can do there: https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html)

If the second time, instead of re-building from source, you do conan .... --build=missing, as the previous binary is there, it will have the same revision and be in the same folder.

[pwqbot]

Thanks for the feedback. I see what is happening: you are building openssl from source twice. The build of openssl is not deterministic/hermetic/reproducible. Each time it will create a different binary, with a different checksum, and that produces a new revision, so it will be in a different folder (Just in case, achieving deterministic builds is super challenging, and outside of Conan scope, nothing that Conan can do there: blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html)

If the second time, instead of re-building from source, you do conan .... --build=missing, as the previous binary is there, it will have the same revision and be in the same folder.

Thanks! So if I download binary from conan center the folder will be the same? I am building with gcc12, it seems like gcc12 binary package is not yet supported in conan center.

[memsharded]

Thanks, I am building with gcc12, it seems like gcc12 binary package is not yet supported in conan center?

I guess not.
But in any case, it shouldn't be an issue of existing binaries in ConanCenter, but an issue that the second build from source shouldn't be done at all, instead using the --build=missing will avoid it.
Also a note that in production, relying directly on ConanCenter binaries is not recommended in general (not just for Conan and C++, but for all package management in general). If you are using Conan in production, it is strongly recommended to use your own server to store your packages and dependencies. This way it doesn't matter if ConanCenter doesn't have the gcc12 binaries, you build them once, store them in the server, and don't need to re-build it from sources again. Also, it will make you way more robust against possible changes in ConanCenter recipes, network outages, etc.

[pwqbot]

Thanks, I am building with gcc12, it seems like gcc12 binary package is not yet supported in conan center?

I guess not. But in any case, it shouldn't be an issue of existing binaries in ConanCenter, but an issue that the second build from source shouldn't be done at all, instead using the --build=missing will avoid it. Also a note that in production, relying directly on ConanCenter binaries is not recommended in general (not just for Conan and C++, but for all package management in general). If you are using Conan in production, it is strongly recommended to use your own server to store your packages and dependencies. This way it doesn't matter if ConanCenter doesn't have the gcc12 binaries, you build them once, store them in the server, and don't need to re-build it from sources again. Also, it will make you way more robust against possible changes in ConanCenter recipes, network outages, etc.

Thanks! What is the best practice to store conan packages on local server, should I fork the conan-center-index and rebuild the packages I need and upload them to my server?

[memsharded]

Thanks! What is the best practice to store conan packages on local server, should I fork the conan-center-index and rebuild the packages I need and upload them to my server?

Yes, this is what many other Conan users are doing, and it seems to be a super reliable and convenient way for production, with reasonable effort. Actually, there are ongoing and planned features for making this even easier:

• The new conan-extensions repo is getting commands that could help to build things from a conan-center-index fork: https://github.com/conan-io/conan-extensions/pulls Very preliminary, but expected to keep improving in the following weeks and months.
• A new idea about using a fork as a Conan remote directly: POC: Remotes from local file tree #10433

[pwqbot]

Thanks! What is the best practice to store conan packages on local server, should I fork the conan-center-index and rebuild the packages I need and upload them to my server?

Yes, this is what many other Conan users are doing, and it seems to be a super reliable and convenient way for production, with reasonable effort. Actually, there are ongoing and planned features for making this even easier:

• The new conan-extensions repo is getting commands that could help to build things from a conan-center-index fork: conan-io/conan-extensions/pulls Very preliminary, but expected to keep improving in the following weeks and months.
• A new idea about using a fork as a Conan remote directly: POC: Remotes from local file tree #10433

Hi @memsharded, thank you for your suggestions. Currently, I have set up a CI machine and to upload binary files to Artifactory, I execute conan install --requires={pkg_name}/{version} --build=missing, followed by conan upload {pkg_name}/{version} -r =my_server}. This method is convenient as it eliminates the need to clone and update the conan-center-index, but I am unsure if it qualifies as a good practice.

[memsharded]

Not terrible, but not great either :)

• You are mixing binaries, created in ConanCenter, and your own binaries. You will have a mixture of binaries created in your own machines and in ConanCenter, and this is not great. It would be better if you also build the binaries for existing configurations in ConanCenter, just make sure that all binaries are yours. It is a little bit more of extra CI time, but totally worth it.
• If you depend on those ConanCenter recipes and do that continuously, you will be getting a continuos flow of recipe updates from ConanCenter too, new binaries, etc. While there might be interesting updates, it can also introduce some inestability, some occasional breaking changes or issues, etc. Building from your own fork guarantees complete immutability and full control of the dependencies without having to use extra things as lockfiles.

As usual, it depends on your needs, resources, constraints...