[question] Conan 2.x - Store conan create json output directly in conan package

[Jannis1994]

What is your question?

Hello together,
in the past we used the conaninfo.txt file for checking dependencies. Now in Conan2 I saw it is better to use the json output of conan create as some details are not in conaninfo.txt anymore (Reference: #16763).
For instance I am missing:
[full_settings]
[full_options]
[recipe_hash]

I only get [options] and [requires]

So I used the following command:

conan create . -o *:variant=$variant -o *:adg_compatibility=$adg_version --user test -f json --update > conan_info.json

My question would be now, how can I add this conan_info.json file to conan package after already creating it?
Is this only possible with the metadata option or is there something similiar to conaninfo.txt.

Using the conanfile.py package method and taking it from the recipe folder was also not possible without errors.
One idea I also thought of was executing the conan create steps manually but then it is not a one-liner anymore.

Best regards,
Jannis

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @Jannis1994

Thanks for your question.

Yes, at this moment that json can only be added to the package metadata after the package has been created.
Note that the conan create could also have a --build=missing, or even --build=*, and it will build not only the current package, but also the dependencies. But that json file that you are capturing from the command cannot be built after the dependencies have been built first, so it is kind of a chicken and egg problem, if the end goal is to have some file stored inside every package that is being built.

This is the reason we are working to add a new capability to conan, see: #17203

The feature is called like sbom feature, but note that the graph.json created by conan commands is nothing but a complete graph bill of materials. It is possible that we will rename that feature to be more generic.

One of the key points for that feature is that every package should create the manifest/graph.json/sbom of itself, that is, including its subgraph of dependencies, and computing that subgraph explicitly is not available yet in Conan. But #17203 is adding that capability, so the idea is that creating and storing such file is doable, either directly in recipes, or in the new sbom plugin.

[Jannis1994]

Hi @memsharded thanks for your quick answer.
Then I will look for another way. Just to quickly clarify, the reduced output in conaninfo.txt is right?
There is nothing I am missing?

Old conaninfo.txt in Conan1.X:

[settings]


[requires]
    baseimage/24393_594_1

[options]
    adg_compatibility=50.1.1
    variant=GEN3

[full_settings]


[full_requires]
    baseimage/24393_594_1@test_channel/stable:4699249c164c74520477d3086cedd40fd03

[full_options]
    adg_compatibility=50.1.1
    variant=GEN3
    baseimage:adg_compatibility=50.1.1

[recipe_hash]
    382be077743eecf5591423ab9

[env]

New conaninfo.txt in Conan2.X:

[options]
adg_compatibility=50.1.1
variant=GEN3
[requires]
baseimage/icas1_50@test_channel/stable

Also the requires package version looks diferent.

In our project we used the conaninfo.txt really often, especially the properties conan.package.*

[memsharded]

The conaninfo.txt file in Conan 2 has been reduced, to contain exclusively the information that defines the package_id. In fact, the package_id is now just the hash of the file, this simplifies a lot the management and also the understanding of the different effects of the different factors (settings, options, dependencies) on the package-id.

The conaninfo.txt in Conan 1.X had a lot of information there that wasn't part of the package_id at all.

The [requires] part of the conaninfo.txt has also changed, because Conan 2 implements a much better and thorough model of the effects of the dependencies versions in the consumer packages. This model now takes into account things like the package types (being static, shared, interface, etc), and can save a lot of unnecessary re-building from source, while being very safe. You have more information about this model in https://docs.conan.io/2/reference/binary_model.html

[Jannis1994]

Thanks for the clarification.

Best regards,
Jannis

[Jannis1994]

closed

[memsharded]

Thanks to you for the feedback!

If you can, please follow #17203 too, it would be great to have your feedback when this is merged and release. Thanks again!

[Jannis1994]

Sure I will consider it, as soon as it is merged.

[Jannis1994]

Hello again, during migration to Conan2 we saw another problem. It is mainly about the former conan.requires attribute visible in Artifactory when selecting conaninfo.txt properties view. In the past some other scripts used it for further evaluation and it would be great if we could add this attribute atleast manually. Is it possible to extend the standard attributes (conan.package.channel, conan.package.license, conan.package.name etc,) with a custom attribute? I saw methods like package_info() but I dont get it visible in the properties overview and I dont want to add it as an option, as it is mainly a parameter like the version, we also have other custom properties in mind, that we would like to add.
So increasing the metadata would be great and be able to add custom attributes which are also visible like url, license, author, description etc.

Was thinking of something like the following:

def package_info(self):               
        self.conf_info.define("conan.package.test_attribute", "test_value123")

Could you maybe help us again?
If it is not possible, I think we have to use conan graph info and put the output to a json file. Do you know where the information is now stored within the conan package, if it is not anymore the conaninfo.txt? Or is it only reachable via a conan command and not directly in one metadata file?
Otherwise we may would use this approach without properties but files with the content we need: https://docs.conan.io/2/devops/metadata.html and self.package_metadata_folder in build method

Thank you and best regards,
Jannis

[memsharded]

Hello again, during migration to Conan2 we saw another problem. It is mainly about the former conan.requires attribute visible in Artifactory when selecting conaninfo.txt properties view. In the past some other scripts used it for further evaluation and it would be great if we could add this attribute atleast manually. Is it possible to extend the standard attributes (conan.package.channel, conan.package.license, conan.package.name etc,) with a custom attribute? I saw methods like package_info() but I dont get it visible in the properties overview and I dont want to add it as an option, as it is mainly a parameter like the version, we also have other custom properties in mind, that we would like to add.

Whatever is added to the self.info system, which is the one automatically defining those properties in Artifactory server, the main issue with that is that it is not only metadata anymore, but it will always become part of the package_id. And that will easily break the usage of the package binaries, if that metadata information do not match correctly also at consumption time.

The recommended way to define Artifactory properties would be to use the Conan extension commands, in the conan-extensions repo, and use the conan art:property command (https://github.com/conan-io/conan-extensions/blob/main/extensions/commands/art/cmd_property.py) to define properties, and you could call this directly from your CI pipeline code, to add any arbitrary properties to recipes and/or packages.

If you could/should use metadata files instead, it might depend on the specific use case and usage patterns. If it is just a couple of small key-value things, Artifactory properties might be better, but if more information is needed, maybe metadata files would be better. Note there is some work in #17203 to help generating "manifest" files like SBOMs while creating packages, without needing to modify or hardcode this in all recipes.

[Jannis1994]

Thanks for your detailed answer. I think the metadata approach fits good. Unfortunately it is only available for Conan2.X and not for Conan1.X. We would like to have a similar file location for legacy releases in conan1 format and new releases in conan2 format.
I guess something like: package/hash1/hash2/metadata is not possible in conan1 or are there any workarounds?

[memsharded]

Thanks for your detailed answer. I think the metadata approach fits good. Unfortunately it is only available for Conan2.X and not for Conan1.X. We would like to have a similar file location for legacy releases in conan1 format and new releases in conan2 format.
I guess something like: package/hash1/hash2/metadata is not possible in conan1 or are there any workarounds?

Storing things in extra folders inside the server side paths (or client side paths in Conan 1.X) can be quite problematic and break things. Conan 2 allowed this because it implemented some new logic to filter or manage differently "metadata" folders, but in Conan 1.X this will easily fail.

I'd probably focus efforts in upgrading to Conan 2 first, then try the metadata approach (also, because the new tools like the one in that PR will be in Conan 2.X). If this is completely impossible, maybe doing something like we have done in ConanCenter in the past, and we have seen other users doing similar things: define some parallel storage, for example in a generic repo, that follows similar folder conventions using the package/revision/package_id/package_revision folders to store things. And try to keep the logic completely separated from recipes to allow the transition to Conan 2, for example using hooks or something like that.

[memsharded]

Also an extra info, to clarify, just in case.: The idea is that any metadata approach (both Artifactory properties or Conan metadata files) is for actual strict metadata, that is, the main package resolution and usage shouldn't really require the metadata in the process, and could be able to work even without that metadata (metadata files in Conan 2 are actually not downloaded in every conan install).

[Jannis1994]

Thank you @memsharded for the suggestions. We will take one of this approaches.
Our problem is that we have two work streams. One is only legacy support and conan should not be upgraded anymore. The other one will be regulary maintained in the future and should use the latest conan.
Will the SBOMs future feature cause a breaking interface, like the change from Conan1 to 2? Or will it be just additional help?

Thank you.

[memsharded]

Will the SBOMs future feature cause a breaking interface, like the change from Conan1 to 2? Or will it be just additional help?

No, it will not be breaking at all, it will be released in one Conan 2.X versions, and the idea is that users will not be affected at all.

[Jannis1994]

Ok, thank you and for your effort putting in Conan2

[Jannis1994]

Hi @memsharded
I have two questions left, I got them from a colleague.

1. You mentioned that the conaninfo.txt file in Conan 2 has been reduced, is there also a plan to remove it later completely? We are asking because then we would switch already now to another approach on our side.
2. Is it possible to set properties like it is visible for conaninfo.txt (conan.package.author, conan.package.license) also for files in the new metadata folder directly from the conanfile.py logic? Or do I need to use there the jfrog rt sp command in a shell script, in order to be able to set properties for files? With properties we dont have to download files to check its content from the outside. I also guess adding a property directly to conaninfo.txt will also be then no good idea with jfrog rt sp, as you already explained with the hash calculation topic, right?

Thank you.

[memsharded]

You mentioned that the conaninfo.txt file in Conan 2 has been reduced, is there also a plan to remove it later completely? We are asking because then we would switch already now to another approach on our side.

No, there are no plans to remove it, the conaninfo.txt is a critical part of the Conan package system. It has been reduced to be a direct mapping to the package_id hash. So all the previous information that was there that was not used to compute the package_id hash has been dropped, but that doesn't mean the file will go away, that is almost impossible. What it is important to know about the file:

• Everything that goes there, every character will be part of the package_id hash
• This file is managed and created entirely by Conan, it is not allowed for users to modify it

Is it possible to set properties like it is visible for conaninfo.txt (conan.package.author, conan.package.license) also for files in the new metadata folder directly from the conanfile.py logic?

No, this is not possible, because of the above, there is a 1:1 relation now between the conaninfo.txt and package_id, so it cannot be used as a way to store or try to create meta-information.

There is no way to do such a thing from conanfile.py logic (there was never a way, neither in Conan 1), because this is mostly a server side specific thing, not related to the package itself.

Or do I need to use there the jfrog rt sp command in a shell script, in order to be able to set properties for files? With properties we dont have to download files to check its content from the outside. I also guess adding a property directly to conaninfo.txt will also be then no good idea with jfrog rt sp, as you already explained with the hash calculation topic, right?

We have the new conan art:property command in the conan-extensions that has a more direct mapping between Conan references (like recipe or package reference), which simplifies defining properties in Artifactory. But yes, using the jfrog rt commands or the conan art:property commands is the way to go.

Please let me know if this helps.

[Jannis1994]

Thank you again, that was exactly my question.

[Jannis1994]

• Everything that goes there, every character will be part of the package_id hash
• This file is managed and created entirely by Conan, it is not allowed for users to modify it

So you cannot recommend something like the following, even if it is technically possible after package creation to add a property (just tried it):
jfrog rt sp "repository/company/element/version/_//package//*/conaninfo.txt" "testing=some_value"

Then I only would do it to metadata after package creation and uploading to artifactory:
jfrog rt sp "repository/company/element/version/_//package//*/metadata/conan_metadata.json" "testing=some_value"

[memsharded]

jfrog rt sp "repository/company/element/version/_//package//*/conaninfo.txt" "testing=some_value"

Yes, this is possible, sorry I didn't understand it. What is not possible is to modify the contents of the conaninfo.txt file. You can add properties to it with jfrog rt sp without problems. The conan art:property command is just abstracting away the path details of "repository/company/element/version/_//package//*/conaninfo.txt", and using the pkg/version#revision:package_id#pkg_revision identifier. (Note that the command might set the property to different package_ids.

[Jannis1994]

Ah okay that is great, thank you, that helps us alot .