Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug] requires of a dependency of a dependency in build context not visible #16333

Open
maximilianmuehlbauer opened this issue May 23, 2024 · 5 comments · May be fixed by #16539
Open

[bug] requires of a dependency of a dependency in build context not visible #16333

maximilianmuehlbauer opened this issue May 23, 2024 · 5 comments · May be fixed by #16539
Assignees
@memsharded
Labels
stage: triaging
Milestone
2.5.0

Comments

@maximilianmuehlbauer
Copy link

Describe the bug

conan 2.3.0

For building Matlab Code with s-functions, certain packages have to be propagated with build=True, visible=True traits. It seems however that the dependencies of such packages in build context disappear at some point.

As minimal example, we have the following dependency tree:
pkgb -> libpkgb -> libpkgb_dependency

where pkgb contains the sfunction which should only reside in build context (think about a cross build scenario, the sfunction has to execute in Matlab only at build time and generates code), this sfunction depends on libpkgb (some shared library) and the generated code also depends on libpgkb so that one has to be available in both contexts. libpkgb_dependency (another shared library) is just a dependency of libpkgb, so it also has to be available in both contexts.

This s-function is then consumed by pkga (e.g. a Simulink model) as follows:
pkga (host context) -> libpkgb
pkga (build context) -> pkgb with visible=True trait.

It now seems that the dependencies of pkgb in build context disappear when pkga is consumed by pkga_consumer in host context. This makes it impossible to e.g. aggregate Matlab packages with s-functions for opening Matlab inside a VirtualBuildEnv to e.g. generate code.

Please see below zip file for exemplary conanfiles and the requirement traits used (maybe those are also incomplete?).

How to reproduce it

Execute ./create_all.sh:
pkga has libpkgb in its build requirements as expected:

pkga/sth@sth/sth: Forced build from source
Requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkga/sth@sth/sth#5de6164f0525938b5251b51dce86bb93:a9fea83b8cb8528171eec79a036ec70971964fe6 - Build
Build requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkgb/sth@sth/sth#64a4a8d5db953083748e775d6a8e6ac2:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache

pkga_consumer has libpkgb not anymore in its build requirements:

pkga_consumer/sth@sth/sth: Forced build from source
Requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkga/sth@sth/sth#5de6164f0525938b5251b51dce86bb93:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
    pkga_consumer/sth@sth/sth#36e174720d405f5bf14b41af15132a1e:163876653e2bf77f8d6c9a484c16abf689d068ed - Build
Build requirements
    pkgb/sth@sth/sth#64a4a8d5db953083748e775d6a8e6ac2:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
Skipped binaries
    libpkgb/sth@sth/sth, libpkgb_dependency/sth@sth/sth

dependency_propagation_build.zip
@memsharded
Copy link
Member

Hi @maximilianmuehlbauer

Thanks for your report.

To summarize the graph would be:

pkga_consumer -(requires)-> pkga(shared) --------(requires)--------->libb(shared) -(requires)-> libb_dep(shared)
                                    \--(build-require) ->pkgb(shared)-->/

I'd like to start with a recommendation: build=True, visible=True will be really discouraged in most cases. I have already had some experience trying it, and I would advice against this.

If the intention is that the pkga_consumer needs some tool-requires for itself, because it needs such tool requires to build itself, then that means that pkgb (shared-library) should be a direct dependency for pkga_consumer and it shouldn't be inherited from pkga

It would be also good to understand how shared libraries can be used as "tool-requires" (build=True), as this puts such libraries in the "build" context, so the binaries might actually be incompatible (cross-compilation) or inconsistent/incorrect (building for debug will still bring Release for the build context by default).

Thanks very much for your reproducible code, I will use it to make a test, and understand what could be happening.

@memsharded memsharded self-assigned this May 23, 2024
@maximilianmuehlbauer
Copy link
Author

Thanks for the quick reply!

I'd like to start with a recommendation: build=True, visible=True will be really discouraged in most cases. I have already had some experience trying it, and I would advice against this.

Yes, Matlab is a very special case... one might have a model that depends on a Simulink library (like a C++ header-only lib), that depends on a library, ... and at some point there's an sfunction. For building the final model, all of that is needed in the build environment which is where we run Matlab to generate the C++ code.

If the intention is that the pkga_consumer needs some tool-requires for itself, because it needs such tool requires to build itself, then that means that pkgb (shared-library) should be a direct dependency for pkga_consumer and it shouldn't be inherited from pkga

Yes, essentially that is what we want; however making an explicit tool requirement from pkga_consumer to the sfunction is hard as the name, version etc. of that sfunction would need to stay in sync with a possibly long dependency tree in between. We would like to give Matlab developers the possibility to depend on a library which might come with such sfunction in some place without explicitly having to specify that sfunction.

It would be also good to understand how shared libraries can be used as "tool-requires" (build=True), as this puts such libraries in the "build" context, so the binaries might actually be incompatible (cross-compilation) or inconsistent/incorrect (building for debug will still bring Release for the build context by default).

Yes, those libraries will be incompatible (and cross-compiling actually works with pkga as root). The sfunction in pkgb might need libpkgb to generate code (but it's a normal requires of pkgb -> to my understanding, it should land in build context as soon as pkg is put in build context in that case) while the generated code links against libpkgb for possibly a different architecture.

@memsharded
Copy link
Member

I have been having a look and so far it seems the build=True trait has higher priority than the visible=True one, so build-requires are still not being fully propagated. I will keep investigating tomorrow if it is possible to do this propagation without risks of breaking.

@memsharded memsharded added this to the 2.5.0 milestone Jun 2, 2024
@memsharded
Copy link
Member

This is quite challenging, and needs to be considered (small) step by step, defining how build requirements need to propagate when made visible requires some deeper research and careful consider what could be breaking. I am assigning it for 2.5 (2.4 will be soon), to start working on it, but it might take longer time and require a few iterations and releases to move forward.

@maximilianmuehlbauer
Copy link
Author

This is quite challenging, and needs to be considered (small) step by step, defining how build requirements need to propagate when made visible requires some deeper research and careful consider what could be breaking. I am assigning it for 2.5 (2.4 will be soon), to start working on it, but it might take longer time and require a few iterations and releases to move forward.

Thanks for the feedback! We're happy to test with our use case as soon as that's possible. From having a quick look, could it be that this issues comes from the following lines where visible=False is being set:

conan/conans/model/requires.py

Lines 280 to 281 in 36148df

downstream_require = Requirement(require.ref, headers=False, libs=False, build=True,
run=True, visible=False, direct=False)

maximilianmuehlbauer added a commit to maximilianmuehlbauer/conan that referenced this issue Jun 25, 2024
@maximilianmuehlbauer
fix(requires): don't stop propagating requires in build context conan…
f7c37b2 
…-io#16333
@maximilianmuehlbauer maximilianmuehlbauer linked a pull request Jun 25, 2024 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@memsharded memsharded

Labels
stage: triaging
Projects
None yet
Milestone
2.5.0
Development

Successfully merging a pull request may close this issue.

fix(requires): don't stop propagating requires in build context #16333 maximilianmuehlbauer/conan
2 participants
@maximilianmuehlbauer @memsharded