[build]sonic-sairedis build fails sometimes while building metadata #1294
Comments
dgsudharsan
changed the title
dgsudharsan
changed the title
|
@kcudnik could you check this in 202305 branch |
|
Build fails sometines? And sometimes works ? |
|
Which branch it fails ? |
|
This is 202305. I believe it fails sometimes. |
|
Is this built on pipeline or locally ? |
|
@kcudnik This is built through pipeline. |
|
if it fails sometimes then there is some race condition or instability in the pipeline if the build fails |
|
what does it mean to build sairedis with cache enabled? what cache ? |
|
We do have a way to build with cache enabled. We specify these parameters during make |
|
Maybe not everything is cached if it sometimes passes |
|
One more thing is, it is seen sometimes without cache too. I have shared those logs. I believe this issue happens due to parallel build. May be should we need to set some ordering to avoid the issue. As you can see the linker cannot find libsaimetadata_la-saimetadata.o. However this target exists and may be not built when linker executes? |
|
Parallel build is not supported |
|
@kcudnik If that's the case can you please raise fix to ensure sonic-sairedis is explicitly overriding parallel build using build options? Currently it picks from the sonic-build system DEB_BUILD_OPTIONS='nocheck parallel=30' |
|
Parallel check was never tested, will need to investigate how to fix that, in the meantime can we disable parallel build ? |
|
sairedis_buildfail_with_debug.txt |
|
can you build this without parallel on your side ? this seems like python targets are build before sairedis/meta, currently i don't know how to set any dependencies in parallel build that some specific targets would need to be build before some other targets, i don't know make enough for this, and our pipelines are building single threaded, and everything is working fine is that a big slowdown on your side ? and how you are starting your build in sairedis ? what command/switches you are adding to make/configure ? |
|
@dgsudharsan did you find a way to solve this ? i am facing similar issue for 202305 build, can you please help? |
|
If python build is build first before SAI then metadata is not existing, Im not familiar with Makefile dependencies to add them, but almost all of the projects depend on SAI here, @dgsudharsan how you enabled parallel build ? So far we didn't have parallelism enabled and everything was fine, is it some automatic change in az pipeline? I have limited access untill the end of the year, I'm on vacation |
|
We can use explicit dependency on each target so it will always execute in specific order, like here https://stackoverflow.com/questions/9159960/order-of-processing-components-in-makefile but it defeats entire purpose of parallelism, so not sure if this make sense at all to run in multithread env, we could try to build SAI metadata first, and then check whether multi core pass, and add new restrictions, python extensions and tests should be compiled at the end |
Description
sonic-sairedis build fails sometimes with the below logs
Attaching the full log of sonic-sairedis with and without cache
sairedis_buildfail.log
sairedis_buildfail_without_cache.log
Steps to reproduce the issue:
Describe the results you received:
Building sairedis fails
Describe the results you expected:
No build failure is expected
Output of
show version:Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: