Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Verticals are not publishing all packages #4984

Open
NikolaMilosavljevic opened this issue Mar 27, 2025 · 16 comments
Open

Verticals are not publishing all packages #4984

NikolaMilosavljevic opened this issue Mar 27, 2025 · 16 comments
Labels
area-unified-build

Comments

@NikolaMilosavljevic
Copy link
Member

I've only checked AzureLinux_x64_Cross_x64 - it publishes packages from few repos today. It used to publish many more.

The issue seems to have started with this VMR SHA: c9986621b8c673b2db87d00d57e9bf69ed1d830f

Build with previous SHA was good.

Diff: https://github.com/dotnet/dotnet/compare/7fe01d9..c998662

This seems related to changes in vertical publishing code in dotnet/sdk#47338

@jkoritzinsky
@jkoritzinsky
Copy link
Member

@NikolaMilosavljevic now each Unified Build vertical only publishes the packages that are unique for that vertical. For verticals other than Windows_x64, this usually means that there are very few packages published. Are there any packages you see that are missing after the final join job?

@NikolaMilosavljevic
Copy link
Member Author

@NikolaMilosavljevic now each Unified Build vertical only publishes the packages that are unique for that vertical. For verticals other than Windows_x64, this usually means that there are very few packages published. Are there any packages you see that are missing after the final join job?

OK. Yes, there are some missing packages, here's the full list (versions from older build):

Microsoft.Diagnostics.NETCore.Client.0.2.616707.nupkg
Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-preview.25166.107.nupkg
Microsoft.NET.Sdk.Aspire.Manifest-8.0.100.8.2.2.nupkg
NuGet.Build.Tasks.6.14.0-preview.1.32767.nupkg
NuGet.Build.Tasks.Console.6.14.0-preview.1.32767.nupkg
NuGet.Build.Tasks.Pack.6.14.0-preview.1.32767.nupkg
NuGet.CommandLine.XPlat.6.14.0-preview.1.32767.nupkg
NuGet.Commands.6.14.0-preview.1.32767.nupkg
NuGet.Common.6.14.0-preview.1.32767.nupkg
NuGet.Configuration.6.14.0-preview.1.32767.nupkg
NuGet.Credentials.6.14.0-preview.1.32767.nupkg
NuGet.DependencyResolver.Core.6.14.0-preview.1.32767.nupkg
NuGet.Frameworks.6.14.0-preview.1.32767.nupkg
NuGet.LibraryModel.6.14.0-preview.1.32767.nupkg
NuGet.PackageManagement.6.14.0-preview.1.32767.nupkg
NuGet.Packaging.6.14.0-preview.1.32767.nupkg
NuGet.ProjectModel.6.14.0-preview.1.32767.nupkg
NuGet.Protocol.6.14.0-preview.1.32767.nupkg
NuGet.Resolver.6.14.0-preview.1.32767.nupkg
NuGet.Versioning.6.14.0-preview.1.32767.nupkg
Razor.Diagnostics.Analyzers.Test.9.0.0-preview.25166.107.nupkg
RazorDeployment.9.0.0-preview.25166.107.nupkg
VS.Redist.Common.NetCore.AppHostPack.arm64_x86.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.AppHostPack.x64_arm64.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.AppHostPack.x64_x86.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.AppHostPack.x86_arm64.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.SdkPlaceholder.arm64.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.SdkPlaceholder.x86.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Templates.arm64.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Templates.x86.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Toolset.arm64.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Toolset.x86.10.0.10.0.100-preview.4.25176.115.nupkg

This list was discovered by diffing the package list from a build right before your PR was merged. Some of these might make sense, but some packages are certainly produced by verticals, but missing, like Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-preview.25169.101.nupkg. This is produced by Windows vertical, published to vertical artifacts, but not to PackageArtifacts, as seen in latest full build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2672789&view=artifacts&pathAsName=false&type=publishedArtifacts

Perhaps the issue is that this particular package now has ci versioning: Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-ci.nupkg

@jkoritzinsky
Copy link
Member

The NuGet packages are being tracked by #4974

The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.

I see the Razor packages in PackageArtifacts.

The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

@jkoritzinsky jkoritzinsky mentioned this issue Mar 27, 2025
Merged
1 task
@NikolaMilosavljevic
Copy link
Member Author

The NuGet packages are being tracked by #4974

The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.

I see the Razor packages in PackageArtifacts.

The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

Scenario-tests package is important and its absence is currently blocking Installer tests. They are failing even though the pipeline shows warnings for those jobs.

@NikolaMilosavljevic
Copy link
Member Author

The NuGet packages are being tracked by #4974
The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.
I see the Razor packages in PackageArtifacts.
The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

Scenario-tests package is important and its absence is currently blocking Installer tests. They are failing even though the pipeline shows warnings for those jobs.

Interestingly, scenario-tests package is missing from Windows_x64 manifest: https://artprodcus3.artifacts.visualstudio.com/Ab55de4ed-4b5a-4215-a8e4-0a0a5f71e7d8/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2RuY2VuZy9wcm9qZWN0SWQvN2VhOTExNmUtOWZhYy00MDNkLWIyNTgtYjMxZmNmMWJiMjkzL2J1aWxkSWQvMjY3Mjc4OS9hcnRpZmFjdE5hbWUvV2luZG93c194NjRfQXJ0aWZhY3Rz0/content?format=file&subPath=%2Fmanifests%2FRelease%2FWindows_x64.xml

@NikolaMilosavljevic
Copy link
Member Author

The NuGet packages are being tracked by #4974
The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.
I see the Razor packages in PackageArtifacts.
The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

Scenario-tests package is important and its absence is currently blocking Installer tests. They are failing even though the pipeline shows warnings for those jobs.

Interestingly, scenario-tests package is missing from Windows_x64 manifest: https://artprodcus3.artifacts.visualstudio.com/Ab55de4ed-4b5a-4215-a8e4-0a0a5f71e7d8/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2RuY2VuZy9wcm9qZWN0SWQvN2VhOTExNmUtOWZhYy00MDNkLWIyNTgtYjMxZmNmMWJiMjkzL2J1aWxkSWQvMjY3Mjc4OS9hcnRpZmFjdE5hbWUvV2luZG93c194NjRfQXJ0aWZhY3Rz0/content?format=file&subPath=%2Fmanifests%2FRelease%2FWindows_x64.xml

Previous good build that included ScenarioTests package - Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-preview.25167.101.nupkg:

https://dev.azure.com/dnceng/internal/_build/results?buildId=2665355&view=artifacts&pathAsName=false&type=publishedArtifacts

Vertical manifests do not include nuget packages, unless they're under assets, which is only true for various *.symbols.nupkg.

@mthalman
Copy link
Member

Scenario tests isn't a shipping thing. Why is it important for it to have its package published?

@NikolaMilosavljevic
Copy link
Member Author

NikolaMilosavljevic commented Mar 28, 2025 •
edited
Loading

Scenario tests isn't a shipping thing. Why is it important for it to have its package published?

This is about publishing packages to VMR's PackageArtifacts. We do publish all Shipping and NonShipping packages there, with some exceptions - all of them seem to be regressions due to some recent changes in publishing and join-verticals steps.

Scenario-tests are consumed by installer-tests jobs, in Validation stage. We can't build this repo at that time, as it would build all dependencies, and incur the cost of building the whole VMR again.

@NikolaMilosavljevic
Copy link
Member Author

@mmitche

@jkoritzinsky
Copy link
Member

I changed the publishing to PackageArtifacts and BlobArtifacts to only publish the files we actually want to publish externally. Is scenario-tests staging the asset after we've already made the merged manifest?

@NikolaMilosavljevic
Copy link
Member Author

I changed the publishing to PackageArtifacts and BlobArtifacts to only publish the files we actually want to publish externally. Is scenario-tests staging the asset after we've already made the merged manifest?

Scenario-tests repo gets built during Run Tests step.

@jkoritzinsky
Copy link
Member

That would explain why it isn't available, as it isn't in the asset manifest for the vertical. I'll try to think of some solutions here that don't require us to become less strict on our publishing/join steps.

@NikolaMilosavljevic
Copy link
Member Author

NikolaMilosavljevic commented Mar 28, 2025 •
edited
Loading

That would explain why it isn't available, as it isn't in the asset manifest for the vertical. I'll try to think of some solutions here that don't require us to become less strict on our publishing/join steps.

Would it make sense to update the asset manifest right after each repo is built? That would solve the issue with repos that aren't built in main build, i.e. during test step.

@jkoritzinsky
Copy link
Member

We produce the asset manifest at the end of the build and we currently don't have tooling to update a manifest.

Is there a particular reason you need to pull the assets from the final join? For installer testing especially, using the "reuse build artifacts from" mechanism we have might work better. Or adding a new "build pass" that builds scenario-tests fully without rebuilding the rest of the assets so we can consume them in the installer validation jobs.

@jkoritzinsky
Copy link
Member

Alternatively, could we build scenario-tests in the build step and only run the tests in the Test step?

@NikolaMilosavljevic
Copy link
Member Author

NikolaMilosavljevic commented Mar 28, 2025 •
edited
Loading

I'll explore the suggested options. Building scenario-tests repo independently from running the tests might be the best option.

Reusing build artifacts from another job doesn't work across stages, unfortunately. To clarify - we can directly download artifacts from any vertical. What we cannot do it express dependency on a vertical, which is a job in another stage.

While this will work with existing pipeline implementation (installer job), it will block planned refactoring of build stages, to reduce the number of build entry points. We would end up with installer job reusing build steps that are currently in job vmr-build.yml. It might still work with some modifications, to, i.e. allow a installer job to consume artifacts from vertical build stage, but skip adding the dependsOn property as dependencies across stages are not possible - https://github.com/dotnet/sdk/blob/9a24d5459290b488dcf92ba0194d56b5b308c9b3/eng/pipelines/templates/jobs/vmr-build.yml#L152

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-unified-build
Projects
.NET Unified Build
Status: No status
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jkoritzinsky @MichaelSimons @NikolaMilosavljevic @mthalman