Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling #58612

jkoritzinsky · 2024-10-24T20:08:53Z

Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling (for Linux installers)

Replace the custom infrastructure in ASP.NET Core for building shared frameworks, .debs, and .rpms with the Arcade infrastructure

Description

Replace the custom infrastructure in this repo with the Arcade infrastructure. This provides a number of benefits:

Crossgen2/R2R support is hooked up correctly through the SDK support
Construction of targeting and runtime packs uses standard tooling, so less for aspnetcore to maintain and free updates to "best practices" from Arcade
Support for building deb and rpm packages on any Linux distro

Fixes #48013
Fixes #49486
Fixes #58073
Contributes to dotnet/source-build#3986

Depends on dotnet/arcade#15200
Depends on an SDK with dotnet/sdk#44436 (#58862)
~~Depends on dotnet/runtime#109169~~
Replaces #58600
Depends on dotnet/arcade#15208
Depends on dotnet/arcade#15209
Depends on #58934

…the same contents except for some simplification of the analyzer layout and the PlatformManifest (which will be implemented correctly as part converting the runtime pack)

… to look into that.

…ojects directly restores the sfxprojs.

…suming projects to pick up.

…ade PR.

…ew sfxproj and archive-based projects

…uild MSIs with this tooling (yet)

…the build better

…ensures we don't try to build AspNetCore.sln

…g packages output.

wtgodbe · 2024-10-28T16:56:27Z

Whoa, awesome! Feel free to ping me on Teams if/when you want me to take a look

… the native project references "just work" without a separate target.

…nto sfx-aspnetcore

…aspnetcore

This reverts commit b1b144d.

…aspnetcore

ViktorHofer · 2024-11-11T15:32:53Z

@jkoritzinsky do you plan to touch this code path as well?

aspnetcore/src/Installers/Windows/Wix.targets

Lines 33 to 43 in 5278d28

    
           <ItemGroup> 
        
             <PackageReference Include="Microsoft.DotNet.Build.Tasks.Installers" Version="$(MicrosoftDotNetBuildTasksInstallersVersion)" /> 
        
           </ItemGroup> 
        
           <PropertyGroup> 
        
             <MicrosoftDotNetBuildTasksInstallersTaskAssembly Condition="'$(MSBuildRuntimeType)' == 'Core'">$(NuGetPackageRoot)microsoft.dotnet.build.tasks.installers\$(MicrosoftDotNetBuildTasksInstallersVersion)\tools\netcoreapp2.1\Microsoft.DotNet.Build.Tasks.Installers.dll</MicrosoftDotNetBuildTasksInstallersTaskAssembly> 
        
             <MicrosoftDotNetBuildTasksInstallersTaskAssembly Condition="'$(MSBuildRuntimeType)' != 'Core'">$(NuGetPackageRoot)microsoft.dotnet.build.tasks.installers\$(MicrosoftDotNetBuildTasksInstallersVersion)\tools\net472\Microsoft.DotNet.Build.Tasks.Installers.dll</MicrosoftDotNetBuildTasksInstallersTaskAssembly> 
        
           </PropertyGroup> 
        
           <UsingTask TaskName="CreateLightCommandPackageDrop" AssemblyFile="$(MicrosoftDotNetBuildTasksInstallersTaskAssembly)" /> 
        
           <UsingTask TaskName="CreateLitCommandPackageDrop" AssemblyFile="$(MicrosoftDotNetBuildTasksInstallersTaskAssembly)" />

The hardcodes there are problematic and shouldn't exist at all. But before someone fixing this (by just using proper PackageReference) I wanted to double check if you intend to change this.

jkoritzinsky · 2024-11-11T17:06:37Z

I was going to handle messing with any Wix installers in a different PR (as I'm less confident in getting the upgrade codes and things right).

wtgodbe · 2024-11-12T19:40:33Z

eng/Publishing.props

@@ -31,13 +31,15 @@
    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.version" UploadPathSegment="Runtime" Condition="'$(PublishInstallerBaseVersion)' == 'true'" />

    <!-- The following installers create checksums -->
-    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.deb" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
+    <_InstallersToPublish Include="$(ArtifactsDir)packages\**\*.deb" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />


Is it the common pattern to publish installers to the packages dir?

Yes, dotnet/runtime and dotnet/windowsdesktop publish all installers to the packages dir.

wtgodbe · 2024-11-12T19:43:22Z

global.json

-    "Microsoft.DotNet.Helix.Sdk": "10.0.0-beta.24560.1"
+    "Microsoft.DotNet.Helix.Sdk": "10.0.0-beta.24560.1",
+    "Microsoft.DotNet.SharedFramework.Sdk" : "10.0.0-beta.24560.1",
+    "Microsoft.Build.NoTargets": "3.7.0",


Why these versions/when should we update them?

These are the versions used by dotnet/runtime today. I believe they're the newest versions. We generally try to update them when new versions come out, but we don't have a regular cadence.

src/Framework/App.Ref/src/Microsoft.AspNetCore.App.Ref.sfxproj

wtgodbe · 2024-11-12T19:53:28Z

src/Framework/App.Runtime/src/aspnetcore-runtime-composite.proj

@@ -0,0 +1,66 @@
+<Project Sdk="Microsoft.Build.Traversal">
+  <PropertyGroup>


What's the purpose of this proj?

This project builds an archive containing the .NET runtime and the ASP.NET Core runtime with the Composite ReadyToRun image (which is used in some of our docker images).

wtgodbe · 2024-11-12T19:56:19Z

Can you run an internal build of this branch so we can inspect the produced SharedFx/Targeting packs packages/installers?

jkoritzinsky · 2024-11-12T19:59:55Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2580559&view=results

ViktorHofer · 2024-11-13T20:00:34Z

According to the official build, the targeting pack project is built and published from multiple legs:

##[error].packages\microsoft.dotnet.arcade.sdk\10.0.0-beta.24560.1\tools\SdkTasks\PublishBuildAssets.proj(42,5): error : Repeated Package entry: 'Microsoft.AspNetCore.App.Ref' - '10.0.0-alpha.2.24562.3' 
D:\a\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\10.0.0-beta.24560.1\tools\SdkTasks\PublishBuildAssets.proj(42,5): error : InvalidOperationException: Duplicate package entries are not allowed for publishing to BAR, as this can cause race conditions and unexpected behavior
D:\a\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\10.0.0-beta.24560.1\tools\SdkTasks\PublishBuildAssets.proj(42,5): error :    at Microsoft.DotNet.Maestro.Tasks.PushMetadataToBuildAssetRegistry.MergeManifests(List`1 parsedManifests) in /_/src/Maestro/Microsoft.DotNet.Maestro.Tasks/src/PushMetadataToBuildAssetRegistry.cs:line 557
D:\a\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\10.0.0-beta.24560.1\tools\SdkTasks\PublishBuildAssets.proj(42,5): error :    at Microsoft.DotNet.Maestro.Tasks.PushMetadataToBuildAssetRegistry.PushMetadataAsync(CancellationToken cancellationToken) in /_/src/Maestro/Microsoft.DotNet.Maestro.Tasks/src/PushMetadataToBuildAssetRegistry.cs:line 105
D:\a\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\10.0.0-beta.24560.1\tools\SdkTasks\PublishBuildAssets.proj(42,5): error :

…ramework SDK.

jkoritzinsky · 2024-11-14T06:13:39Z

Failures in the Linux x64 leg will be fixed by #58934

dotnet-policy-service · 2024-11-21T15:06:53Z

Looks like this PR hasn't been active for some time and the codebase could have been changed in the meantime.
To make sure no conflicting changes have occurred, please rerun validation before merging. You can do this by leaving an /azp run comment here (requires commit rights), or by simply closing and reopening.

jkoritzinsky added 15 commits October 22, 2024 14:49

Add an sfxproj for the ref pack for Microsoft.AspNetCore.App. IT has …

ff532f2

…the same contents except for some simplification of the analyzer layout and the PlatformManifest (which will be implemented correctly as part converting the runtime pack)

Add sfxproj-based runtime framework project

c26b20d

Produce the targeting pack archives

1ff7845

Build the runtime archive. There's some extra rebuilding now, so need…

789e9bf

… to look into that.

Add a (mostly correct, some TODOs) composite runtime image and archive.

edd7c4e

Use traversal projects to ensure that building the runtime archive pr…

ef9422e

…ojects directly restores the sfxprojs.

Add the new .deb packaging to the targeting pack

4b6dd92

Add the aspnetcore runtime deb package to the sfxproj

f6de9ec

Enable building RPM packages

ad492ac

Publish from the new-style projects to the correct subfolders for con…

cf0e833

…suming projects to pick up.

Remove a TODO that's being addressed

8590cb7

Add support for partial R2R rooting based on dotnet/sdk#44436

f5ab929

Reuse global properties and remove TODO addresed in corresponding Arc…

28a3d39

…ade PR.

Replace old shared framework projects and Linux installers with the n…

8df753c

…ew sfxproj and archive-based projects

Remove the NoTargets SDK, ended up using the Traversal SDK instead.

ec21b15

jkoritzinsky added the blocked The work on this issue is blocked due to some dependency label Oct 24, 2024

dotnet-issue-labeler bot added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Oct 24, 2024

jkoritzinsky added 8 commits October 24, 2024 20:20

Fix path to runtime pack for manifest

214ede2

Don't reference the installers package on Windows. We don't want to b…

d46c78a

…uild MSIs with this tooling (yet)

Fix reference from aspnetcore-runtime and sequence the projects into …

940cba6

…the build better

Profive projects to build when only -buildinstallers is passed. This …

46431f2

…ensures we don't try to build AspNetCore.sln

Fix the path of the runtime list for testing.

d19ea9a

Set property with file name

332b97b

Disable APICompat for the archive projects.

3e7e74b

Fix expected output directory for deb/rpm/targz/zip to be the shippin…

ffc4ecf

…g packages output.

jkoritzinsky added 4 commits October 28, 2024 18:19

Fix including zips (as wixpacks are still in the intsallers folder)

b176f09

Build the shared frameworks via the transitive SDK proj only and make…

5ed00aa

… the native project references "just work" without a separate target.

Merge branch 'sfx-aspnetcore' of github.com:jkoritzinsky/aspnetcore i…

58ce10e

…nto sfx-aspnetcore

Merge branch 'main' of https://github.com/dotnet/aspnetcore into sfx-…

faa4b35

…aspnetcore

Use Arcade RPM tooling for the Runtime sfxproj

0b25d0e

jkoritzinsky added the blocked The work on this issue is blocked due to some dependency label Nov 5, 2024

Revert "Update .NET SDK to get composite image fix"

1f1faf6

This reverts commit b1b144d.

jkoritzinsky force-pushed the sfx-aspnetcore branch from 33d9c16 to 1f1faf6 Compare November 5, 2024 22:44

Merge branch 'main' of https://github.com/dotnet/aspnetcore into sfx-…

ba90cf1

…aspnetcore

jkoritzinsky marked this pull request as ready for review November 8, 2024 15:50

jkoritzinsky requested review from wtgodbe, a team, JamesNK, captainsafia and mgravell as code owners November 8, 2024 15:50

Merge branch 'main' of github.com:dotnet/aspnetcore into sfx-aspnetcore

482c5e4

Merge branch 'main' of github.com:dotnet/aspnetcore into sfx-aspnetcore

b03e202

wtgodbe reviewed Nov 12, 2024

View reviewed changes

src/Framework/App.Ref/src/Microsoft.AspNetCore.App.Ref.sfxproj Outdated Show resolved Hide resolved

wtgodbe reviewed Nov 12, 2024

View reviewed changes

jkoritzinsky and others added 4 commits November 13, 2024 12:07

Don't pack the ref pack when only packing rid-specific packages

ed7c875

Remove RpmOwnedDirectory items as they're now provided by the SharedF…

f1a7846

…ramework SDK.

Don't build installers in source-build

cd7c2c6

Merge branch 'main' into sfx-aspnetcore

0d1edd2

dotnet-policy-service bot added the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling #58612

Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling #58612

jkoritzinsky commented Oct 24, 2024 •

edited

Loading

wtgodbe commented Oct 28, 2024

ViktorHofer commented Nov 11, 2024

jkoritzinsky commented Nov 11, 2024

wtgodbe Nov 12, 2024

jkoritzinsky Nov 12, 2024

wtgodbe Nov 12, 2024

jkoritzinsky Nov 12, 2024

wtgodbe Nov 12, 2024

jkoritzinsky Nov 12, 2024

wtgodbe commented Nov 12, 2024

jkoritzinsky commented Nov 12, 2024

ViktorHofer commented Nov 13, 2024

jkoritzinsky commented Nov 14, 2024

dotnet-policy-service bot commented Nov 21, 2024

		@@ -0,0 +1,66 @@
		<Project Sdk="Microsoft.Build.Traversal">
		<PropertyGroup>

Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling #58612

Are you sure you want to change the base?

Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling #58612

Conversation

jkoritzinsky commented Oct 24, 2024 • edited Loading

Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling (for Linux installers)

Description

wtgodbe commented Oct 28, 2024

ViktorHofer commented Nov 11, 2024

jkoritzinsky commented Nov 11, 2024

wtgodbe Nov 12, 2024

Choose a reason for hiding this comment

jkoritzinsky Nov 12, 2024

Choose a reason for hiding this comment

wtgodbe Nov 12, 2024

Choose a reason for hiding this comment

jkoritzinsky Nov 12, 2024

Choose a reason for hiding this comment

wtgodbe Nov 12, 2024

Choose a reason for hiding this comment

jkoritzinsky Nov 12, 2024

Choose a reason for hiding this comment

wtgodbe commented Nov 12, 2024

jkoritzinsky commented Nov 12, 2024

ViktorHofer commented Nov 13, 2024

jkoritzinsky commented Nov 14, 2024

dotnet-policy-service bot commented Nov 21, 2024

jkoritzinsky commented Oct 24, 2024 •

edited

Loading