Skip to content

Make official build assetless and clean-up #9555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ViktorHofer wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Make official build assetless and clean-up #9555

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e739145
Make official build assetless and clean-up
ViktorHofer Nov 17, 2025
5226cc1
Add unofficial template
ViktorHofer Nov 19, 2025
6f2ac6a
Merge branch 'main' into MakeAssetlessBuildAndCleanup
ViktorHofer Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 1 addition & 28 deletions azure-pipelines-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,55 +2,28 @@ trigger:
batch: true
branches:
include:
- stabilize
- main
- release/*
- internal/release/*
- feature/*
- legacy/*
pr:
branches:
include:
- stabilize
- main
- release/*
- feature/*
- legacy/*

variables:
# Cannot use key:value syntax in root defined variables
- name: _TeamName
value: NETDevUX
- name: _PublishUsingPipelines
value: true
- name: Codeql.Enabled
value: true
- name: EnableReleaseOneLocBuild
value: false
- template: /eng/common/templates/variables/pool-providers.yml

stages:
- stage: build
displayName: Build
jobs:
- ${{ if and( ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
# The localization setup for main branch. Note difference in package ID. Should not be used with release/ branches.
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- template: /eng/common/templates/job/onelocbuild.yml
parameters:
MirrorRepo: templating
LclSource: lclFilesfromPackage
LclPackageId: 'LCL-JUNO-PROD-TMPLTNGMAIN'
MirrorBranch: 'main'
JobNameSuffix: '_main'
condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
- template: /eng/common/templates/jobs/jobs.yml
parameters:
enableMicrobuild: true
enablePublishBuildArtifacts: true
enablePublishTestResults: true
enablePublishBuildAssets: true
enablePublishUsingPipelines: ${{ variables._PublishUsingPipelines }}
enablePublishUsingPipelines: false
enableSourceBuild: true
sourceBuildParameters:
enableInternalSources: true
Expand Down
180 changes: 25 additions & 155 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,181 +1,51 @@
variables:
- name: _PublishUsingPipelines
value: false
- template: /eng/common/templates-official/variables/pool-providers.yml@self

trigger:
batch: true
branches:
include:
- stabilize
- main
- release/*
- internal/release/*
- feature/*
- legacy/*
pr:
branches:
include:
- stabilize
- main
- release/*
- feature/*
- legacy/*
- main
- release/*

variables:
# Cannot use key:value syntax in root defined variables
- name: _TeamName
value: NETDevUX
- name: _PublishUsingPipelines
value: true
- name: Codeql.Enabled
value: true
- name: EnableReleaseOneLocBuild
value: false
- template: /eng/common/templates-official/variables/pool-providers.yml
pr: none

resources:
repositories:
- repository: 1esPipelines
- repository: 1ESPipelineTemplates
type: git
name: 1ESPipelineTemplates/1ESPipelineTemplates
ref: refs/tags/release

# This is necessary because 1ES doesn't work using strategy & matrix
parameters:
- name: buildConfigurations
type: object
default:
# Always build the Release configuration, but never sign for PRs.
- buildConfig: Release

extends:
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
template: v1/1ES.Official.PipelineTemplate.yml@1esPipelines
${{ else }}:
template: v1/1ES.Unofficial.PipelineTemplate.yml@1esPipelines
template: v1/1ES.Official.PipelineTemplate.yml@1ESPipelineTemplates
Copy link
Member

@marcpopMSFT marcpopMSFT Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This yml is used in the templating unofficial pipeline so most of the changes to this file can't be taken as is. If we want to split into a third yml, I guess we can do that but we can't use the official pipelines in the unofficial yml and we need to have legs that actually do builds.

To be clear here, -pr is used for public PR builds on dnceng-public. The other yml is used for CI and for internal PRs. There was more different for internal versus public that we used the CI yml for internal PRs rather than using the PR yml for internal PRs.

Copy link
Member Author

@ViktorHofer ViktorHofer Nov 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to split into a third yml, I guess we can do that but we can't use the official pipelines in the unofficial yml and we need to have legs that actually do builds.

Yes, repositories which have an unofficial internal pipeline need a third entry-point YML file. Pushed a commit. We will need to update the unofficial pipeline post merge to point to the new YML.

Copy link
Member

@MiYanni MiYanni Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, repositories which have an unofficial internal pipeline need a third entry-point YML file. Pushed a commit. We will need to update the unofficial pipeline post merge to point to the new YML.

That is not true. I use the same pipeline entry point for the Installer build currently that runs both the official build AND the unofficial build. dotnet/installer#20657

It is a myth that you need 2 entry points. You just need to know what should only exist within the official build and then conditionally remove it from the unofficial build. For triggers, you can edit the triggers for the unofficial pipeline within the pipeline itself, overriding what is declared in the YAML.

Copy link
Member Author

@ViktorHofer ViktorHofer Nov 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By reusing that YML for both the official build (which has a vastly different layout) and the unofficial build, you will have tons of ifdefs and as you mentioned you would need to override triggers in the UI (for which you need special permissions). Those are all reasons why you want to have a separate file. At the end of the day, I don't have a strong opinion here and you own the repository so you decide. I'm just sharing best practices from my point of view.

@MiYanni maybe you can help us understand how a merged layout would look like in comparison to what I currently have in this PR.

Copy link
Member

@MiYanni MiYanni Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't as many ifdefs as you might think if you design the pipeline for it. That's what I did for the SDK repo and intentionally thought about this situation at the onset. For templating, to be blunt, I don't care about this repo or its pipelines. At this point, my opinion is that we should merge this repo into the SDK repo and call it a day. But we don't have time/resources to do that currently. I'm fine with a separate pipeline here. But saying "it is required to make a new entry point" just isn't a true statement, is the main point I was trying to make. 😝 I only needed to change 4 lines in the Installer YAML to make it work.

Copy link
Member

@marcpopMSFT marcpopMSFT Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it has cost us less to use one yml than it would to maintain two up to this point. I don't know if making the build assetless increases enough to justify an extra yml to maintain.

parameters:
sdl:
createAdoIssuesForJustificationsForDisablement: false
sourceAnalysisPool:
name: $(DncEngInternalBuildPool)
image: 1es-windows-2022
os: windows
policheck:
enabled: true
tsa:
enabled: true
stages:
- stage: build
displayName: Build
jobs:
- ${{ if and( ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
# The localization setup for main branch. Note difference in package ID. Should not be used with release/ branches.
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- template: /eng/common/templates-official/job/onelocbuild.yml@self
parameters:
MirrorRepo: templating
LclSource: lclFilesfromPackage
LclPackageId: 'LCL-JUNO-PROD-TMPLTNGMAIN'
MirrorBranch: 'main'
JobNameSuffix: '_main'
condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
# The localization setup for main branch. Note difference in package ID. Should not be used with release/ branches.
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- template: /eng/common/templates-official/job/onelocbuild.yml@self
parameters:
MirrorRepo: templating
LclSource: lclFilesfromPackage
LclPackageId: 'LCL-JUNO-PROD-TMPLTNGMAIN'
MirrorBranch: 'main'
JobNameSuffix: '_main'
condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
- template: /eng/common/templates-official/jobs/jobs.yml@self
Copy link
Member

@MiYanni MiYanni Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this template do now that it has no jobs defined?

Copy link
Member Author

@ViktorHofer ViktorHofer Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an example of a build pipeline with an assetless build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2841936&view=results

There's one job "Publish Assets" that inserts the build into BAR which results in the source code flow into the VMR. This intentionally doesn't build the repository anymore. In the future, later during .NET 11, we won't need a pipeline for this insertion anymore and Maestro will poll commits instead.

Copy link
Member

@MiYanni MiYanni Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm misunderstanding, but I thought "Publish Assets" publishes the assets from the build. Meaning, you need to build the repo to publish the assets from it. 🤔

Copy link
Member

@akoeplinger akoeplinger Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's true normally but in this case the only thing that gets "published" is the link between the build/commit and the BAR ID so you don't need to build the repo.

Copy link
Member

@MiYanni MiYanni Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. So, is it only for codeflow purposes since there won't be any build assets associated with the DARC entry?

Copy link
Member

@akoeplinger akoeplinger Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct, and this dummy build will go away in the future as Viktor mentioned

parameters:
enableMicrobuild: true
enablePublishBuildArtifacts: true
enablePublishTestResults: true
enablePublishBuildAssets: true
enablePublishUsingPipelines: ${{ variables._PublishUsingPipelines }}
enableSourceBuild: true
enablePublishBuildAssets: false
enablePublishTestResults: false
publishAssetsImmediately: true
sourceBuildParameters:
enableInternalSources: true
enableTelemetry: true
helixRepo: dotnet/templating
templateContext:
sdl:
binskim:
analyzeTargetGlob: +:f|artifacts\bin\**\*.dll;+:f|artifacts\bin\**\*.exe;-:f|artifacts\bin\**\xunit*.dll;-:f|artifacts\bin\**\verify*.dll;
# WORKAROUND: BinSkim requires the folder exist prior to scanning.
preSteps:
- powershell: New-Item -ItemType Directory -Path $(Build.SourcesDirectory)/artifacts/bin -Force
displayName: Create artifacts/bin directory
jobs:
- ${{ each config in parameters.buildConfigurations }}:
- job: Windows_NT_${{ config.buildConfig }}
displayName: Windows_NT ${{ config.buildConfig }}
timeoutInMinutes: 90
pool:
name: $(DncEngInternalBuildPool)
image: 1es-windows-2022
os: windows
variables:
- _BuildConfig: ${{ config.buildConfig }}
- _SignType: test
- ${{ if notin(variables['Build.Reason'], 'PullRequest') }}:
- _SignType: real
- _InternalBuildArgs: ''
# Only enable publishing in non-public, non PR scenarios.
- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
# DotNet-Symbol-Server-Pats provides: microsoft-symbol-server-pat, symweb-symbol-server-pat
# Publish-Build-Assets provides: MaestroAccessToken, BotAccount-dotnet-maestro-bot-PAT
- group: Publish-Build-Assets
- _InternalBuildArgs: /p:DotNetSignType=$(_SignType) /p:TeamName=$(_TeamName)
/p:DotNetPublishUsingPipelines=$(_PublishUsingPipelines)
/p:OfficialBuildId=$(BUILD.BUILDNUMBER)
steps:
# Use utility script to run script command dependent on agent OS.
- template: /eng/common/templates-official/steps/enable-internal-sources.yml@self
- ${{ if in(variables['Build.Reason'], 'PullRequest') }}:
- script: eng/common/cibuild.cmd
-configuration $(_BuildConfig)
-prepareMachine
-integrationTest
$(_InternalBuildArgs)
displayName: Windows Build / Publish
- ${{ else }}:
- script: eng/common/cibuild.cmd
-configuration $(_BuildConfig)
-prepareMachine
/p:Test=false
$(_InternalBuildArgs)
displayName: Windows Build / Publish

- ${{ if in(variables['Build.Reason'], 'PullRequest') }}:
- ${{ each config in parameters.buildConfigurations }}:
- job: OSX_${{ config.buildConfig }}
displayName: OSX ${{ config.buildConfig }}
pool:
vmImage: 'macOS-latest'
variables:
- _BuildConfig: ${{ config.buildConfig }}
- _SignType: none
steps:
- template: /eng/common/templates-official/steps/enable-internal-sources.yml@self
- script: eng/common/cibuild.sh
--configuration $(_BuildConfig)
--prepareMachine
--integrationTest
name: Build
displayName: Build

- ${{ if in(variables['Build.Reason'], 'PullRequest') }}:
- ${{ each config in parameters.buildConfigurations }}:
- job: Linux_${{ config.buildConfig }}
displayName: Linux ${{ config.buildConfig }}
pool:
${{ if eq(variables['System.TeamProject'], 'public') }}:
name: $(DncEngPublicBuildPool)
image: 1es-ubuntu-2204-open
os: linux
${{ if eq(variables['System.TeamProject'], 'internal') }}:
name: $(DncEngInternalBuildPool)
image: 1es-ubuntu-2204
os: linux
variables:
- _BuildConfig: ${{ config.buildConfig }}
- _SignType: none
steps:
- template: /eng/common/templates-official/steps/enable-internal-sources.yml@self
- script: eng/common/cibuild.sh
--configuration $(_BuildConfig)
--prepareMachine
--integrationTest
name: Build
displayName: Build
condition: succeeded()
isAssetlessBuild: true
enableTelemetry: true
21 changes: 11 additions & 10 deletions eng/Versions.props
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
<Project>
<Import Project="Version.Details.props" Condition="Exists('Version.Details.props')" />

<Import Project="Version.Details.props" />

<PropertyGroup>
<VersionPrefix>11.0.100</VersionPrefix>
<!-- When StabilizePackageVersion is set to 'true', this branch will produce stable outputs for 'Shipping' packages -->
<StabilizePackageVersion Condition="'$(StabilizePackageVersion)' == ''">false</StabilizePackageVersion>
<DotNetFinalVersionKind Condition="'$(StabilizePackageVersion)' == 'true'">release</DotNetFinalVersionKind>
<!-- Calculate prerelease label -->
<PreReleaseVersionLabel Condition="'$(StabilizePackageVersion)' != 'true'">alpha</PreReleaseVersionLabel>
<PreReleaseVersionLabel Condition="'$(StabilizePackageVersion)' == 'true' and $(VersionPrefix.EndsWith('00'))">rtm</PreReleaseVersionLabel>
<PreReleaseVersionLabel Condition="'$(StabilizePackageVersion)' == 'true' and !$(VersionPrefix.EndsWith('00'))">servicing</PreReleaseVersionLabel>
<PreReleaseVersionIteration Condition="'$(StabilizePackageVersion)' != 'true'"></PreReleaseVersionIteration>

<PreReleaseVersionLabel>alpha</PreReleaseVersionLabel>
<PreReleaseVersionIteration>1</PreReleaseVersionIteration>
<!-- Allowed values: '', 'prerelease', 'release'. Set to 'release' when stabilizing. -->
<DotNetFinalVersionKind></DotNetFinalVersionKind>

<!-- Arcade feature flags-->
<UsingToolXliff>true</UsingToolXliff>
<FlagNetStandard1XDependencies>true</FlagNetStandard1XDependencies>
</PropertyGroup>

</Project>