slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1218130	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51978
1217971	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#114983
1217937	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122112
1217916	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121679
1217922	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121643
1217869	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122036
1217892	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122120
1218034	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122128
1217732	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122116
1217704	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122114
1217669	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122111
1217677	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1217747	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1217627	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121614
1217597	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121412
1217729	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1217624	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121906
1217548	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122107
1217502	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121911
1217486	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122105
2852162	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1217505	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121072
1217439	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122081
1217451	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122072
1217415	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122036
1217377	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121800
1217349	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122100
1217352	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119575
1217321	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122068
1217271	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119634
1217285	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#115335
1217380	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121800
1217351	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122100
1217251	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122093
2851987	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1217220	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122091
1217205	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121939
1217160	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122090
1217113	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122088
1217097	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121739
1217125	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122072
1217058	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121911
1216977	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121072
1216968	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1216946	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121837
1216955	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121906
1216938	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#120688
1216902	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121072
1216897	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121973
1216848	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121614
1216793	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122088
1216606	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122083
1216753	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121862
1216704	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122085
1216680	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#120823
1216668	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#117788
1216342	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122070
1216675	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121072
1216651	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122075
1216618	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#119432
1216591	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122081
1216575	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122078
1216550	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121637
1216521	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122076
2851417	dotnet-sdk	Darwin	Log	#55692
2851401	dotnet-sdk	Darwin	Log	#55691
1216478	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1216523	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122076
1216515	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1216392	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122072
1216417	dotnet/runtime	osx-x64 release Runtime_Release	Log
1216350	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119575
1216336	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121800
1216303	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122036
1216314	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122073
1216285	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122010
1212695	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121251
1216209	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122065
1216199	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#120634
1216175	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#121412
1216237	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#121956
1216168	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122069
1216157	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122068
1216049	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121072
1216040	dotnet/runtime	osx-arm64 Release Installer_Build_And_Test	Log
1216061	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1216020	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122056
1215929	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#120893
1215856	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122023
1215866	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#119432
1215847	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122065
1215793	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#121906
1215822	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122056
1215808	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122064
1215785	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1215787	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1215765	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122023
1215705	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#122059
1215753	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122056
2850545	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
Displaying 100 of 626 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
49	132	626

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.