dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1175749	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51281
2817122	dotnet-sdk	Run TestBuild Tests	Log	#54384
1175915	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51227
2816988	dotnet-sdk	Run FullFramework Tests	Log	#54384
1175624	dotnet/sdk	Run TestBuild Tests	Log	#51279
2816851	dotnet-sdk	Run FullFramework Tests	Log	#54384
1175288	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50864
2816274	dotnet-sdk	Run TestBuild Tests	Log	#54384
1174788	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51239
1174699	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51256
1173991	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51239
2815592	dotnet-sdk	Run TestBuild Tests	Log	#54237
1174012	dotnet/sdk	Run TestBuild Tests	Log	#51170
1173188	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51143
1173688	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
2815236	dotnet-sdk	Run TestBuild Tests	Log	#54237
2815238	dotnet-sdk	Run TestBuild Tests	Log	#54241
1173394	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51229
1173161	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51127
1173268	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51225
1173066	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51153
1173026	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51227
1172776	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51220
1172485	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1172371	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50468
1172277	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50946
1171494	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51210
2813417	dotnet-sdk	Run TestBuild Tests	Log	#54237
2813249	dotnet-sdk	Run TestBuild Tests	Log	#54237
1170569	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51195
1170388	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51193
1169937	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1170024	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51188
1169929	dotnet/sdk	Run TestBuild Tests	Log	#51181
2812347	dotnet-sdk	Run TestBuild Tests	Log	#53989
1169580	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1169201	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51108
1168959	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1168962	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51095
1168953	dotnet/sdk	Run TestBuild Tests	Log	#51028
2811379	dotnet-sdk	Run FullFramework Tests	Log	#53989
1168628	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1168666	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51172
1168639	dotnet/sdk	Run TestBuild Tests	Log	#51028
1168565	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51073
1168130	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51073
2810571	dotnet-sdk	Run TestBuild Tests	Log	#53989
2810499	dotnet-sdk	Run TestBuild Tests	Log	#53988
2810570	dotnet-sdk	Run TestBuild Tests	Log	#54089
1167839	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50946
1167258	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51078
1167076	dotnet/sdk	Run TestBuild Tests	Log	#50973
2808071	dotnet-sdk	Run TestBuild Tests	Log	#53989
1165227	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1165061	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51073
1165019	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1164469	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51117
1163029	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51073
1161644	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51063
1161459	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1160432	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1160274	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1160020	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51035
1159860	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51039
1158916	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
1158808	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51019
1158448	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50992
2801933	dotnet-sdk	Run TestBuild Tests	Log	#53783
1158291	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50992
1157845	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51004
2801227	dotnet-sdk	Run FullFramework Tests	Log	#53608
1157462	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
2801229	dotnet-sdk	Run TestBuild Tests	Log	#53607
1157189	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50974
1157258	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50951
2801063	dotnet-sdk	Run TestBuild Tests	Log	#53608
1157262	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50983
2801064	dotnet-sdk	Run TestBuild Tests	Log	#53613
1156801	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50901
2800634	dotnet-sdk	Run Tests in Helix	Log	#53551
2800311	dotnet-sdk	Run TestBuild Tests	Log	#53608
1156426	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50946
2799435	dotnet-sdk	Run TestBuild Tests	Log	#53608
1154474	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50948
1154766	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50954
1153553	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50942
1154767	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50955
1152951	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50924
1154760	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50949
2798308	dotnet-sdk	Run Tests in Helix	Log	#53551
1153414	dotnet/sdk	Run TestBuild Tests	Log	#50940
2797453	dotnet-sdk	Run TestBuild Tests	Log	#53420
2797097	dotnet-sdk	Run TestBuild Tests	Log	#53503
2797003	dotnet-sdk	Run FullFramework Tests	Log	#53444
2796885	dotnet-sdk	Run TestBuild Tests	Log	#53444
1151867	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50867
1151670	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50897
2796041	dotnet-sdk	Run TestBuild Tests	Log	#53444
1151416	dotnet/sdk	Run FullFramework Tests	Log	#50898
2796042	dotnet-sdk	Run TestBuild Tests	Log	#53445
Displaying 100 of 108 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
9	39	108

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.