Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

Open
pavelsavara opened this issue Jul 24, 2023 · 11 comments
Open

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

pavelsavara opened this issue Jul 24, 2023 · 11 comments
Assignees
@radekdoulik
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Milestone
10.0.0

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Jul 24, 2023 •
edited by build-analysis bot
Loading

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": ["emcc", "SIGKILL"],
  "ExcludeConsoleLog": false
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707
Error message validated: [emcc SIGKILL]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/25/2024 7:29:32 AM UTC

Report

Build Definition Test Pull Request
1005960 dotnet/runtime WasmTestOnChrome-ST-System.Xml.Linq.SDMSample.Tests.WorkItemExecution
1006055 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113765
1006048 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114285
1005987 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114281
1005973 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114187
1005882 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1005857 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114190
1005651 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114266
1005854 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114194
1005817 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113899
1001744 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113240
1005753 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114277
1005712 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114261
1005698 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #100729
1005617 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113286
1005593 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114083
1005579 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114252
1005541 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114263
1004699 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114227
1005502 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #110818
1005506 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114190
1005296 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1005242 dotnet/runtime WasmTestOnChrome-ST-System.Xml.XmlSerializer.ReflectionOnly.Tests.WorkItemExecution
1005430 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114145
1005407 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114261
1005380 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114101
1005370 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114226
1005219 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114080
1005225 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111791
1005183 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114148
1005171 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1005153 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114249
1005140 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114228
1005118 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114252
1005089 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114080
1005083 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114256
1005040 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114137
1005037 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113111
1004939 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114252
1004611 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1004916 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114249
1004734 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114155
1004888 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113976
1003296 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113286
1004820 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114245
1004750 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114243
1004739 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114163
1004695 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114187
1004711 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114240
1004743 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
1004581 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1004551 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1004574 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114237
1004592 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113899
1004524 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114085
1004520 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114164
1004538 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1004490 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114080
1004469 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114101
1004337 dotnet/runtime normal-System.Runtime.Tests.WorkItemExecution #114223
1004434 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114231
1004416 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114155
1004386 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114226
1004397 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114228
1004391 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114227
1004351 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114225
1004326 dotnet/runtime normal-IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution
1004313 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113313
1004288 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114173
1004300 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1004282 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114215
1004249 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1004210 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114087
1004186 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114194
1004145 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114211
1004125 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114210
1003904 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #100729
1004092 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114190
1003986 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #114185
1004012 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111126
1003494 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114192
1003950 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114080
1003804 dotnet/runtime WasmTestOnChrome-ST-System.Xml.XmlSerializer.ReflectionOnly.Tests.WorkItemExecution
999726 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113876
1003817 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114200
1003811 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114194
1003797 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114145
1003791 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113689
1003757 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113313
1003735 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1003738 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114196
1003668 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114080
1002966 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114164
1003522 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114155
1003319 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114155
1003236 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113907
1003184 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114171
1003202 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1003120 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114080
1003022 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114087
Displaying 100 of 751 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
43 227 751
@pavelsavara pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Codegen-AOT-mono wasm-aot-test WebAssembly AOT Test Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jul 24, 2023
@pavelsavara pavelsavara added this to the 8.0.0 milestone Jul 24, 2023
@ghost
Copy link

ghost commented Jul 24, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": "emcc.*SIGKILL",
  "ExcludeConsoleLog": true
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Author: pavelsavara
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, area-Codegen-AOT-mono, wasm-aot-test, Known Build Error, os-browser
Milestone: 8.0.0

@pavelsavara pavelsavara mentioned this issue Jul 24, 2023
Merged
@lewing
Copy link
Member

lewing commented Jul 24, 2023

Is this it hitting the ulimit of the container?

@lewing lewing self-assigned this Jul 24, 2023
@radical
Copy link
Member

radical commented Jul 24, 2023 

[ 7455.672962] [ 2825] 1000 2825 158937 41558 684032 0 0 python3 
[ 7455.672964] [ 4160] 102 4160 67335 453 180224 0 0 rsyslogd 
[ 7455.672966] [ 4181] 0 4181 455864 16951 634880 0 0 amacoreagent 
[ 7455.672968] [ 4453] 0 4453 298556 3977 249856 0 0 auoms 
[ 7455.672969] [ 4495] 0 4495 215902 6240 204800 0 0 auomscollect 
[ 7455.672971] [ 4750] 102 4750 191707 14176 622592 0 0 mdsd 
[ 7455.672973] [ 4967] 0 4967 334103 3030 253952 0 0 azsecd 
[ 7455.672975] [ 5046] 0 5046 352600 4193 270336 0 0 azsecmond 
[ 7455.672977] [ 32457] 0 32457 317321 3577 258048 0 0 azsecd 
[ 7455.672979] [ 2226] 0 2226 376486 334278 2924544 0 0 clamscan 
[ 7455.672981] [ 22067] 0 22067 180189 824 114688 0 -998 containerd-shim 
[ 7455.672983] [ 22089] 1000 22089 1159 16 57344 0 0 helix_docker_wo 
[ 7455.672985] [ 22156] 1000 22156 1159 17 53248 0 0 execute.sh 
[ 7455.672987] [ 22212] 1000 22212 5047 87 81920 0 0 bash 
[ 7455.672989] [ 22215] 1000 22215 68517493 67031 1163264 0 0 dotnet 
[ 7455.672990] [ 23320] 1000 23320 1159 17 61440 0 0 sh 
[ 7455.672992] [ 23321] 1000 23321 34332 22818 335872 0 0 python3 
[ 7455.672994] [ 23345] 1000 23345 1420573 1386396 11264000 0 0 wasm-opt 
[ 7455.672995] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/a2a4080691301bb74e3b282eab0b6f638e9d6f1227e62967b7fd46654706e554,task=wasm-opt,pid=23345,uid=1000 
[ 7455.673018] Out of memory: Killed process 23345 (wasm-opt) total-vm:5682292kB, anon-rss:5545584kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11000kB oom_score_adj:0

@lewing
Copy link
Member

lewing commented Jul 25, 2023

lets disable these tests in aot for now

@lewing lewing mentioned this issue Aug 13, 2023
Open
3 tasks
@lewing
Copy link
Member

lewing commented Aug 13, 2023

closting as a duplicate of dotnet/dnceng#450

@lewing lewing closed this as completed Aug 13, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 12, 2023
@pavelsavara
Copy link
Member Author

I see this again https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-99833-merge-af0760c542d34f95a8/normal-System.Runtime.Tests/1/console.f849dad3.log?helixlogtype=result

@pavelsavara pavelsavara reopened this Apr 2, 2024
@lewing
Copy link
Member

lewing commented Apr 24, 2024

Analysis claims there are recent failures but the links are not updated?

@pavelsavara
Copy link
Member Author

we run most of those problematic cases only in runtime-wasm pipeline. Which needs to be triggered manually.
I updated the filter

@lewing lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 2, 2024
@dotnet dotnet unlocked this conversation May 27, 2024
@akoeplinger akoeplinger modified the milestones: 8.0.0, 9.0.0 May 27, 2024
@lewing lewing modified the milestones: 9.0.0, 10.0.0 Aug 15, 2024
@lewing
Copy link
Member

lewing commented Feb 12, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

@lewing lewing assigned radekdoulik and unassigned lewing Feb 12, 2025
@pavelsavara pavelsavara mentioned this issue Mar 21, 2025
Merged
@lewing
Copy link
Member

lewing commented Mar 21, 2025

we hope to see this ramp down now after #113755

@pavelsavara
Copy link
Member Author

pavelsavara commented Mar 21, 2025 •
edited
Loading

Maybe we will need to go down to -O0, I think I saw SIGKILL after #113755 , now we are at -O2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@radekdoulik radekdoulik

Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
5 participants
@radical @lewing @pavelsavara @akoeplinger @radekdoulik