crash -> dont start

[Envel-Nikita-Gutsenkov]

[11:46:45 WARN]: [PopulationDensity] Task #2 for PopulationDensity v5.12 generated an exception
java.lang.ThreadDeath: null
at java.lang.Thread.stop(Unknown Source) [?:1.8.0_251]
at net.minecraft.server.v1_15_R1.MinecraftServer.stop(MinecraftServer.java:736) ~[patched_1.15.2.jar:git-Tuinity-"cb75b52"]
at net.minecraft.server.v1_15_R1.DedicatedServer.stop(DedicatedServer.java:754) ~[patched_1.15.2.jar:git-Tuinity-"cb75b52"]
at net.minecraft.server.v1_15_R1.MinecraftServer.close(MinecraftServer.java:712) ~[patched_1.15.2.jar:git-Tuinity-"cb75b52"]
at org.spigotmc.WatchdogThread.run(WatchdogThread.java:236) ~[patched_1.15.2.jar:git-Tuinity-"cb75b52"]
[11:46:45 ERROR]: Main thread terminated by WatchDog due to hard crash
java.lang.ThreadDeath: null

[RoboMWM]

ThreadDeath...?

Need more log

[zedwick]

Having just updated my own server I experienced what I believe to be the same issue. It's caused by the scan region task taking too long, and the server stopping itself. I've not fixed the issue, but my mitigation has been to add a new config option to allow disabling the ScanRegionTask entirely.

I don't have this problem on my test server, only in production.

[12:36:56] [Paper Watchdog Thread/ERROR]: --- DO NOT REPORT THIS TO PAPER - THIS IS NOT A BUG OR A CRASH  - git-Paper-89 (MC: 1.16.1) ---                                                                          
[12:36:56] [Paper Watchdog Thread/ERROR]: The server has not responded for 10 seconds! Creating thread dump                                                                                                        
[12:36:56] [Paper Watchdog Thread/ERROR]: ------------------------------                                                                                                                                           
[12:36:56] [Paper Watchdog Thread/ERROR]: Server thread dump (Look for plugins here before reporting to Paper!):                                                                                                   
[12:36:56] [Paper Watchdog Thread/ERROR]: Chunk wait task info below:                                                                                                                                              
[12:36:56] [Paper Watchdog Thread/ERROR]: -366,-23 in 'world:                                                                                                                                                      
[12:36:56] [Paper Watchdog Thread/ERROR]: Load Task - none                                                                                                                                                         
[12:36:56] [Paper Watchdog Thread/ERROR]: Save Task - none                                                                                                                                                         
[12:36:56] [Paper Watchdog Thread/ERROR]: Chunk Holder - non-null                                                                                                                                                  
[12:36:56] [Paper Watchdog Thread/ERROR]: Chunk Status - minecraft:full                                                                                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]: Chunk Ticket Status - minecraft:full                                                                                                                                     
[12:36:56] [Paper Watchdog Thread/ERROR]: Chunk Holder Status - minecraft:features                                                                                                                                 
[12:36:56] [Paper Watchdog Thread/ERROR]: Chunk Holder Priority - 2                                                                                                                                                
[12:36:56] [Paper Watchdog Thread/ERROR]: ------------------------------                                                                                                                                           
[12:36:56] [Paper Watchdog Thread/ERROR]: Current Thread: Server thread                                                                                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:       PID: 31 | Suspended: false | Native: false | State: TIMED_WAITING                                                                                                  
[12:36:56] [Paper Watchdog Thread/ERROR]:       Stack:                                                                                                                                                             
[12:36:56] [Paper Watchdog Thread/ERROR]:               java.base@11.0.7/jdk.internal.misc.Unsafe.park(Native Method)                                                                                              
[12:36:56] [Paper Watchdog Thread/ERROR]:               java.base@11.0.7/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)                                                                    
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.IAsyncTaskHandler.bk(IAsyncTaskHandler.java:131)                                                                        
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.IAsyncTaskHandler.awaitTasks(IAsyncTaskHandler.java:120)                                                                
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:511)                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.IChunkProvider.getChunkAt(SourceFile:16)                                                                                
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//org.bukkit.craftbukkit.v1_16_R1.CraftWorld.getChunkAt(CraftWorld.java:409)                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:               me.ryanhamshire.PopulationDensity.PopulationDensity.scanRegion(PopulationDensity.java:1264)                                                                
[12:36:56] [Paper Watchdog Thread/ERROR]:               me.ryanhamshire.PopulationDensity.ScanOpenRegionTask.run(ScanOpenRegionTask.java:27)                                                                       
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//org.bukkit.craftbukkit.v1_16_R1.scheduler.CraftTask.run(CraftTask.java:99)                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//org.bukkit.craftbukkit.v1_16_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:468)                                                 
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.MinecraftServer.b(MinecraftServer.java:1291)                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.DedicatedServer.b(DedicatedServer.java:377)                                                                             
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.MinecraftServer.a(MinecraftServer.java:1212)                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.MinecraftServer.v(MinecraftServer.java:1000)                                                                            
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.MinecraftServer.lambda$a$0(MinecraftServer.java:177)                                                                    
[12:36:56] [Paper Watchdog Thread/ERROR]:               app//net.minecraft.server.v1_16_R1.MinecraftServer$$Lambda$3044/0x0000000840829440.run(Unknown Source)                                                     
[12:36:56] [Paper Watchdog Thread/ERROR]:               java.base@11.0.7/java.lang.Thread.run(Thread.java:834)                                                                                                     
[12:36:56] [Paper Watchdog Thread/ERROR]: ------------------------------                                                                                                                                           
[12:36:56] [Paper Watchdog Thread/ERROR]: --- DO NOT REPORT THIS TO PAPER - THIS IS NOT A BUG OR A CRASH ---                                                                                                       
[12:36:56] [Paper Watchdog Thread/ERROR]: ------------------------------  

Which repeats until Paper stops the server. Will need to figure out why it's taking so long to complete the region scan.

[RoboMWM]

Tbh, if we can get this to use Paper's API for chunk loading, I wouldn't be opposed (with a fallback so this can still be published to spigot) since spigot is no longer hosting the Bukkit API on their maven repo. Otherwise any other mitigation you wish to implement is fair game.

[zedwick]

There is PaperLib.

PaperLib is a plugin library for interfacing with Paper specific APIs (such as async chunk loading), with graceful fallbacks maintaining compatibility with both the Bukkit and Spigot API's.

Which would make it easy to use async chunk loading/generating on Paper, and sync chunk loading/generating on Spigot. Though I've not tried this yet, myself.

[RoboMWM]

Ah yea, for some reason I forgot about PaperLib. I've used it in a project and it is pretty easy to use.

I believe I've used this in another project, but here's where I used it along with TaskChain, which may also be helpful if you're looking to refactor that code logic. https://github.com/MLG-Fortress/WildWormholes

[zedwick]

TaskChain

I don't need to use the region scan feature on my server very often, so this isn't a high priority for me to work on. If anyone else is interested in improving its performance, please feel free to submit a PR. Otherwise I will probably look at this eventually.

[RoboMWM]

The easiest optimization is probably to not have it run on server startup or delay it to run several minutes afterwards, and then keep the interval as-is for future scans.

[DerekWearmouth]

The easiest optimization is probably to not have it run on server startup or delay it to run several minutes afterwards, and then keep the interval as-is for future scans.

The issue is that this crash issues is trigger more than on just scans, its trigger between scans as well. I mention as my scans are set to every 24hrs and we experienced massive crash./rollbacks 8 times yesterday.