High update diff on MapMgr

[Undergarun]

Since couple of weeks ago, i notice a MapMgr load some ticks good 50-200ms and others significantly high, 400 - 700 ms. Is possible any improvement to speedup something on MapMgr to avoid "server delay"?

P.D: vmap.Dynamic.DoubleCheck = 0 and 1000 ppl

[rsa]

this long time, or some time ago? don't remember any big changes for CPU use last year...

[rsa]

PS what your visibility radiuses in config?

[Undergarun]

3-4 weeks ago. But now more players than month ago, so source possible older.

Visibility.GroupMode = 0
Visibility.Distance.Continents = 90
Visibility.Distance.Instances = 400
Visibility.Distance.BGArenas = 180
Visibility.Distance.InFlight = 100
Visibility.Distance.Grey.Unit = 1
Visibility.Distance.Grey.Object = 10
Visibility.RelocationLowerLimit = 10
Visibility.AIRelocationNotifyDelay = 1000

[rsa]

may be new mmap quality? try disable.

[Undergarun]

How? Disabling whole mmaps?

[rsa]

no, tempdelete mmap files from one map (with max lags) and see on result.

[Undergarun]

Mmmmm seems to be related to Spell System. Getting back MaxSpellCastInChain to 25 seems to decrease load significatly, so possible some spells with endless trigger castings. U.u

[rsa]

possible... need string with this limitation from your logs.

[Undergarun]

Getting back to mmaps checks, still high MapMgr with high population. Is not the same adding maps ids to mmap.ignoreMapIds config param than delete mmap files from folder?

[rsa]

same,but not requires config reload.

[Undergarun]

Confirmed, mmaps issue. Good performance with mmaps = 0

[rsa]

mmaps requires many CPU time as intenended. possible solution - 1. lower MMAP quality (as before last changes) 2. increase the parallelization of the processing (currently i'm try this).

[mns]

MTMMAP :D

[Undergarun]

Seems as without mmaps load decreased (Obviously, less cpu resources needed) but still high update diff in MapMgr, so possible not related to changes in mmaps. Let's continue with work on paralelizing but please, check cause on other possible places.

Thank you!

[Undergarun]

Does "SoftIRQ" indicate high locking?

See this:

Dedicated R2 machine. No MySQL or any other software running then OS. Whole activity comes from R2 runtime.

MapUpdate.Threads = 10
MapUpdate.DynamicThreadsCount = 1
VMSS.Enable = 0

[rsa]

no. "softIRQ" not related to user task (like mangos) always, and indicate very high rate of IO operations, maked by kernel. this - file IO, network IO.
need make some file operations (like logging, vmaps/mmaps reading) asynchronous...
very good diagram. optimal CPU load.

[Undergarun]

Without [mr2958] ab430e0, [mr2959] 366282a, [mr2960] f372ae0, [mr2956] 4e83d6e

Some crashes:

In Map::SendObjectUpdates: http://pastebin.com/uLaM3B5c

In Map::FindObject: http://pastebin.com/mBjEC2QD

[rsa]

first - please, point me to src/game/Map.cpp:2219 in your source,
second - in 2962

[Undergarun]

            obj->BuildUpdateData(update_players); in void Map::SendObjectUpdates()

[Undergarun]

Till [mr2957] stable. Singleton locks seems to cause freezes. MapPersistentSate cycle test in progress.

[rsa]

use singleton locks instead of manual not must cause any freezes (in theory :). but this may identify weaknesses in the code

[Undergarun]

Please, check possible weaknesses. If MapPersistentState changes stable, i'll continue testing with "Null locking in ObjectAccessor" and then back to singleton locks.

"lock in BuildObjectUpdates" also under testing, hopefully no issues here, and other crash fixed.

Thank you.

[Undergarun]

Notice weird behaviour, more players == stable. Less players == unstable. I'm having crashes from 9:00 am to 15:00 pm and then nothing... if server crash during stable period, it'll be crashing till next load peak.

[Undergarun]

http://pastebin.com/4iGP4W2n VMSS on.
http://pastebin.com/Z74ZqmTi

[rsa]

VMSS make output in regular logfile. first dump - normal crush (must be correct handled by VMSS), second - broken (no point for search).

[Undergarun]

Possible source of HighUpdateDiff on MapMgr in:

SpellCastResult Spell::CheckCast(bool strict) caused by taking too long grid searchers due to "[mr2858] Fix searching for nearest creatures/gameobjects".

This the last possible source since is not a mmaps issue and not MaxSpellCastInChain. I revert this commit and i'll be test performance on next hours.

[rsa]

thx. possible need some simplify in grid search.

[Undergarun]

Just a little bit faster (50-70ms), but low diff still needed! :(

[rsa]

after
c0b572a
3939075
e2d3432
my test server (10s grid unload + 100 random bots) not crushed oved 6hours (before - crush every 3-4 hour). avg load less on 3-4% then before, update diff stable.
try also check vmap.Dynamic.DoubleCheck option - many CPU required.

[Undergarun]

http://pastebin.com/m3wnDbcZ Heavily unstable with MapPtrs... also false freezes detect.

vmap.Dynamic.DoubleCheck disabled for now, till less update diff.

[Undergarun]

Question: You know why sWorld.Update(diff); in WorldRunnable::run() loop start with ~50ms of diff?

[Undergarun]

Answer: #define WORLD_SLEEP_CONST 50 XD Forget the question!

[Undergarun]

As i understand MapMgr diff == Map.UpdateThread with highest Update() time, right? And till all MapUpdateThreads complete its Update() cycle, World::Update does not continue. Am i right?

[rsa]

right, but not full - diff NOT LESS then higest map update time (thread may work with some small maps before 571 update). also currently world thread may continue - guard timer implemented, but only 2-3 times :)