OutOfMemory crash on warming all-cities over zoom level 12

[sprutner]

Looks like the JVM runs out of memory. Any thoughts on the best way to resolve this? Would like to warm these zoom levels all at once instead of with multiple jobs ideally.

<--- Last few GCs --->

[41282:0x105000600]    36453 ms: Scavenge 1154.8 (1421.6) -> 1154.1 (1422.6) MB, 2.6 / 0.0 ms  (average mu = 0.145, current mu = 0.115) allocation failure
[41282:0x105000600]    36457 ms: Scavenge 1154.9 (1422.6) -> 1154.4 (1423.1) MB, 1.9 / 0.0 ms  (average mu = 0.145, current mu = 0.115) allocation failure
[41282:0x105000600]    36470 ms: Scavenge 1156.3 (1423.1) -> 1155.8 (1423.6) MB, 3.2 / 0.0 ms  (average mu = 0.145, current mu = 0.115) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x2d4472b841bd]
    1: StubFrame [pc: 0x2d4472b95610]
Security context: 0x3efb3781e6c9 <JSObject>
    2: getTiles [0x3efb60ceab51] [/usr/local/lib/node_modules/@alvarcarto/tilewarm/node_modules/@mapbox/tile-cover/index.js:~56] [pc=0x2d4472ebbe35](this=0x3efb602c9bd9 <Object map = 0x3efb5d9d9549>,geom=0x3efbcf0b2f59 <Object map = 0x3efb5d9e4651>,limits=0x3efbc0615d49 <Object map = 0x3efb5d9de981>)
    3: /* anonymous ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100033dab node::Abort() [/usr/local/bin/node]
 2: 0x10003557e node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node]
 3: 0x1001776a7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100177649 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10045004a v8::internal::Heap::UpdateSurvivalStatistics(int) [/usr/local/bin/node]
 6: 0x100451ced v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 7: 0x10044f3af v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10044e300 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x1004564f7 v8::internal::Heap::AllocateRawWithRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x100434108 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
11: 0x10062d8c5 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x2d4472b841bd
[1]    41282 abort      tilewarm http://tiles-cloudfront.life360.com/tile/\{z\}/\{x\}/\{y\}.png```

[kimmobrunfeldt]

Interesting, this utility shouldn't consume much memory, sounds like a potential bug.

As a workaround you could try to increase the max old memory space with: NODE_OPTIONS=--max_old_space_size=4096 tilewarm ...

[kimmobrunfeldt]

Right after I commented I realized that the urls generated by this utility are saved in memory. Potentially trying to keep tens of millions of url strings in an array. Definitely not the best design, and should be fixed to use e.g. generators. I won't be fixing this for a while at least, but open to a PR.