Crucible buffer size limit when querying data

[sferlin]

Crucible seems to have a limit for a 64kB buffer when querying data.

A corrupt json entry is returned when crucible is queried for an interval (or for a resolution) that makes it fill its buffer.

Example with an invalid json, if I want e.g. 10min. worth of samples (with throughput/s):

/usr/bin/crucible get metric --source uperf --type Gbps --run $crucible_uuid --begin $begin_ts --end $end_ts --output-format json --resolution $samples_churn_interval

  {
        "begin": 1746984725131,
        "end": 1746984726815,
        "value": 1058.1054806694362
      },
      {
        "begin": 1746984726816,
        "end": 1746984728500,
        "value": 1059.0304927893176

A workaround is to query crucible multiple times with non-overlapping intervals, in batches, and work with the timestamps.

[k-rister]

@atheurer Any thoughts on this given your recent work?

[atheurer]

With recent work it seems to be a different problem, at least with a resolution of 1000:

# crucible get metric --period 33D2C7AC-3B35-11F0-9D53-8A2D77D88935 --type Busy-CPU --source mpstat --resolution 1000

<--- Last few GCs --->

[165635:0x5638b9408000]    59855 ms: Scavenge (interleaved) 4044.4 (4120.7) -> 4044.4 (4123.7) MB, pooled: 0 MB, 23.87 / 0.00 ms  (average mu = 0.608, current mu = 0.432) allocation failure;
[165635:0x5638b9408000]    59894 ms: Scavenge 4050.7 (4127.0) -> 4050.7 (4148.0) MB, pooled: 0 MB, 18.96 / 0.00 ms  (average mu = 0.608, current mu = 0.432) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x7f15fdea5958 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/lib64/libnode.so.127]
 2: 0x7f16000a5b84 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/lib64/libnode.so.127]
 3: 0x7f16000a5e7a v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/lib64/libnode.so.127]
 4: 0x7f16002ee4db  [/lib64/libnode.so.127]
 5: 0x7f160030921d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/lib64/libnode.so.127]
 6: 0x7f16002dfc9b v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib64/libnode.so.127]
 7: 0x7f16002e0388 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib64/libnode.so.127]
 8: 0x7f16002bb16b v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/lib64/libnode.so.127]
 9: 0x7f16002aa8be v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/lib64/libnode.so.127]
10: 0x7f16004269ce v8::internal::JsonParser<unsigned char>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [/lib64/libnode.so.127]
11: 0x7f160042e756  [/lib64/libnode.so.127]
12: 0x7f1600435c33 v8::internal::JsonParser<unsigned char>::ParseJson(v8::internal::Handle<v8::internal::Object>) [/lib64/libnode.so.127]
13: 0x7f1600129dc7 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [/lib64/libnode.so.127]
14: 0x7f159ff2c3b6

[atheurer]

This will be addressed with perftool-incubator/CommonDataModel#138

[k-rister]

We believe this has been fixed. Please re-open if you the problem reoccurs.

[sferlin]

Is this a recent fix? I have tested with my local crucible here (a few commits behind) and it is still manifesting the same issue with --resolution 1000

[k-rister]

See perftool-incubator/CommonDataModel#138