Q&A what does utilization metric show?

[scMarkus]

Question

I am confused about how the utilization. in my case I do see 100% file sink utilization and buffers filing up but the cpu usage is relatively low. The sink is processing about 2k events a second (at times even more).

My question is how to understand the utilization metric as well as how I may increase throughput of the sink?

in case vector is trying to flush every single event to disk I could image that the disk IO performance is the bottle neck?
My intuition would have been that the parsing in the transform component would be the most cpu expensive process followed by the sinks CSV encoding. But in both cases I would expected higher cpu usage.

Vector Config

---
data_dir: /logs/
api:
  enabled: true
sources:
  prom_in:
    type: internal_metrics
    scrape_interval_secs: 15
    tags:
      host_key: ""
  in:
    type: file
    file_key: ""
    host_key: ""
    fingerprint:
      strategy: device_and_inode
    include: [/logs/log.in]

transforms:
  prom_pars:
    type: filter
    inputs: [prom_in]
    condition: |-
      includes([
        "component_received_events_total",
        "component_sent_events_total",
        "component_received_event_bytes_total",
        "component_sent_event_bytes_total",
        "internal_metrics_cardinality",
        "utilization",
        "buffer_events",
        "buffer_received_events_total",
      ], .name)

  pars:
    type: remap
    inputs: [in]
    source: |
      line = parse_csv(string(.message) ?? "err=no_string", "\t") ?? ["csv_parser_error"]
      . = parse_query_string(line[6]) ?? {"ses": "query_parsing_error"}

      float_ts, err = parse_float(line[0])
      .timestamp = if (err == null) {
        from_unix_timestamp(to_int(float_ts * 1000), "milliseconds") ?? now()
      } else {
        now()
      }
      .a = line[1]
      .b = line[2]
      .c = line[3]
      .d = line[4]
      .e = line[5]

      . = map_values(., false) -> |value| {
        if is_array(value) {
          join(value, "\t") ?? encode_json(value)
        } else {
          value
        }
      }

sinks:
  prom_up:
    type: prometheus_exporter
    inputs: [prom_pars]
    buffer:
      when_full: drop_newest
    address: 0.0.0.0:9101
    suppress_timestamp: true
  up:
    type: file
    inputs: [pars]
    path: /upload/up/%Y-%m-%dT%H:%M
    buffer:
      type: memory
      max_events: 10000
    framing:
      method: newline_delimited
    encoding:
      codec: csv
      csv:
        delimiter: "\t"
        fields:
          - timestamp
          # - ... (15 other ones)

Vector Logs

No response

[pront]

Hi @scMarkus, from the docs here, a value of 1 (100%) would indicate a that the sink is never idle. Which means there are always events to process. I would also examine the buffer metrics to get a better picture.

For better performance, we will probably have to make the file sink more efficient. For example, see existing issue: #20394

[scMarkus]

@pront thanks a lot for your response.
this was clarifying to me. in terms of the file sink limitations I will look into the related issue. thanks again :)