Is there any way to drop chunk for one output if that output upstream can't catch up?

[rendyfebry]

Hi

I have a fluent bit sidecar that sends logs to Cloudwatch and Fluentd.

[INPUT]
    Name           forward
    unix_path      /var/run/fluent.sock
    Mem_Buf_Limit  100MB
    storage.type   filesystem

// Some filter

[OUTPUT]
    Name             cloudwatch
    Match            *
    region           ap-northeast-2
    log_group_name   ${SERVICE_NAME}-${ENV}-lg
    log_stream_name  /app/$(ecs_task_id)

[OUTPUT]
    Name                        forward
    Match                       *
    Host                        ${LOG_FORWARD_HOST}
    Port                        ${LOG_FORWARD_PORT}
    Time_as_Integer             True
    Tag                         tail.${ROLE_NAME}.tail

The problem is, sometimes there's a time when our Fluentd server can't keep up with the logs throughput. And when this happens it will put backpressure on our fluent bit and pause the input plugin altogether and we lose logs entirely.

So, is there a way to drop the chunk if the fluent bit is unable to deliver them to one output without affecting the other output?

Maybe something similar to drop_oldest_chunk behavior on Fluentd buffer.

[agup006]

One parameter that might be able to perform this is storage.total_limit_size in one of the outputs. I'm not 100% familiar with the behavior of two output plugins but I believe it will mimic the behavior of drop_oldest_chunk

[patrick-stephens]

Does the retry configuration solve this?
If you set it to no_retries for forward then it should just move on?
https://docs.fluentbit.io/manual/administration/scheduling-and-retries

[JeffLuoo]

The other input plugin won't be affected. The implementation of storage.total_limit_size is per output plugin.