Support "spillover" to a different destination when receiving backpressure from default destination

[edinel]

Is your feature request related to a problem? Please describe.
Today, if I have a situation where the volume of content (let's assume logs for now) being pushed out by fluent bit exceeds what the destination can successfully accept and ingest, we get a backpressure signal. This may come in many forms, but generally, the response to it is to buffer the data to-be-sent until there is more than can be held onto, and then to drop it. Rather than that, we would like to have that data sent to a different, lower-priority destination which will always accept the data without backpressure (e.g. a public cloud blob store)

Secondarily, we would then like to be able to easily "rehydrate" that data later from the fallback destination and re-send it to the primary destination.

Describe the solution you'd like
We would like an option for backpressure to, instead of causing the oldest or newest logs to be dropped, to instead cause those logs to be routed to a separately configured destination.

Describe alternatives you've considered
Possibly the logs could be written to a blob storage and continuously rehydrated to get to the destination, but this will result in substantially more network traffic than would otherwise be necessary (and more storage of data in that blob store)

Additional context
The goal here is to have a situation in which, even if the target destination becomes unavailable or bogged down, logs are never lost -- obviously provided that the remote blob store remains available. :-)

[edsiper]

what would be the ideal conditions when the secondary is used ?

cc: @cosmo0920 this sounds pretty similar to Fluentd functionality

[cosmo0920]

This could be kicked off after the retrying limits should be exceeded situations.
However, the implementation of Fluentd behaves slightly different approach.
It execute the secondary outputs on exceeded limit ratio. This is set up as 0.8. So, 80% of the retrying limits are consumed for the retrying mechanism, secondary outputs which are specified under <secondary> section should be executed on Fluentd.

[patrick-stephens]

I think there was a previous feature request on this...

[patrick-stephens]

Yeah I think duplicate of #6407