Can we have multiple outputs support in fluent bit #5509

ujala-singh · 2022-05-25T11:51:08Z

ujala-singh
May 25, 2022

I am running this config:

fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
    
    [INPUT]
        Name                tail
        Path                /var/log/containers/*.log
        Docker_Mode         On
        Docker_Mode_Parser  multi_line
        Tag                 containers.*
        Refresh_Interval    1
        Rotate_Wait         60
        Mem_Buf_Limit       100MB
        Buffer_Max_Size     1MB
        Buffer_Chunk_Size   100k
        Skip_Long_Lines     Off
        DB                  /tail-db/tail-containers-state-sumo.db
        DB.Sync             Normal
    [INPUT]
        Name                tail
        Path                /var/log/fluentd/*/*/trace/*.log
        Parser              json
        Tag                 trace.*
        Path_Key            tailed_path
        Refresh_Interval    1
        Rotate_Wait         60
        Mem_Buf_Limit       100MB
        Buffer_Max_Size     1MB
        Buffer_Chunk_Size   100k
        Skip_Long_Lines     Off
        DB                  /tail-db/tail-trace-sumo.db
        DB.Sync             Normal
    [INPUT]
        Name                tail
        Path                /var/log/fluentd/*/*/apache/*-error.log
        Parser              apache2
        Tag                 apache.error.*
        Path_Key            tailed_path
        Refresh_Interval    1
        Rotate_Wait         60
        Mem_Buf_Limit       10MB
        Buffer_Max_Size     1MB
        Buffer_Chunk_Size   100k
        Skip_Long_Lines     Off
        DB                  /tail-db/tail-apache-error-sumo.db
        DB.Sync             Normal
    [INPUT]
        Name                tail
        Path                /var/log/fluentd/*/*/apache/*-access.log
        Parser              apache2
        Tag                 apache.access.*
        Path_Key            tailed_path
        Refresh_Interval    1
        Rotate_Wait         60
        Mem_Buf_Limit       10MB
        Buffer_Max_Size     1MB
        Buffer_Chunk_Size   100k
        Skip_Long_Lines     Off
        DB                  /tail-db/tail-apache-access-sumo.db
        DB.Sync             Normal
    
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
    
    [OUTPUT]
        Name          forward
        Match         *
        Host          ${FLUENTD_LOGS_SVC}.${NAMESPACE}.svc.cluster.local.
        Port          24321
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

In this Host is a service cluster endpoint. But I have more than one STS of fluentd in three AZ's which are having two different svc enpoints. How can I do this efficiently??

Basically I want to forward the data from fluent-it to three different svc endpoints.

Answered by patrick-stephens

May 25, 2022

Yes, you can send to multiple outputs of different types too. It's one of the benefits of Fluent Bit that it is vendor agnostic. You can even send subsets of data to different outputs.
See https://docs.fluentbit.io/manual/concepts/data-pipeline/router

View full answer

ujala-singh · 2022-05-25T11:51:54Z

ujala-singh
May 25, 2022
Author

If something like this is feasible:

[OUTPUT]
        Name          forward
        Match         *
        Host          ${FLUENTD_LOGS_SVC_1a}.${NAMESPACE}.svc.cluster.local.
        Port          24321
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

[OUTPUT]
        Name          forward
        Match         *
        Host          ${FLUENTD_LOGS_SVC_1b}.${NAMESPACE}.svc.cluster.local.
        Port          24321
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

[OUTPUT]
        Name          forward
        Match         *
        Host          ${FLUENTD_LOGS_SVC_1c}.${NAMESPACE}.svc.cluster.local.
        Port          24321
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

1 reply

patrick-stephens Mar 27, 2023
Maintainer

Yes, all should be fine - same as sending to three hosts.

Similarly you can use different types of outputs as well.

patrick-stephens · 2022-05-25T18:20:13Z

patrick-stephens
May 25, 2022
Maintainer

Yes, you can send to multiple outputs of different types too. It's one of the benefits of Fluent Bit that it is vendor agnostic. You can even send subsets of data to different outputs.
See https://docs.fluentbit.io/manual/concepts/data-pipeline/router

0 replies

ujala-singh · 2022-05-26T06:19:11Z

ujala-singh
May 26, 2022
Author

@patrick-stephens I tried using this, but somehow logs are getting pushed to only one output worker not all of them:

[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545182.206968378.flb', retry in 76 seconds: task_id=362, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545206.231770939.flb', retry in 511 seconds: task_id=708, input=tail.1 > output=forward.1 (out_id=1)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545206.264011914.flb', retry in 640 seconds: task_id=704, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545164.318201865.flb', retry in 238 seconds: task_id=409, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545288.574720804.flb', retry in 26 seconds: task_id=1343, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545325.640325085.flb', retry in 174 seconds: task_id=1680, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545197.205522889.flb', retry in 291 seconds: task_id=629, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545140.206105574.flb', retry in 25 seconds: task_id=228, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545222.910017801.flb', retry in 251 seconds: task_id=836, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545288.208967791.flb', retry in 279 seconds: task_id=1336, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:32] [ warn] [engine] failed to flush chunk '1-1653545117.209606887.flb', retry in 104 seconds: task_id=33, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:33] [error] [output:forward:forward.2] no upstream connections available
[2022/05/26 06:14:33] [ warn] [engine] failed to flush chunk '1-1653545222.925405178.flb', retry in 787 seconds: task_id=838, input=tail.3 > output=forward.2 (out_id=2)
[2022/05/26 06:14:33] [error] [output:forward:forward.2] no upstream connections available
[2022/05/26 06:14:33] [error] [output:forward:forward.0] no upstream connections available
[2022/05/26 06:14:33] [error] [output:forward:forward.0] no upstream connections available
[2022/05/26 06:14:33] [error] [output:forward:forward.0] no upstream connections available
[2022/05/26 06:14:33] [error] [output:forward:forward.0] no upstream connections available
[2022/05/26 06:14:33] [ warn] [engine] failed to flush chunk '1-1653545341.554374280.flb', retry in 640 seconds: task_id=1778, input=tail.3 > output=forward.2 (out_id=2)
[2022/05/26 06:14:33] [ warn] [engine] failed to flush chunk '1-1653545148.992032558.flb', retry in 551 seconds: task_id=287, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:33] [ warn] [engine] failed to flush chunk '1-1653545337.206911479.flb', retry in 155 seconds: task_id=1737, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:33] [ warn] [engine] failed to flush chunk '1-1653545358.463221993.flb', retry in 185 seconds: task_id=1955, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:14:33] [ warn] [engine] failed to flush chunk '1-1653545377.206796927.flb', retry in 45 seconds: task_id=1219, input=tail.0 > output=forward.0 (out_id=0)

Although logs are getting succeeded only for output 0:

[2022/05/26 06:04:45] [ info] [engine] flush chunk '1-1653545056.585580723.flb' succeeded at retry 1: task_id=90, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:04:46] [ info] [engine] flush chunk '1-1653545061.402631314.flb' succeeded at retry 1: task_id=102, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:04:46] [ info] [engine] flush chunk '1-1653545059.352915125.flb' succeeded at retry 1: task_id=87, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:04:48] [ info] [engine] flush chunk '1-1653545050.495142597.flb' succeeded at retry 2: task_id=74, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:03] [ info] [engine] flush chunk '1-1653545065.596543580.flb' succeeded at retry 2: task_id=114, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:03] [ info] [engine] flush chunk '1-1653545048.577024314.flb' succeeded at retry 3: task_id=69, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:03] [ info] [engine] flush chunk '1-1653545053.563572644.flb' succeeded at retry 2: task_id=80, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:04] [ info] [engine] flush chunk '1-1653545081.352590026.flb' succeeded at retry 1: task_id=162, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:05] [ info] [engine] flush chunk '1-1653545032.352574874.flb' succeeded at retry 3: task_id=23, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:06] [ info] [engine] flush chunk '1-1653545056.364585895.flb' succeeded at retry 2: task_id=89, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:06] [ info] [engine] flush chunk '1-1653545087.651266008.flb' succeeded at retry 1: task_id=179, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:06] [ info] [engine] flush chunk '1-1653545058.851870208.flb' succeeded at retry 2: task_id=96, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:06] [ info] [engine] flush chunk '1-1653545083.355706012.flb' succeeded at retry 1: task_id=169, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:07] [ info] [engine] flush chunk '1-1653545028.353351566.flb' succeeded at retry 3: task_id=9, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:07] [ info] [engine] flush chunk '1-1653545061.667869978.flb' succeeded at retry 2: task_id=103, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:08] [ info] [engine] flush chunk '1-1653545091.637853259.flb' succeeded at retry 1: task_id=190, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:08] [ info] [engine] flush chunk '1-1653545037.352705098.flb' succeeded at retry 3: task_id=35, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:15] [ info] [engine] flush chunk '1-1653545090.351760446.flb' succeeded at retry 1: task_id=185, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:16] [ info] [engine] flush chunk '1-1653545045.764222155.flb' succeeded at retry 3: task_id=60, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:16] [ info] [engine] flush chunk '1-1653545078.353601106.flb' succeeded at retry 2: task_id=153, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:17] [ info] [engine] flush chunk '1-1653545035.352419766.flb' succeeded at retry 3: task_id=31, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:17] [ info] [engine] flush chunk '1-1653545088.582889261.flb' succeeded at retry 1: task_id=180, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:18] [ info] [engine] flush chunk '1-1653545062.353733352.flb' succeeded at retry 2: task_id=105, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:28] [ info] [engine] flush chunk '1-1653545085.627933538.flb' succeeded at retry 2: task_id=176, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:29] [ info] [engine] flush chunk '1-1653545043.851498688.flb' succeeded at retry 3: task_id=55, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:41] [ info] [engine] flush chunk '1-1653545111.356730259.flb' succeeded at retry 1: task_id=231, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:46] [ info] [engine] flush chunk '1-1653545107.355290776.flb' succeeded at retry 2: task_id=103, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:46] [ info] [engine] flush chunk '1-1653545117.354242188.flb' succeeded at retry 1: task_id=180, input=tail.0 > output=forward.0 (out_id=0)
[2022/05/26 06:05:47] [ info] [engine] flush chunk '1-1653545101.354066065.flb' succeeded at retry 2: task_id=216, input=tail.0 > output=forward.0 (out_id=0)

Currently this config I am using:

[OUTPUT]
        Name          forward
        Match         *containers*
        Host          ${FLUENTD_LOGS_SVC}.${NAMESPACE}.svc.cluster.local.
        Port          24321
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

    [OUTPUT]
        Name          forward
        Match         *trace*
        Host          ${FLUENTD_LOGS_SVC_b}.${NAMESPACE}.svc.cluster.local.
        Port          24322
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

    [OUTPUT]
        Name          forward
        Match         *apache*
        Host          ${FLUENTD_LOGS_SVC_c}.${NAMESPACE}.svc.cluster.local.
        Port          24323
        Retry_Limit   False
        tls           off
        tls.verify    on
        tls.debug     1
        # Disable keepalive for better load balancing
        net.keepalive off

Fluent-bit version: fluent/fluent-bit:1.9.3-debug

0 replies

patrick-stephens · 2022-05-26T19:25:39Z

patrick-stephens
May 26, 2022
Maintainer

What are the variables set to and can you confirm they're reachable from the pod? You're running the debug image so can exec in and there's a lot of debug tools provided plus you can install more.

0 replies

AHMEDSAAA · 2022-11-29T12:54:42Z

AHMEDSAAA
Nov 29, 2022

so what was the final output of this discussion and how you have been able to let fluent-bit forward the logs to different BackEnd servers endpoints

1 reply

patrick-stephens Mar 27, 2023
Maintainer

It should work fine, I've done similar things with no issues.
I've also set up outputs to different types of backend too, e.g. Loki, Elastic and others all at the same time.

nagarajkoppa · 2023-02-28T12:26:36Z

nagarajkoppa
Feb 28, 2023

Could you let me know the output of this discussion

1 reply

patrick-stephens Mar 27, 2023
Maintainer

Not really sure what to say: Fluent Bit supports multiple outputs (of multiple types as well).

dmytro-riabichenko · 2023-03-27T11:20:56Z

dmytro-riabichenko
Mar 27, 2023

Any update on this issue?

1 reply

patrick-stephens Mar 27, 2023
Maintainer

Not sure what to say, it should work fine. Guessing some kind of config issue.

randomizedcoder · 2023-10-25T01:36:33Z

randomizedcoder
Oct 25, 2023

G'day @patrick-stephens ,

I hope you are doing well. Looks like you've been traveling which I'm sure was fun.

I was wondering the same thing about multiple outputs and was searching around and looking at issues/discussions.

I suspect what would help peoples question is an example in the https://docs.fluentbit.io/manual/concepts/data-pipeline/router page showing send the same message to x2 outputs.

The examples today are:

[INPUT] my_cpu -> OUTPUT my_cpu
[INPUT] my_mem -> OUTPUT my_mem

Cool. So that's 1:1 mapping.

[INPUT] my_cpu ->
>-----> OUTPUT my_cpu + my_mem
[INPUT] my_mem ->/

( Forgive the poor ASCII art )

Cool. So that's 2:1 mapping.

But what there is not an example of is

[INPUT] my_cpu -> OUTPUT my_cpu_output1
[INPUT] my_cpu -> OUTPUT my_cpu_output2
[INPUT] my_cpu -> OUTPUT my_cpu_outputN

1:3 or 1:N mapping

I think you are suggesting this will "just work", but as you know, we all just want to copy and pasta ;)

1 reply

patrick-stephens Oct 30, 2023
Maintainer

Sure, be great to have a docs update for it.
https://github.com/fluent/fluent-bit-docs/blob/master/concepts/data-pipeline/router.md

lianghuiyuan · 2024-09-07T02:54:41Z

lianghuiyuan
Sep 7, 2024

[INPUT] my_cpu -> OUTPUT my_cpu_output1
[INPUT] my_cpu -> OUTPUT my_cpu_output2
[INPUT] my_cpu -> OUTPUT my_cpu_outputN

follow example is working

# Input Configuration
[INPUT]
    Name  cpu
    Tag   my_tag

# Output to Loki
[OUTPUT]
    Name  loki
    Match my_tag
    Host  loki-server
    Port  3100
    labels job=fluentbit
    # Additional Loki-specific configurations

# Output to Kafka
[OUTPUT]
    Name  kafka
    Match my_tag
    Brokers 127.0.0.1:9092
    Topics my_topic
    # Additional Kafka-specific configurations

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can we have multiple outputs support in fluent bit #5509

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Can we have multiple outputs support in fluent bit #5509

ujala-singh May 25, 2022

Replies: 9 comments · 5 replies

ujala-singh May 25, 2022 Author

patrick-stephens Mar 27, 2023 Maintainer

patrick-stephens May 25, 2022 Maintainer

ujala-singh May 26, 2022 Author

patrick-stephens May 26, 2022 Maintainer

AHMEDSAAA Nov 29, 2022

patrick-stephens Mar 27, 2023 Maintainer

nagarajkoppa Feb 28, 2023

patrick-stephens Mar 27, 2023 Maintainer

dmytro-riabichenko Mar 27, 2023

patrick-stephens Mar 27, 2023 Maintainer

randomizedcoder Oct 25, 2023

patrick-stephens Oct 30, 2023 Maintainer

lianghuiyuan Sep 7, 2024

ujala-singh
May 25, 2022

Replies: 9 comments 5 replies

ujala-singh
May 25, 2022
Author

patrick-stephens Mar 27, 2023
Maintainer

patrick-stephens
May 25, 2022
Maintainer

ujala-singh
May 26, 2022
Author

patrick-stephens
May 26, 2022
Maintainer

AHMEDSAAA
Nov 29, 2022

patrick-stephens Mar 27, 2023
Maintainer

nagarajkoppa
Feb 28, 2023

patrick-stephens Mar 27, 2023
Maintainer

dmytro-riabichenko
Mar 27, 2023

patrick-stephens Mar 27, 2023
Maintainer

randomizedcoder
Oct 25, 2023

patrick-stephens Oct 30, 2023
Maintainer

lianghuiyuan
Sep 7, 2024