plugin not working for pubsub lite on td-agent

[kunalv89]

Hi Team, I'am using this plugin as a fluentd output. but am using pubsub lite instead of pubsub. I don't get any error and any response even though the topic has not been created using this. Please suggest.

<match ****> @type gcloud_pubsub project ***************6 key /etc/td-agent/****.json topic topic_3 autocreate_topic true max_messages 1000 max_total_size 9800000 max_message_size 4000000s timeout 60 <buffer> @type file path /var/log/td-agent/buffer/test flush_interval 3s </buffer> <format> @type json </format> </match>

[mia-0032]

@kunalv89 Hi, I'm sorry for my late reply. Was the problem solved?

[kunalv89]

@kunalv89 Hi, I'm sorry for my late reply. Was the problem solved?

@mia-0032 , Thanks for the respond, Yes problem has been resolved, i have migrated my work from td-agent to fluentd.
but again i'm facing issue while creating dynamic topic name and here is my configuration parameters. am not sure, is i'am using right placeholder.

<match tool_event> @type gcloud_pubsub project ********36 key /home/fluentd/pqr.json topic %{toolManagerName} autocreate_topic true max_messages 1000 max_total_size 9800000 max_message_size 4000000 timeout 60 <buffer> @type file path /var/log/fluentd/buffer flush_interval 3s </buffer> <format> @type json </format> </match>

Note : %{toolManagerName}, is coming in the data as a key as below
data o/p - {"processid":"9","toolmanagerName":"loft","paramid":"****"}

here is my gcloud_pubsub version -
fluent-plugin-gcloud-pubsub-custom (1.6.0)

[mia-0032]

@kunalv89 Could you try below config? This configuration uses buffer placeholder feature of Fluentd.
https://docs.fluentd.org/configuration/buffer-section#placeholders

<match tool_event>
  @type gcloud_pubsub
  project ********36
  key /home/fluentd/pqr.json
  topic ${toolManagerName}
  autocreate_topic true
  max_messages 1000
  max_total_size 9800000
  max_message_size 4000000
  timeout 60
  <buffer toolManagerName>
    @type file path /var/log/fluentd/buffer
    flush_interval 3s
  </buffer>
  <format>
    @type json
  </format>
</match> 

[kunalv89]

@kunalv89 Could you try below config? This configuration uses buffer placeholder feature of Fluentd. https://docs.fluentd.org/configuration/buffer-section#placeholders

<match tool_event>
  @type gcloud_pubsub
  project ********36
  key /home/fluentd/pqr.json
  topic ${toolManagerName}
  autocreate_topic true
  max_messages 1000
  max_total_size 9800000
  max_message_size 4000000
  timeout 60
  <buffer toolManagerName>
    @type file path /var/log/fluentd/buffer
    flush_interval 3s
  </buffer>
  <format>
    @type json
  </format>
</match> 

@mia-0032 , this is one is flawless. but just i got to know that, data is going on to PubSub instead of PubSub lite. For PubSub Lite, are we require any additional plugin to transmitting data.

[kunalv89]

Hello @mia-0032 , Can we do PubSub Lite integration with this plugin on fluentd, i have searched alot for pubsub lite plugin but there is no direct plugin available.

[mia-0032]

@kunalv89 This plugin uses google-cloud-pubsub gem. But the gem doesn't support Pub/Sub Lite. Please see googleapis/google-cloud-ruby#8930 .
If the gem will support Pub/Sub Lite, I will support Pub/Sub Lite on this plugin.