Feature/bug - no timeout specification

[PauloAugusto-Asos]

Feature Request / Bug Report?

The configs for this plugin do not allow us to specify the Timeout and I'm getting Timeout warnings.

Problem:

As per (lack of, maybe?) the documentation there's no way to specify the timeout value to ship the data to the InfluxDB DB.

I am getting timeout warnings that I don't know where they're coming from nor know how "much" the violation is is.

12:20:01.656 [[main]>worker7] WARN logstash.outputs.influxdb - Failed to flushoutgoing items {:outgoing_count=>1, :exception=>"Manticore::SocketTimeout", :backtrace=>["C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:in call'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/response.rb:79:in call'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:289:in post!'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-influxdb-4.0.0/lib/logstash/outputs/influxdb.rb:181:in post'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-influxdb-4.0.0/lib/logstash/outputs/influxdb.rb:169:in flush'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:221:in buffer_flush'","org/jruby/RubyHash.java:1342:in each'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in buffer_flush'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:193:in buffer_flush'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in buffer_receive'", "C:/PerfTeam_logstash-5.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-influxdb-4.0.0/lib/logstash/outputs/influxdb.rb:163:in receive'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/outputs/base.rb:92:in multi_receive'", "org/jruby/RubyArray.java:1613:in each'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/outputs/base.rb:92:in multi_receive'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:19:in multi_receive'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/output_delegator.rb:47:in multi_receive'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:390:in output_batch'", "org/jruby/RubyHash.java:1342:in each'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:346:in worker_loop'", "C:/PerfTeam_logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:306:in start_workers'"]}

My LogStash config is:
output
{
_ _ influxdb
_ _ {
_ _ _ _ host => "xxx.yyy.zzz"
_ _ _ _ port => 8086
_ _ _ _ db => "IIS"
_ _ _ _ retention_policy => "MyDefault"
_ _ _ _ measurement => "access_logs"
_ _ _ _ coerce_values => {
_ _ _ _ _ _ 'timetaken' => 'integer'
_ _ _ _ }
_ _ _ _ data_points => {
_ _ _ _ _ _ "timetaken" => "%{timetaken}"
_ _ _ _ _ _ "host" => "%{host}"
_ _ _ _ _ _ "server" => "%{server}"
_ _ _ _ _ _ "blabla" => "%{blabla}"
_ _ _ _ }
_ _ _ _ # use_event_fields_for_data_points => false
_ _ _ _ send_as_tags => [
_ _ _ _ _ _ "host",
_ _ _ _ _ _ "server",
_ _ _ _ _ _ "blabla"
_ _ _ _ ]
_ _ _ _ #flush_size => 100
_ _ _ _ idle_flush_time => 29
_ _ _ _ #time_precision => "s"
_ _ }
}

Proposal:

output {
_ _ influxdb {
_ _ _ _ timeout => 30

[PauloAugusto-Asos]

Just noticed something of interest. Even though the "Workers" limit is defaulted at 1, I'm getting those exceptions for workers reported the likes of "worker7" or "worker6" or...

I tried forcing "workers => 1" to no avail. I'm wondering whether that parameter is being respected. It might also be the cause of the timeouts, concurrency problems? Whenever I try to stop LogStash I get a pile of errors, some mentioning "concurrency".

[knepperjm]

I am seeing the same errors. Any luck with this @PauloAugusto-Asos?

[PauloAugusto-Asos]

@knepperjm
Hi,
No luck with it but to be honest I stopped shipping data to InfluxDB and moved to ElasticSearch, unfortunately...

^ I was collecting web access logs and often we have several exactly equal access log for the same time. The lines shipped to InfluxDB would of course override one another and Logstash doesn't have any mechanism I could figure out to solve those conflicts. I also investigated how to aggregate data into (ex:) 1 minute buckets but couldn't figure that out either until I gave up and started shipping the data to ElasticSearch.

[jordansissel]

Your report is for the influxdb output v4.0.0 which is a bit different from master (v5.0.0?) -- can you upgrade the plugin and try again?

[jordansissel]

I would suggest that instead of making the timeout configurable (you cannot predict the duration of network partitions), that instead of dropping the data, make it retry? I'll leave the design up to y'all, I don't know anything about influxdb, so it's hard to advise.

[knepperjm]

@PauloAugusto-Asos I was able to get the data I wanted into influxdb with the following configuration. Essentially, I just created the data points map like this:

influxdb { data_points => { "client1" => "%{client1}" "client_xff1" => "%{client_xff1}" "client2" => "%{client2}" "client_xff2" => "%{client_xff2}" "country_name" => "%{[geoip][country_name]}" "continent_code" => "%{[geoip][continent_code]}" "city_name" => "%{[geoip][city_name]}" "country_code" => "%{[geoip][country_code2]}" "location" => "%{[geoip][location]}" } ...
The documentation for the plugin could be better. Also, although the timeout would be nice, I did toggle / watch modifying varying flush_sizes for the plugin. If you enable, write-tracing in InfluxDB you can see the writes coming in from logstash and at what frequency.

[mircopolo]

My PR #55 should be in in version 5.0 of the plugin (merged 28th April) and provides a initial_delay and max_retries (backoff is exponentially increasing the value of max retries, including the option of -1 for infinite retries). The documentation for the plugin on logstash does not seem to reflect this yet for some reason.

https://github.com/logstash-plugins/logstash-output-influxdb/blob/master/docs/index.asciidoc#plugins-outputs-influxdb-initial_delay

Basically under the hood I switched it to using the ruby client for InfluxDb rather than HTTP directly

https://github.com/influxdata/influxdb-ruby.