Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opensearch version and distribution check fails if request path is set #69

Open
ConaghFM opened this issue Jul 17, 2022 · 0 comments
Open

Comments

@ConaghFM
Copy link

Steps to replicate

Set the path parameter in the output plugin config (the block) to post requests to a specific index URL.
It is necessary to be able to do this for access control, but it causes the version/distribution check to make it's request to the index instead of the base URL, and does not get the version and distribution parameters it is requesting:

2022-07-17 10:24:50 +0000 [error]: #0 unexpected error error_class=OpenSearch::UnsupportedProductError error="The client is not supported for the provided version and distribution combination."
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:78:in 'verify_version_and_distribution'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:74:in 'verify_open_search'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:47:in 'method_missing'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-api-1.0.0/lib/opensearch/api/actions/info.rb:45:in 'info'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:518:in 'detect_os_major_version'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:510:in 'block in handle_last_seen_os_major_version'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/opensearch_index_template.rb:70:in 'retry_operate'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:507:in 'handle_last_seen_os_major_version'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:383:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:132:in 'add_match'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in 'block in configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in 'each'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in 'run_configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:719:in 'block in run_worker'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in 'main_process'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in 'run_worker'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in '<top (required)>'
2022-07-17 10:24:50 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in 'require'
2022-07-17 10:24:50 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in 'require'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in '<top (required)>'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in 'load'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in '

'

Expected Behavior or What you need to ask

The plugin queries the base URL for version/distribution check, then changes to the specified path when posting write operations.

Using Fluentd and OpenSearch plugin versions

  • OS version
    5.4.181-99.354.amzn2.x86_64
  • Bare Metal or within Docker or Kubernetes or others?
    fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-opensearch-1.0
  • Fluentd v1.0 or later
    • paste result of fluentd --version or td-agent --version
      fluentd 1.14.6
  • OpenSearch plugin version
    • paste boot log of fluentd or td-agent
    • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

*** LOCAL GEMS ***

addressable (2.8.0)
aws-eventstream (1.2.0)
aws-partitions (1.574.0)
aws-sdk-core (3.130.0)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (2.2.24, default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
domain_name (0.5.20190701)
etc (default: 1.1.0)
excon (0.92.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
ffi (1.15.5)
ffi-compiler (1.0.1)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.14)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-json-in-json-2 (1.0.2)
fluent-plugin-kubernetes_metadata_filter (2.9.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.3)
fluent-plugin-parser-cri (0.1.1)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-systemd (1.0.5)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http (4.4.1)
http-accept (1.7.0)
http-cookie (1.0.4)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.1)
json (default: 2.3.0)
jsonpath (1.1.0)
kubeclient (4.9.3)
logger (default: 1.4.2)
lru_redux (1.1.0)
matrix (default: 0.2.0)
mime-types (3.4.1)
mime-types-data (3.2022.0105)
msgpack (1.5.1)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
netrc (0.11.0)
observer (default: 0.1.0)
oj (3.11.0)
open3 (default: 0.1.0)
opensearch-api (1.0.0)
opensearch-ruby (1.0.0)
opensearch-transport (1.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
prime (default: 0.1.1)
prometheus-client (4.0.0)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.6)
racc (default: 1.4.16)
rake (13.0.6)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
recursive-open-struct (1.1.3)
reline (default: 0.1.5)
rest-client (2.1.0)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.2.5)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
systemd-journal (1.4.2)
timeout (default: 0.1.0)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
unf (0.1.4)
unf_ext (0.0.8.1)
uri (default: 0.10.0)
webrick (1.7.0, default: 1.6.1)
yajl-ruby (1.4.2)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

  • OpenSearch version (optional)
    1.3.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant