Maximum size fields and Index Fields when doing walks

[eperry]

While doing very liberal SNMP Walks on a server I came across this issue

Field Names can get exceedingly long and SNMP Walks can create a Huge amount of field names

[2018-11-04T18:58:32,704][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"snmp-2018.11.04", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x22bdd815>], :response=>{"index"=>{"_index"=>"snmp-2018.11.04", "_type"=>"logs", "_id"=>"lcAp4WYBIR0n9FiBt-hA", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of mapping depth [20] in index [snmp-2018.11.04] has been exceeded due to object field [iso.org.dod.internet.mgmt.mib-2.dismanEventMIB.dismanEventMIBObjects.mteEvent.mteEventTable.mteEventEntry.mteEventEntryStatus.6.95.115.110.109.112.100.95.109.116.101.84.114]"}}}}

[2018-11-04T18:47:28,169][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"snmp-2018.11.04", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x78eb628b>], :response=>{"index"=>{"_index"=>"snmp-2018.11.04", "_type"=>"logs", "_id"=>"dpof4WYBf0IKXA2OUQXI", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [6000] in index [snmp-2018.11.04] has been exceeded"}}}}

I thought I would report it, though I am sure it will need some thought, on how to deal with very long mibs

I was thinking the following

• Try to perform some kind of Grouping
• Too Many fields.
  I was thinking about Request: paging/limit results sets logstash-input-snmp#24 where I could for "single" documents add an option to convert from.

  {
     "OID": oidvalue
  }

to something like

  {
      oid_name: "OID"
      value: oid_value
  }

Not sure how you would do it with one large document

[geekpete]

I wonder if Flattened field type could be used here for some benefit?
https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html