Issue #24 Limit size of Document.

[eperry]

I took a first shot at writing a "Mode" parameter which specifies how to create Logstash Events. For Issue #24

Batch = the current way, where all OID's are in a Single document. (Default)

Single = Each OID will be put in to its one document.

Single mode later on enhanced to provide additional information about about each OID, like description and so on like SNMPINFO provides.

[colinsurprenant]

thanks for your contribution @eperry - so using the single mode, each top-level OID would be in its own document? what if a sub-tree of one of the OIDs is where its gets "big"?

The each_pair iteration will only iterate over the top level elements of the hash.

For example

irb(main):019:0> {:a=>1, :b=>2, :c=>{:d=>3, :e=>4}}.each_pair{|k, v| puts("#{k} => #{v}")}
a => 1
b => 2
c => {:d=>3, :e=>4}

So using this example, events would be created for a, b and c.

Is that you intention here?

[eperry]

Yes that is correct, each "attribute" of the object event, would become a seperate object.

The sudo code you wrote would be accurate in its description ( as I read it).

The intent, is that

Batch: is all information is included in event becomes one document ( which could be XX MB )

Single: is only one OID in a single Document. (Future possible OID description,....)

When I get to work tomorrow, I will provide a couple example of the documents created, as I put the code in to "QA" already

[colinsurprenant]

@eperry My hesitation is about a walk operation result where, for example, it could contain 2 OIDs: the first one with a scalar value and the second with a whole subtree of data which could be of unknown size. Using this example, your suggested strategy would yield two events; the first one with the OID and the scalar value and a second one with a whole subtree of data.

So I am wondering if this approach will work practically?

[eperry]

Well abstractly your correct but that is not how the "event" field is being created within the SNMP Document. It is a flat list of key:values the value is either a string or in. There are no subtree's

{
type: snmp
@timestamp: 0000000
"MIB.OID":   "stringvalue"
"MIB2.oid2": intvalue
"mib3.oid":   "valueX"
}

so this above document would be split into 3 different documents

{
type: snmp
@timestamp: 0000000
"MIB.OID":   "stringvalue"
}

{
type: snmp
@timestamp: 0000000
"MIB2.oid2": intvalue
}
{
type: snmp
@timestamp: 0000000
"mib3.oid":   "valueX"
}

The reason to do this. So if you wanted to list all the installed RPM, Netstat or even all the running processes.
A walk of the process list could have hundreds of entries which could make a very big document,

snmpwalk -v2c  -c public myserver  HOST-RESOURCES-MIB

splitting this into separate documents, prevents "HUGE" documents from being created. Some of the snmp walks I am creating have 20k different metrics, and while I probably don't want that many. even a document with 5000 fields is a lot.

[colinsurprenant]

@eperry damn - disregard my subtree hallucination, indeed everything is flattened.
Ok, let's move forward with this idea, it's an easy addition and can provide flexibility on how you want your data events created.

Here's what I suggest: I would like this refactored a bit and tests added, depending on what you prefer we can either discuss here what I would like or I can create a new PR, copy your commit (so your authorship & history is preserved) and I can make some modifications on top of it.

Let me know what you prefer for this.
Thanks!

[edperry]

No worries, I had to think about it a little bit too. It is a good question.

I am not a great programmer (System Admin by trade) , so any help you can provide would be greatly appreciated to clean it up and be acceptable. I will be glad to do what is needed but I would need a little extra coaching to do it myself.