event.duration incorrectly reported

[ipninichuck]

Running RockNSM 2.4.2
Modified Logstash output to send data to Elastic Cloud
ECS pipeline

While running initial tests using the ECS pipeline I found event.duration reporting time scales that made no sense and did not match results obtained with the non-ECS pipeline. The same version of RockNSM was being used in both instances. The old fields show proper decimal values showing expected durations for connections and dns requests. While the ECS acquired values are being given in hours, days and years. No changes to the filters have been made in logstash. I am wondering if this is a general issue or something I am only encountering before attempting to fix the problem. I have provided screenshots comparing the duration fields obtained from both the original and followed by the ECS. I encountered the issue while monitoring data of web activity on my laptop to verify that my pipeline was functioning properly.

[neu5ron]

this could be a kibana index pattern setting of how it displays/shows and converts the value.
Can you show me what it is set as for you?
Once in Kibana, do following:
Management (tab) > Index Patterns > Select the index Pattern from the screenshots you posted > In the filter bar type in "event.duration" > select "Edit" (the pencil icon on the far right) > then post a screenshot of that screen.

I have a feeling this is related to event.duration being (or had at one time atleast) discussed as being nanoseconds - we convert seconds (how Zeek is recorded) into nanoseconds. Thus if Kibana index pattern is not set to format on nanoseconds then this could explain the issues.

[neu5ron]

^ @ipninichuck ping

[ipninichuck]

You were correct. The index pattern was set for seconds not nanoseconds. Once I made this change the values fit expected ranges.

[dhminch]

In my experience on 2.5.1-2004, I'm seeing event.duration values that should have an index value set to microseconds, not seconds (what is set by default on RockNSM deployment) and also not nanoseconds as mentioned above.

I did a sample SSH connection to a system and ran sleep 60, so the duration should be roughly 60 seconds. Here's what I see in Kibana before making any changes:

However, that's incorrect and the raw Zeek log confirms the correct value of approximately 60 seconds.

conn.log entry:
uid duration
CJOHkyFryLTqmMlje 63.267544

The JSON shown for this event shows a event.duration raw value of 63267543.79272461. I believe this occurs due to /etc/logstash/conf.d/logstash-599-filter-zeek-hook.conf which does event.set('[event][duration]', event_duration * 1000000) which would convert Zeek seconds into microseconds, even though the comment says its converting milliseconds to nanoseconds.

The fix is easy once I realized what was going on. I switched the event.duration index from Seconds to Microseconds and got the expected results.

I'm too new to RockNSM to understand why the Zeek seconds are being converted by Logstash in the first place, or why the index is set to the wrong time resolution, but I wanted to provide additional feedback to the devs and other users that are trying to make sense of the duration field. Thanks!