Flush hits when no activity from stdin

[Knyffen]

Description:

The script only processes the logs after len(hits) >= options.recorder_max_payload_size * len(Recorders.recorders) valid hits are detected. With the default settings, this is 200 hits or if EOF is hit.

When the input file is stdin (e.g. when using syslog), and if used on a low traffic site, this makes it seem like nothing is happening, causing confusion (according to #258 and my own experiences) since the payload limit isn't hit and we never reach EOF.

I have added a check where if using stdin and the buffer is empty, then the current hits are sent immediately, even if we haven't reached the max payload size.

Caveats/potential issues/things to discuss:

• Admittedly, I haven't researched the select syscall, and the detection is based on this. But I can confirm on my system using syslog-ng (sorry, but I don't have any clue on how to formally test syscalls).
• This only works for Unix systems.
• I wonder if there theoretically exists a potential setting where given a very high traffic site, if the script sleeps for one second then the entire stdin buffer could potentially be filled?
• I have not added an option for toggling this feature, as it ideally should hopefully never be preferable to turn it off.

Review

• Functional review done
• Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• Security review done
• Wording review done
• Code review done
• Tests were added if useful/possible
• Reviewed for breaking changes
• Developer changelog updated if needed
• Documentation added if needed
• Existing documentation updated if needed

[Knyffen]

Just to comment on the test that failed with python 3.5:
Regarding the failing test for python 3.5, honestly, I don't know why the test should suddenly fail, since it has file_ = 'logs/netscaler.log' and my new code only triggers if file == sys.stdin.