Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for getting the right log entries via Thruk #14

Open
GuusH opened this issue May 20, 2015 · 0 comments
Open

Fix for getting the right log entries via Thruk #14

GuusH opened this issue May 20, 2015 · 0 comments

Comments

@GuusH
Copy link
Contributor

GuusH commented May 20, 2015

Hi,

I've been having issues with the reporting functionality in Thruk. See shinken-solutions/shinken#1566 for my original bug report. After digging I found it's actually a small problem in this module that's causing the problem. Naparuba's statement that getting logs via the livestatus module will blow up if you have a very large number of logs is very true, however currently this module gets all logs instead of the relevant logs when using the reporting functionality in Thruk, which makes that more or less useless.

Details:
This is the relevant part of the livestatus command that's being issued by Thruk for an "SLA Hostgroup" report:

Filter: class = 2
Filter: message ~~ TIMEPERIOD TRANSITION: 24x7
Filter: type = TIMEPERIOD TRANSITION
Filter: message ~~ 24x7
And: 2
Or: 2
Or: 3

The make_mongo_filter code translates that into (nice piece of coding btw, changing the postfix livestatus protocol into the prefix mongo filter!):

[{ '$or' : 
[{ '$or' : 
[{ '$and' : 
[{ 'time' : { '$exists' : True } }, 
{ 'type' : 'TIMEPERIOD TRANSITION' }]
}, 
{ 'time' : { '$exists' : True } }
] 
},

The 2 'message' filter attributes are gone and replaced by these dummy 'time' entries. These are always true and cause the filter to get all logs. This causes my "report from hell" to request 7 GB of logs and then churns the broker to death in about 5 minutes by eating all memory and eating 100% cpu. And that's just our test environment: our production environment has 10 times as much log entries...
So I added the 'message' attribute to the 'good_attributes' and 'string_attributes' lists in the make_mongo_filter function, et voila, now this is the result:

[{ '$or' : 
[{ '$or' : 
[{ '$and' : 
[{ 'message' : { '$regex' : '24x7', '$options' : 'i' } }, 
{ 'type' : 'TIMEPERIOD TRANSITION' }] 
}, 
{ 'message' : { '$regex' : 'TIMEPERIOD TRANSITION: 24x7', '$options' : 'i' } }
] 
},

That's more like it! After this fix the report completes in about a minute and does not consume huge amounts of memory or cpu.
Here's the pull request: #13

I hope you'll merge it.

PS: there's something wrong with the Travis integration, since that fails on PRs for this module on stuff that wasn't even touched. See also my other PR (#11) that updates this module to be MongoDB 3 compatible :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant