Mongo can handle larger databases now

[Halk0]

Made changes to mongodb backend logic so that it no longer creates multiple queries to backend. Now mongodb backend constructs one filter from the request and queries data from mongodb just once. Previously multiple queries were made and filtering was done based on document IDs returned from those queries, this was bad practice since mongodb would have to go trough all documents multiple times filtering them using a huge list of IDs returned from previous queries. Using just one filter allows mongodb to handle the single query internally without having to match document IDs to a sometimes very large array of IDs.

Closes #173

[CLAassistant]

All committers have signed the CLA.

[Halk0]

This pull request doesn't contain a test, since triggering the bug mentioned in Issue #173
requires generating and writing huge amounts of documents to mongodb, that would make
testing much slower and require alot of resources. If this test is seen as a necessary addition
I can write one.

[chisholm]

Here is a sample of an alternative approach I came up with which uses windowing (this is for finding max versions):

[
    {
        "$setWindowFields": {
            "partitionBy": "$id",
            "output": {
                "_max_version": {
                    "$max": "$_manifest.version"
                }
            }
        }
    },
    {
        "$match": {
            "$expr": {
                "$eq": [
                    "$_manifest.version",
                    "$_max_version"
                ]
            }
        }
    },
    {
        "$unset": [
            "_max_version"
        ]
    }
]

I don't know how it would compare in terms of efficiency. Perhaps it is more efficient since it doesn't require sorting all of the documents first?

[scottpendlebury]

I've been facing this exact same problem recently as I've had to account for a worst case configuration in Sentinel of having a connector set to pull all indicators once a minute, I've tested your change and the difference is like night and day. I'm able to pull ~11k indicators in 1.3 seconds now. Many thanks for this @Halk0

[Halk0]

@chisholm thats nice, I made some changes based your approach!
I also noticed that in the first commit loop that was on lines 95-115 was completely
unnecessary, since lines below that (116-117) did the exact same thing, but it was much
more simple, and doesn't require loops