+mdb atlas vectordb [clean_final]

[ranfysvalle02]

Why are these changes needed?

MongoDB has been ranked as the best vector database(https://www.mongodb.com/blog/post/atlas-vector-search-commands-highest-developer-nps-retool-state-ai-2023-survey) in the Retool AI report, so it is quite important to add MongoDB vector search as an option for Autogen RAG.

You can easily start the MongoDB vector search on a free tier M0 MongoDB Atlas cluster. Free tier cluster provides the full functionality of the MongoDB vector search. https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/

But why is MongoDB such a standout? Well, there are a few key reasons.

MongoDB Atlas integrates smoothly with existing databases. For organizations already using MongoDB, this means a seamless expansion into the vector storage—no major system overhauls required!
MongoDB Atlas is built to handle operational heavy-lifting. It excels when serving large-scale, mission-critical applications, offering robustness and reliability where it counts.
MongoDB's flexibility in handling a variety of data types and structures makes it perfectly suited to the complexity of vector embeddings.

As such, implementing MongoDB as a Retrieval Agent can unlock new potential in your AI applications, bringing the full power of vector storage to bear.

Related issue number: 711

Closes #711
Closes #2996

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[ranfysvalle02]

Sorry @thinkall @Hk669 -- I know I have been a bit annoying, but I think this time we got it 👍

[thinkall]

Hi @ranfysvalle02 , thank you. I can see the issues of notebook have not been addressed. See comments here: #2996

[ranfysvalle02]

The simple change of

    assert collection.name == collection_name
    assert collection.collection_name == collection_name

Highlighted that I need to create a "wrapper" class around the MongoDB collection, similar to what pgvector did.

class Collection:
    """
    A Collection object for PGVector.

    Attributes:
        client: The PGVector client.
        collection_name (str): The name of the collection. Default is "documents".
        embedding_function (Callable): The embedding function used to generate the vector representation.
            Default is None. SentenceTransformer("all-MiniLM-L6-v2").encode will be used when None.
            Models can be chosen from:
            https://huggingface.co/models?library=sentence-transformers
        metadata (Optional[dict]): The metadata of the collection.
        get_or_create (Optional): The flag indicating whether to get or create the collection.
    """

but for MongoDB. Will be working on this @Hk669

[ranfysvalle02]

@thinkall - Can you please tell me how to 'fix' the notebook? Or perhaps have it as a 'suggested commit'? I'll be addressing the notebook and any final touches later today.

[thinkall]

@thinkall - Can you please tell me how to 'fix' the notebook? Or perhaps have it as a 'suggested commit'? I'll be addressing the notebook and any final touches later today.

What about run it successfully in your local env and remove only the sensitive info? A new user should be able to run it by fill in the missed message, which should only be the config_list.

So, the connect string of mongodb should not be empty, the one I suggested in your last PR worked for me. Does it work for you? The one you previously used didn't work for me and was not connecting to the docker container.

The output of the chat in the last cell is not correct. Could you please check my previous comments and the pgvector notebook example?

[thinkall]

The simple change of

    assert collection.name == collection_name
    assert collection.collection_name == collection_name

Highlighted that I need to create a "wrapper" class around the MongoDB collection, similar to what pgvector did.

class Collection:
    """
    A Collection object for PGVector.

    Attributes:
        client: The PGVector client.
        collection_name (str): The name of the collection. Default is "documents".
        embedding_function (Callable): The embedding function used to generate the vector representation.
            Default is None. SentenceTransformer("all-MiniLM-L6-v2").encode will be used when None.
            Models can be chosen from:
            https://huggingface.co/models?library=sentence-transformers
        metadata (Optional[dict]): The metadata of the collection.
        get_or_create (Optional): The flag indicating whether to get or create the collection.
    """

but for MongoDB. Will be working on this @Hk669

It's OK, no need to wrap a Collection class for mongodb. PGVector doesn't have a collection class, that's why we have it in AutoGen vector implementation. The key for AutoGen vectordb is to follow the protocols in base.py.

[ranfysvalle02]

I see what you mean @thinkall !

I found the issue with the notebook and notebook output!

 # MongoDB Atlas database
        },
        "get_or_create": False,  # set to False if you don't want to reuse an existing collection
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

vs

 # MongoDB Atlas database
        },
        "get_or_create": True, 
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

We are close!!!

I'll push the fix/code later today

[thinkall]

I see what you mean @thinkall !

I found the issue with the notebook and notebook output!

 # MongoDB Atlas database
        },
        "get_or_create": False,  # set to False if you don't want to reuse an existing collection
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

vs

 # MongoDB Atlas database
        },
        "get_or_create": True, 
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

We are close!!!

I'll push the fix/code later today

No errors here. I've fixed this and made a commit.

[thinkall]

The retrieve_docs is not working as expected. No doc is returned. Either the query pipeline or the atlas local env is not functional.

[ranfysvalle02]

On it! Thank you so much for helping debug! There is something wrong with the implementation I believe -- will debug this shortly.

[ranfysvalle02]

@thinkall - It is an implementation bug! its an issue with the index_name :) WIl fix shortly.

[ranfysvalle02]

@thinkall finally tracked this down --- its all about the index! the create_collection method

def create_collection(collection_name: str,
                        overwrite: bool = False,
                        get_or_create: bool = True) -> Any

does not use 'index_name' or 'similarity' -- which I had added. Working on a fix!

[ranfysvalle02]

@thinkall - I finally got it to run, but I have to add a strange programmatic arbitrary delay for things to work. I am working on a more elegant solution. After ~15seconds it works. Anything 5 seconds or less fails.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 79.68750% with 39 lines in your changes missing coverage. Please review.

Project coverage is 19.88%. Comparing base (03259b2) to head (95e2f79).
Report is 10 commits behind head on main.

Files	Patch %	Lines
autogen/agentchat/contrib/vectordb/mongodb.py	80.85%	22 Missing and 14 partials ⚠️
autogen/agentchat/contrib/vectordb/base.py	25.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3000       +/-   ##
===========================================
- Coverage   32.45%   19.88%   -12.58%     
===========================================
  Files          93       95        +2     
  Lines       10109    10426      +317     
  Branches     2172     2388      +216     
===========================================
- Hits         3281     2073     -1208     
- Misses       6544     8214     +1670     
+ Partials      284      139      -145     

Flag	Coverage Δ
unittests	19.83% <79.68%> (-12.63%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hk669]

@thinkall - I finally got it to run, but I have to add a strange programmatic arbitrary delay for things to work. I am working on a more elegant solution. After ~15seconds it works. Anything 5 seconds or less fails.

https://github.com/microsoft/autogen/actions/runs/9629801423/job/26560601555?pr=3000 -> can you look into the tests that are failing. thanks @ranfysvalle02

[thinkall]

Hi @ranfysvalle02 , there is still some issues with the notebook and test, could you please help investigate? Btw, please use pre-commit run --all-files to make sure the format is good. Thank you so much!

[thinkall]

Changing this will lead to changes to all the other dbs. It's better to update the keys in the results of mongodb wrapper.

[thinkall]

With this setting, I always get empty returns like below:

2024-06-30 21:18:02,516 - autogen.agentchat.contrib.vectordb.mongodb - INFO - Search index vector_index created successfully.
VectorDB returns doc_ids:  [[]]
No more context, will terminate.

Are you sure the free version of mongodb works?

[thinkall]

Can you run the notebook with a mongodb instance deployed with the docker-compose.yml provided at the beginning and share the actual connection_string you used?

[thinkall]

Can you avoid printing out all the embeddings?

[thinkall]

A default value should be given.

[thinkall]

Hi @ranfysvalle02 , I've fixed the mongodb test errors. It looks like the distance_threshold is not working. I'm not sure if it's not working for free mongodb or there is some issue with the code.

[thinkall]

Please add a test with distance_threshold set. It's not working in my local env.

[thinkall]

Don't return embedding by default.

[thinkall]

Suggested change

	# if overwrite is False and get_or_create is False, raise a ValueError
	if not overwrite and not get_or_create:
	raise ValueError("If overwrite is False, get_or_create must be True.")
	collection_names = self.db.list_collection_names()
	if collection_name not in collection_names:
	# Create a new collection
	return self.db.create_collection(collection_name)
	if overwrite:
	self.db.drop_collection(collection_name)
	if get_or_create:
	# The collection already exists, return it.
	return self.db[collection_name]
	else:
	# get_or_create is False and the collection already exists, raise an error.
	raise ValueError(f"Collection {collection_name} already exists.")
	# if overwrite is False and get_or_create is False, raise a ValueError
	if overwrite:
	self.db.drop_collection(collection_name)
	collection_names = self.db.list_collection_names()
	if collection_name not in collection_names:
	# Create a new collection
	return self.db.create_collection(collection_name)
	if get_or_create:
	# The collection already exists, return it.
	return self.db[collection_name]
	else:
	# get_or_create is False and the collection already exists, raise an error.
	raise ValueError(f"Collection {collection_name} already exists.")