Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Add PostgresVectorStore Memory connector. #9324

Conversation

lossyrob
Copy link
Contributor

@lossyrob lossyrob commented Oct 18, 2024

This PR adds a PostgresVectorStore and related classes to Microsoft.SemanticKernel.Connectors.Postgres.

Motivation and Context

As part of the move to having memory connectors implement the new Microsoft.Extensions.VectorData.IVectorStore architecture (see https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0050-updated-vector-store-design.md), each memory connector needs to be updated with the new architecture. This PR tackles updating the existing Microsoft.SemanticKernel.Connectors.Postgres package to include this implementation. This will supercede the PostgresMemoryStore implementation.

Some high level comments about design:

  • PostgresVectorStore and PostgresVectorStoreRecordCollection get injected with an IPostgresVectorStoreDbClient. This abstracts the database communication and allows for unit tests to mock database interactions.
  • The PostgresVectorStoreDbClient gets passed in a NpgsqlDataSource from the user, which is used to manage connections to the database. The responsibility of connection pool lifecycle management is on the user.
  • The IPostgresVectorStoreDbClient is designed to accept and produce the storage model, which in this case is a Dictionary<string, object?> . This is the intermediate type that is mapped to by the IVectorStoreRecordMapper.
  • The PostgresVectorStoreDbClient also takes a IPostgresVectorStoreCollectionSqlBuilder, which generates SQL command information for interacting with the database. This abstracts the SQL queries related to each task, and allows for future expansion. This is particularly targeted at creating a AzureDBForPostgre vector store that will enable alternate vector implementations like DiskANN, while leveraging the same database client as the Postgres connector.
  •  The integration tests for the vector store utilize Docker.Net to bring up a pgvector/pgvector docker container, which test are run against.

Contribution Checklist

Work in progress, some methods are not implemented yet.
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Oct 18, 2024
@lossyrob
Copy link
Contributor Author

lossyrob commented Nov 1, 2024

@westey-m most recent comments addressed, thanks!

@Hanake0
Copy link

Hanake0 commented Nov 26, 2024

Hi @westey-m and @lossyrob,

Thanks for the work on this feature.
Sorry for the inconvenience if that is not the right place to ask, but do you guys know when will it be available in the Microsoft.SemanticKernel.Connectors.Postgres package?

We are currently using the legacy Memory Store but that was recently marked as legacy and the connector for Postgres is currently marked as in-development, but as i see here it is probably at-least ready for alpha testing.

Is it available for testing already in any channel or is any ETA available for it?

Again, sorry if that is not the right place to ask.

Thanks!

@westey-m
Copy link
Contributor

westey-m commented Nov 26, 2024

@Hanake0, no problem. It's been a bit busy lately with various deadlines, but I was meaning to get back to this again this week. With the start of the holiday period, giving an ETA will be difficult, but we'll certainly be progressing it as a priority.

Copy link
Member

@dmytrostruk dmytrostruk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small comments, but in general looks good to me! Thanks for your contribution!

@markwallace-microsoft markwallace-microsoft added this pull request to the merge queue Dec 16, 2024
Merged via the queue into microsoft:main with commit c7a371e Dec 16, 2024
17 checks passed
@lossyrob lossyrob deleted the feature/postgres-vector-store-dotnet branch January 2, 2025 17:50
@lossyrob
Copy link
Contributor Author

lossyrob commented Jan 2, 2025

@dmytrostruk @westey-m Thanks for getting this over the finish line! I was out on parental leave but am back and can take on any follow up work. Feel free to tag me in any related issues etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation kernel Issues or pull requests impacting the core kernel memory .NET Issue or Pull requests regarding .NET code
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

6 participants