Cosmos Full Text Search support #35868

maumar · 2025-03-29T02:02:05Z

Adding model building API to configure property as full-text search enabled, as well as setup the index for it,
Adding model validation (e.g. FTS index not matching FTS property),
Adding EF.Functions stubs and translations for FullTextContains, FullTextContainsAll, FullTextContainsAny, FullTextScore and RRF (for hybrid),
Adding logic in SelectExpression to produce ORDER BY RANK when necessary,
Adding validation when attempting to mix with ORDER BY RANK with regular ORDER BY,
Rewrite OFFSET/LIMIT from parameter to constant when ORDER BY RANK is present.

outstanding work:

support for FTS Container building using Azure.ResourceManager.CosmosDb (currently blocked)
support for Owned types (adjust model in tests and fix paths) - this also needs to happen for vector search
add model building support for default language,
add more tests for hybrid search,
add more model validation tests for invalid scenarios (index on multiple columns, FTS on non-string etc)
add xml docs,
clean up exception messages and put them in a resource file,

maumar · 2025-03-29T02:08:32Z

src/EFCore.Cosmos/Query/Internal/CosmosQueryableMethodTranslatingExpressionVisitor.cs

@@ -927,7 +927,19 @@ protected override ShapedQueryExpression TranslateCast(ShapedQueryExpression sou

        if (TranslateLambdaExpression(source, keySelector) is SqlExpression translation)
        {
-            ((SelectExpression)source.QueryExpression).ApplyOrdering(new OrderingExpression(translation, ascending));
+            if (translation is SqlFunctionExpression { IsScoringFunction: true })


this is a bit magical (using ORDER BY vs ORDER BY RANK based on the type of argument passed to the OrderBy linq method). Initially I had an separate OrderByRank method (on CosmosQueryablEextension), so users could be explicit about wanting to order by scoring function, but hit all sorts of issues in the nav expansion - we need process the navigations inside the scoring function and/or push includes via pending selector/orderby, but Nav expansion doesn't know about the new OrderByRank method and there is no good way to extend it :(

Yep, adding a queryable operator to EF is kinda impossible because of our nav expansion design, I've ran into this several times in the past.

Will review when this is out of draft, we'll see what makes sense here...

maumar · 2025-03-29T02:11:48Z

src/EFCore.Cosmos/Query/Internal/Expressions/SqlFunctionExpression.cs

+    {
+    }
+
+    internal SqlFunctionExpression(


will add to SqlExpressionFactory later

maumar · 2025-03-29T02:13:39Z

src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs

@@ -276,9 +298,26 @@ private static async Task<bool> CreateContainerIfNotExistsOnceAsync(
            containerProperties.VectorEmbeddingPolicy = new VectorEmbeddingPolicy(embeddings);
        }

-        if (vectorIndexes.Any())
+        if (vectorIndexes.Any()/* || fullTextIndexes.Any()*/)


will fix and clean-up this

test/EFCore.Cosmos.FunctionalTests/FullTextSearchCosmosTest.cs

Copilot

Pull Request Overview

This PR adds support for full-text search in Cosmos by introducing new full-text index types, translators, model validations, and extensions.

Introduces full-text indexing support in container creation and client wrappers.
Adds a new CosmosFullTextSearchTranslator and corresponding DbFunctions extensions.
Updates model validation and Fluent API extensions to include full-text search configuration.

Reviewed Changes

Copilot reviewed 23 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs	Adds support for full-text indexing in container creation, but the new full-text index assignment may override vector index settings.
src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs	Modifies method checking logic that may prematurely reject valid method calls by changing && to
src/EFCore.Cosmos/Query/Internal/Translators/CosmosFullTextSearchTranslator.cs	Implements a new translator for full-text search functions.
src/EFCore.Cosmos/Query/Internal/Expressions/SqlFunctionExpression.cs	Introduces an extra constructor parameter to flag scoring functions.
src/EFCore.Cosmos/Query/Internal/Expressions/SelectExpression.cs	Adds a RankOrdering property and methods to apply rank-based ordering.
src/EFCore.Cosmos/Query/Internal/CosmosQueryableMethodTranslatingExpressionVisitor.cs	Adjusts ordering logic to support scoring functions and rank ordering.
src/EFCore.Cosmos/Query/Internal/CosmosQuerySqlGenerator.cs	Updates SQL generation to include rank ordering handling.
src/EFCore.Cosmos/Metadata/Internal/CosmosAnnotationNames.cs	Defines new annotations for full-text search configuration.
src/EFCore.Cosmos/Infrastructure/Internal/CosmosModelValidator.cs	Validates full-text index configuration, ensuring only single-property indexes with language annotation are defined.
src/EFCore.Cosmos/Extensions/CosmosPropertyExtensions.cs, CosmosPropertyBuilderExtensions.cs, CosmosIndexExtensions.cs, CosmosIndexBuilderExtensions.cs, CosmosDbFunctionsExtensions.cs	Adds Fluent API and extension methods for configuring full-text search on properties and indexes.

Files not reviewed (3)

Directory.Packages.props: Language not supported
src/EFCore.Cosmos/EFCore.Cosmos.csproj: Language not supported
src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported

Comments suppressed due to low confidence (2)

src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs:33

Changing the logical operator from '&&' to '||' alters the intended logic and causes the translator to return null for methods declared on CosmosDbFunctionsExtensions that are not named 'VectorDistance'. Revisit the condition to ensure it only returns null when both conditions are unmet.

if (method.DeclaringType != typeof(CosmosDbFunctionsExtensions) || method.Name != nameof(CosmosDbFunctionsExtensions.VectorDistance))

src/EFCore.Cosmos/Extensions/CosmosPropertyExtensions.cs:137

[nitpick] The Experimental attribute for full-text search methods is using 'CosmosVectorSearchExperimental' instead of the more appropriate 'CosmosFullTextSearchExperimental'. This may cause confusion regarding the feature's experiment status.

[Experimental(EFDiagnostics.CosmosVectorSearchExperimental)]

src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs

SamMonoRT · 2025-03-31T15:07:25Z

src/EFCore.Cosmos/Extensions/CosmosDbFunctionsExtensions.cs

@@ -52,6 +52,41 @@ public static T CoalesceUndefined<T>(
        T expression2)
        => throw new InvalidOperationException(CoreStrings.FunctionOnClient(nameof(CoalesceUndefined)));

+    /// <summary>
+    /// TODO
+    /// </summary>


leverage CoPilot to fill these in, and then review quickly

SamMonoRT · 2025-03-31T15:08:20Z

src/EFCore.Cosmos/EFCore.Cosmos.csproj

@@ -12,6 +12,7 @@
    <NoWarn>$(NoWarn);EF9101</NoWarn> <!-- Metrics is experimental -->
    <NoWarn>$(NoWarn);EF9102</NoWarn> <!-- Paging is experimental -->
    <NoWarn>$(NoWarn);EF9103</NoWarn> <!-- Vector search is experimental -->


Is this still experimental - I remember a discussion to update these, or am I mis-remembering?

Yeah, you're right - I think it's no longer experimental. Open an issue for us to make this non-experimental for 10?

maumar · 2025-04-01T00:04:07Z

src/EFCore.Cosmos/Query/Internal/Expressions/RankOrderingExpression.cs

+///     any release. You should only use it directly in your code with extreme caution and knowing that
+///     doing so can result in application failures when updating to a new Entity Framework Core release.
+/// </summary>
+public class RankOrderingExpression(SqlExpression expression)


We could use OrderingExpression and set IsAscending to true, cause we don't generate ASC for it on Cosmos, just DESC for descending. But it feels a bit hacky. Upside is less public surface. Thoughts?

- Adding model building API to configure property as full-text search enabled, as well as setup the index for it, - Adding model validation (e.g. FTS index not matching FTS property), - Adding EF.Functions stubs and translations for FullTextContains, FullTextContainsAll, FullTextContainsAny, FullTextScore and RRF (for hybrid), - Adding logic in SelectExpression to produce ORDER BY RANK when necessary, - Adding validation when attempting to mix with ORDER BY RANK with regular ORDER BY, - Rewrite OFFSET/LIMIT from parameter to constant when ORDER BY RANK is present. outstanding work: - support for FTS Container building using Azure.ResourceManager.CosmosDb (currently blocked) - support for Owned types (adjust model in tests and fix paths) - this also needs to happen for vector search - add model building support for default language, - add more tests for hybrid search, - add more model validation tests for invalid scenarios (index on multiple columns, FTS on non-string etc) - add xml docs, - clean up exception messages and put them in a resource file, Fixes #35476 Fixes #35853 Fixes #35867

roji · 2025-04-01T07:26:39Z

@maumar just to note that I'm holding back from reviewing since this is still in draft (you seemed to maybe want people to take a look in yesterday's meeting). Let me know if you want a review etc.

maumar commented Mar 29, 2025

View reviewed changes

test/EFCore.Cosmos.FunctionalTests/FullTextSearchCosmosTest.cs Outdated Show resolved Hide resolved

maumar requested a review from Copilot March 29, 2025 02:22

Copilot AI reviewed Mar 29, 2025

View reviewed changes

src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs Outdated Show resolved Hide resolved

SamMonoRT reviewed Mar 31, 2025

View reviewed changes

maumar force-pushed the cosmos_fts_take_omega branch 2 times, most recently from 0f39e44 to a836b63 Compare March 31, 2025 23:41

maumar commented Apr 1, 2025

View reviewed changes

maumar force-pushed the cosmos_fts_take_omega branch from a836b63 to 898cc5f Compare April 1, 2025 00:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cosmos Full Text Search support #35868

Cosmos Full Text Search support #35868

maumar commented Mar 29, 2025 •

edited

Loading

maumar Mar 29, 2025 •

edited

Loading

roji Mar 31, 2025

maumar Mar 29, 2025

maumar Mar 29, 2025 •

edited

Loading

Copilot AI left a comment

SamMonoRT Mar 31, 2025

SamMonoRT Mar 31, 2025

roji Apr 1, 2025

maumar Apr 1, 2025 •

edited

Loading

roji commented Apr 1, 2025

Cosmos Full Text Search support #35868

Are you sure you want to change the base?

Cosmos Full Text Search support #35868

Conversation

maumar commented Mar 29, 2025 • edited Loading

maumar Mar 29, 2025 • edited Loading

Choose a reason for hiding this comment

roji Mar 31, 2025

Choose a reason for hiding this comment

maumar Mar 29, 2025

Choose a reason for hiding this comment

maumar Mar 29, 2025 • edited Loading

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

SamMonoRT Mar 31, 2025

Choose a reason for hiding this comment

SamMonoRT Mar 31, 2025

Choose a reason for hiding this comment

roji Apr 1, 2025

Choose a reason for hiding this comment

maumar Apr 1, 2025 • edited Loading

Choose a reason for hiding this comment

roji commented Apr 1, 2025

maumar commented Mar 29, 2025 •

edited

Loading

maumar Mar 29, 2025 •

edited

Loading

maumar Mar 29, 2025 •

edited

Loading

maumar Apr 1, 2025 •

edited

Loading