Release 9.0

[flobernd]

Changelog

1. Summary

1.1 Request Method/API Changes

1.1.1 Synchronous Request APIs

Synchronous request APIs are no longer marked as obsolete. We received some feedback about this deprecation and decided to revert it.

1.1.2 Separate Type Arguments for Request/Response

It is now possible to specify separate type arguments for requests/responses when executing request methods:

var response = await client.SearchAsync<Person, JsonObject>(x => x
    .Query(x => x.Term(x => x.Field(x => x.FirstName).Value("Florian")))
);

var documents = response.Documents; // IReadOnlyCollection<JsonObject>

The regular APIs with merged type arguments are still available.

1.2 Improved Fluent API

The enhanced fluent API generation is likely the most notable change in the 9.0 client.

This section describes the main syntax constructs generated based on the type of the property in the corresponding object.

1.2.1 ICollection<E>

Note: This syntax already existed in 8.x.

new SearchRequestDescriptor<Person>()
    .Query(q => q
        .Bool(b => b
            .Must(new Query())                           // Scalar: Single element.
            .Must(new Query(), new Query())              // Scalar: Multiple elements (params).
            .Must(m => m.MatchAll())                     // Fluent: Single element.
            .Must(m => m.MatchAll(), m => m.MatchNone()) // Fluent: Multiple elements (params).
        )
    );

1.2.2 IDictionary<K, V>

The 9.0 client introduces full fluent API support for dictionary types.

new SearchRequestDescriptor<Person>()
    .Aggregations(new Dictionary<string, Aggregation>()) // Scalar.
    .Aggregations(aggs => aggs                           // Fluent: Nested.
        .Add("key", new MaxAggregation())                // Scalar: Key + Value.
        .Add("key", x => x.Max())                        // Fluent: Key + Value.
    )
    .AddAggregation("key", new MaxAggregation())         // Scalar.
    .AddAggregation("key", x => x.Max());                // Fluent.

Warning

The Add{Element} methods have different semantics compared to the standard setter methods.

Standard fluent setters set or replace a value.

In contrast, the new additive methods append new elements to the dictionary.

For dictionaries where the value type does not contain required properties that must be initialized, another syntax is generated that allows easy addition of new entries by just specifying the key:

// Dictionary<Name, Alias>()

new CreateIndexRequestDescriptor("index")
    // ... all previous overloads ...
    .Aliases(aliases => aliases // Fluent: Nested.
        .Add("key")             // Key only.
    )
    .Aliases("key")             // Key only: Single element.
    .Aliases("first", "second") // Key only: Multiple elements (params).

If the value type in the dictionary is a collection, additional params overloads are generated:

// Dictionary<Field, ICollection<CompletionContext>>

new CompletionSuggesterDescriptor<Person>()
    // ... all previous overloads ...
    .AddContext("key", 
        new CompletionContext{ Context = new Context("first") },
        new CompletionContext{ Context = new Context("second") }
    )
    .AddContext("key",
        x => x.Context(x => x.Category("first")),
        x => x.Context(x => x.Category("second"))
    );

1.2.3 ICollection<KeyValuePair<K, V>>

Elasticsearch often uses ICollection<KeyValuePair<K, V>> types for ordered dictionaries.

The 9.0 client abstracts this implementation detail by providing a fluent API that can be used exactly like the one for IDictionary<K, V> types:

new PutMappingRequestDescriptor<Person>("index")
    .DynamicTemplates(new List<KeyValuePair<string, DynamicTemplate>>()) // Scalar.
    .DynamicTemplates(x => x                                             // Fluent: Nested.
        .Add("key", new DynamicTemplate())                               // Scalar: Key + Value.
        .Add("key", x => x.Mapping(new TextProperty()))                  // Fluent: Key + Value.
    )
    .AddDynamicTemplate("key", new DynamicTemplate())                    // Scalar: Key + Value.
    .AddDynamicTemplate("key", x => x.Runtime(x => x.Format("123")));    // Fluent: Key + Value.

1.2.4 Union Types

Fluent syntax is now as well available for all auto-generated union- and variant-types.

// TermsQueryField : Union<ICollection<FieldValue>, TermsLookup>

new TermsQueryDescriptor()
    .Terms(x => x.Value("a", "b", "c"))                    // ICollection<FieldValue>
    .Terms(x => x.Lookup(x => x.Index("index").Id("id"))); // TermsLookup

1.3 Improved Descriptor Design

The 9.0 release features a completely overhauled descriptor design.

Descriptors now wrap the object representation. This brings several internal quality-of-life improvements as well as noticeable benefits to end-users.

1.3.1 Wrap

Use the wrap constructor to create a new descriptor for an existing object:

var request = new SearchRequest();

// Wrap.
var descriptor = new SearchRequestDescriptor(request);

All fluent methods of the descriptor will mutate the existing request passed to the wrap constructor.

Note

Descriptors are now implemented as struct instead of class, reducing allocation overhead as much as possible.

1.3.2 Unwrap / Inspect

Descriptor values can now be inspected by unwrapping the object using an implicit conversion operator:

var descriptor = new SearchRequestDescriptor();

// Unwrap.
SearchRequest request = descriptor;

Unwrapping does not allocate or copy.

1.3.3 Removal of Side Effects

In 8.x, execution of (most but not all) lambda actions passed to descriptors was deferred until the actual request was made. It was never clear to the user when, and how often an action would be executed.

In 9.0, descriptor actions are always executed immediately. This ensures no unforeseen side effects occur if the user-provided lambda action mutates external state (it is still recommended to exclusively use pure/invariant actions). Consequently, the effects of all changes performed by a descriptor method are immediately applied to the wrapped object.

1.4 Request Path Parameter Properties

In 8.x, request path parameters like Index, Id, etc. could only be set by calling the corresponding constructor of the request. Afterwards, there was no way to read or change the current value.

In the 9.0 client, all request path parameters are exposed as get/set properties, allowing for easy access:

// 8.x and 9.0
var request = new SearchRequest(Indices.All);

// 9.0
var request = new SearchRequest { Indices = Indices.All };
var indices = request.Indices;
request.Indices = "my_index";

1.5 Field Name Inference

The Field type and especially its implicit conversion operations allowed for null return values. This led to a poor developer experience, as the null-forgiveness operator (!) had to be used frequently without good reason.

This is no longer required in 9.0:

// 8.x
Field field = "field"!;

// 9.0
Field field = "field";

1.6 Uniform Date/Time/Duration Types

The encoding of date, time and duration values in Elasticsearch often varies depending on the context. In addition to string representations in ISO 8601 and RFC 3339 format, also Unix timestamps (in seconds, milliseconds, nanoseconds) or simply seconds, milliseconds, nanoseconds are frequently used.

In 8.x, most date/time values are mapped as DateTimeOffset, which is incorrect or at least unnecessary, since Elasticsearch works (almost) exclusively with UTC data. In addition, the various non-ISO/RFC representations were often not deserialized correctly.

9.0 now represents all date/time values uniformly as DateTime and also uses the native TimeSpan type for all durations.

Note

There are some places where the Elasticsearch custom date/time/duration types are continued to be used. This is always the case when the type has special semantics and/or offers functionality that goes beyond that of the native date/time/duration types (e.g. Duration, DateMath).

1.7 Improved Container Design

In 8.x, container types like Query or Aggregation had to be initialized using static factory methods.

// 8.x
var agg = Aggregation.Max(new MaxAggregation { Field = "my_field" });

This made it mandatory to assign the created container to a temporary variable if additional properties of the container (not the contained variant) needed to be set:

// 8.x
var agg = Aggregation.Max(new MaxAggregation { Field = "my_field" });
agg.Aggregations ??= new Dictionary<string, Aggregation>();
agg.Aggregations.Add("my_sub_agg", Aggregation.Terms(new TermsAggregation()));

Additionally, it was not possible to inspect the contained variant.

In 9.0, each possible container variant is represented as a regular property of the container. This allows for determining and inspecting the contained variant and initializing container properties in one go when using an object initializer:

// 9.0
var agg = new Aggregation
{
    Max = new MaxAggregation { Field = "my_field" },
    Aggregations = new Dictionary<string, Aggregation>
    {
        { "my_sub_agg", new Aggregation{ Terms = new TermsAggregation() } }
    }
};

Warning

A container can still only contain a single variant. Setting multiple variants at once is invalid.

Consecutive assignments of variant properties (e.g., first setting Max, then Min) will cause the previous variant to be replaced.

1.8 Sorting

Applying a sort order to a search request using the fluent API is now more convenient:

var search = new SearchRequestDescriptor<Person>()
    .Sort(
        x => x.Score(),
        x => x.Score(x => x.Order(SortOrder.Desc)),
        x => x.Field(x => x.FirstName),
        x => x.Field(x => x.Age, x => x.Order(SortOrder.Desc)),
        x => x.Field(x => x.Age, SortOrder.Desc)
        // 7.x syntax
        x => x.Field(x => x.Field(x => x.FirstName).Order(SortOrder.Desc))
    );

The improvements are even more evident when specifying a sort order for aggregations:

new SearchRequestDescriptor<Person>()
    .Aggregations(aggs => aggs
        .Add("my_terms", agg => agg
            .Terms(terms => terms
                // 8.x syntax.
                .Order(new List<KeyValuePair<Field, SortOrder>>
                {
                    new KeyValuePair<Field, SortOrder>("_key", SortOrder.Desc)
                })
                // 9.0 fluent syntax.
                .Order(x => x
                    .Add(x => x.Age, SortOrder.Asc)
                    .Add("_key", SortOrder.Desc)
                )
                // 9.0 fluent add syntax (valid for all dictionary-like values).
                .AddOrder("_key", SortOrder.Desc)
            )
        )
    );

1.9 Safer Object Creation

In version 9.0, users are better guided to correctly initialize objects and thus prevent invalid requests.

For this purpose, at least one constructor is now created that enforces the initialization of all required properties. Existing parameterless constructors or constructor variants that allow the creation of incomplete objects are preserved for backwards compatibility reasons, but are marked as obsolete.

For NET7+ TFMs, required properties are marked with the required keyword, and a non-deprecated parameterless constructor is unconditionally generated.

Note

Please note that the use of descriptors still provides the chance to create incomplete objects/requests, as descriptors do not enforce the initialization of all required properties for usability reasons.

1.9 Serialization

Serialization in version 9.0 has been completely overhauled, with a primary focus on robustness and performance. Additionally, initial milestones have been set for future support of native AOT.

In 9.0, round-trip serialization is now supported for all types (limited to all JSON serializable types).

var request = new SearchRequest{ /* ... */ };

var json = client.ElasticsearchClientSettings.RequestResponseSerializer.SerializeToString(
    request, 
    SerializationFormatting.Indented
);

var searchRequestBody = client.ElasticsearchClientSettings.RequestResponseSerializer.Deserialize<SearchRequest>(json)!;

Warning

Note that only the body is serialized for request types. Path- and query properties must be handled manually.

Note

It is important to use the RequestResponseSerializer when (de-)serializing client internal types. Direct use of JsonSerializer will not work.

2. Breaking Changes

This section contains an extensive list of breaking changes.

The 💥 icon indicates changes that are most likely breaking for users, whereas the ⚠️ icon indicates breaking changes that will not affect most users (e.g., changes in fluent descriptors that do not affect the lambda expression syntax).

2.1 💥

Container types now use regular properties for their variants instead of static factory methods (read more).

This change primarily affects the Query and Aggregation types.

// 8.x
new SearchRequest
{
    Query = Query.MatchAll(
        new MatchAllQuery
        {
        }
    )
};

// 9.0
new SearchRequest
{
    Query = new Query
    {
        MatchAll = new MatchAllQuery
        {
        }
    }
};

2.2 💥

Removed the generic version of some request descriptors for which the corresponding requests do not contain inferrable properties.

These descriptors were generated unintentionally.

When migrating, the generic type parameter must be removed from the type, e.g., AsyncSearchStatusRequestDescriptor<TDocument> should become just AsyncSearchStatusRequestDescriptor.

List of affected descriptors:

• AsyncQueryDeleteRequestDescriptor<TDocument>
• AsyncQueryGetRequestDescriptor<TDocument>
• AsyncSearchStatusRequestDescriptor<TDocument>
• DatabaseConfigurationDescriptor<TDocument>
• DatabaseConfigurationFullDescriptor<TDocument>
• DeleteAsyncRequestDescriptor<TDocument>
• DeleteAsyncSearchRequestDescriptor<TDocument>
• DeleteDataFrameAnalyticsRequestDescriptor<TDocument>
• DeleteGeoipDatabaseRequestDescriptor<TDocument>
• DeleteIpLocationDatabaseRequestDescriptor<TDocument>
• DeleteJobRequestDescriptor<TDocument>
• DeletePipelineRequestDescriptor<TDocument>
• DeleteScriptRequestDescriptor<TDocument>
• DeleteSynonymRequestDescriptor<TDocument>
• EqlDeleteRequestDescriptor<TDocument>
• EqlGetRequestDescriptor<TDocument>
• GetAsyncRequestDescriptor<TDocument>
• GetAsyncSearchRequestDescriptor<TDocument>
• GetAsyncStatusRequestDescriptor<TDocument>
• GetDataFrameAnalyticsRequestDescriptor<TDocument>
• GetDataFrameAnalyticsStatsRequestDescriptor<TDocument>
• GetEqlStatusRequestDescriptor<TDocument>
• GetGeoipDatabaseRequestDescriptor<TDocument>
• GetIpLocationDatabaseRequestDescriptor<TDocument>
• GetJobsRequestDescriptor<TDocument>
• GetPipelineRequestDescriptor<TDocument>
• GetRollupCapsRequestDescriptor<TDocument>
• GetRollupIndexCapsRequestDescriptor<TDocument>
• GetScriptRequestDescriptor<TDocument>
• GetSynonymRequestDescriptor<TDocument>
• IndexModifyDataStreamActionDescriptor<TDocument>
• PreprocessorDescriptor<TDocument>
• PutGeoipDatabaseRequestDescriptor<TDocument>
• PutIpLocationDatabaseRequestDescriptor<TDocument>
• PutScriptRequestDescriptor<TDocument>
• PutSynonymRequestDescriptor<TDocument>
• QueryVectorBuilderDescriptor<TDocument>
• RankDescriptor<TDocument>
• RenderSearchTemplateRequestDescriptor<TDocument>
• SmoothingModelDescriptor<TDocument>
• StartDataFrameAnalyticsRequestDescriptor<TDocument>
• StartJobRequestDescriptor<TDocument>
• StopDataFrameAnalyticsRequestDescriptor<TDocument>
• StopJobRequestDescriptor<TDocument>
• TokenizationConfigDescriptor<TDocument>
• UpdateDataFrameAnalyticsRequestDescriptor<TDocument>

2.3 💥

Removed (TDocument, IndexName) descriptor constructors and related request APIs for all requests with IndexName and Id path parameters.

For example:

// 8.x
public IndexRequestDescriptor(TDocument document, IndexName index, Id? id) { }
public IndexRequestDescriptor(TDocument document, IndexName index) { }
public IndexRequestDescriptor(TDocument document, Id? id) { }
public IndexRequestDescriptor(TDocument document) { }

// 9.0
public IndexRequestDescriptor(TDocument document, IndexName index, Id? id) { }
public IndexRequestDescriptor(TDocument document, Id? id) { }
public IndexRequestDescriptor(TDocument document) { }

These overloads caused invocation ambiguities since both, IndexName and Id implement implicit conversion operators from string.

Alternative with same semantics:

// Descriptor constructor.
new IndexRequestDescriptor(document, "my_index", Id.From(document));

// Request API method.
await client.IndexAsync(document, "my_index", Id.From(document), ...);

2.4 💥

Removed AggregationRange/AggregationRangeDescriptor.

Replaced with IAggregationRange/IAggregationRangeBuilder and corresponding implementations DateAggregationRange/DateAggregationRangeDescriptor,
NumberAggregationRange/NumberAggregationRangeDescriptor, and TermAggregationRange/TermAggregationRangeDescriptor.

This change is driven by a modification in the Elasticsearch specification, which is used to generate large parts of the client.

Object API:

// 8.x
new SearchRequest
{
    Aggregations = new Dictionary<string, Aggregation>
    {
        { "my_range", new RangeAggregation
        {
            Ranges = new List<AggregationRange>
            {
                new AggregationRange
                {
                    From = 1.0,
                    To = 10.0
                }
            }
        }}
    }
};

// 9.0
new SearchRequest
{
    Aggregations = new Dictionary<string, Aggregation>
    {
        { "my_range", new RangeAggregation
        {
            Ranges = new List<IAggregationRange>
            {
                new NumberAggregationRange
                {
                    From = 1.0,
                    To = 10.0
                }
            }
        }}
    }
};

Fluent API:

// 8.x
new SearchRequestDescriptor<Person>()
    .Aggregations(aggs =>
        aggs.Add("my_range", agg => agg
            .Range(range => range.Ranges(ranges => ranges.From(1.0).To(10.0)))
        )
    );

// 9.0
new SearchRequestDescriptor<Person>()
    .Aggregations(aggs =>
        aggs.Add("my_range", agg => agg
            .Range(range => range.Ranges(ranges => ranges.Number(number => number.From(1.0).To(10.0))))
        )
    );

2.5 💥

Renamed DateRangeExpression/DateRangeExpressionDescriptor to DateAggregationRange/DateAggregationRangeDescriptor.

This change is related to 2.3.

2.6 💥

In places where previously long or double was used to represent a date/time/duration value, DateTime or TimeSpan is now used instead.

2.7 💥

Removed ExtendedBoundsDate/ExtendedBoundsDateDescriptor, ExtendedBoundsFloat/ExtendedBoundsFloatDescriptor.

Replaced by ExtendedBounds<T>, ExtendedBoundsOfFieldDateMathDescriptor, and ExtendedBoundsOfDoubleDescriptor.

2.8 ⚠️

Removed Field.Format property and corresponding constructor and inferrer overloads.

This property has not been used for some time (replaced by the FieldAndFormat type).

2.9 ⚠️

Field/Fields static factory methods and conversion operators no longer return nullable references but throw exceptions instead (Field) if the input string/Expression/PropertyInfo argument is null.

This makes implicit conversions to Field more user-friendly without requiring the null-forgiveness operator (!) (read more).

2.10 ⚠️

Removed FieldValue.IsLazyDocument, FieldValue.IsComposite, and the corresponding members in the FieldValue.ValueKind enum.

These values have not been used for some time.

2.11 ⚠️

Removed static FieldSort.Empty member.

Sorting got reworked which makes this member obsolete (read more).

2.12 ⚠️

All descriptor types are now implemented as struct instead of class.

References

Closes #8485
Closes #8002
Closes #7792
Closes #8013
Closes #7812
Closes #8335
Closes #8338
Closes #8349
Closes #8227
Closes #8191
Closes #8150
Closes #8479
Closes #8465
Closes #8436
Closes #8435
Closes #8378
Closes #8347
Closes #8309
Closes #8255
Closes #8485
Closes #8149
Closes #8114
Closes #8016
Closes #7968
Closes #7913
Closes #7872
Closes #7855
Closes #7825
Closes #7594
Closes #7508