Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce ApproximateRangeQuery and ApproximateableQuery #13788

Merged
merged 20 commits into from
Sep 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add index creation using the context field ([#15290](https://github.com/opensearch-project/OpenSearch/pull/15290))
- Add fieldType to AbstractQueryBuilder and FieldSortBuilder ([#15328](https://github.com/opensearch-project/OpenSearch/pull/15328)))
- [Reader Writer Separation] Add searchOnly replica routing configuration ([#15410](https://github.com/opensearch-project/OpenSearch/pull/15410))
- [Range Queries] Add new approximateable query framework to short-circuit range queries ([#13788](https://github.com/opensearch-project/OpenSearch/pull/13788))
- [Workload Management] Add query group level failure tracking ([#15227](https://github.com/opensearch-project/OpenSearch/pull/15527))

### Dependencies
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
"search with approximate range":
- do:
indices.create:
index: test
body:
mappings:
properties:
date:
type: date
index: true
doc_values: true

- do:
bulk:
index: test
refresh: true
body:
- '{"index": {"_index": "test", "_id": "1" }}'
- '{ "date": "2018-10-29T12:12:12.987Z" }'
- '{ "index": { "_index": "test", "_id": "2" }}'
- '{ "date": "2020-10-29T12:12:12.987Z" }'
- '{ "index": { "_index": "test", "_id": "3" } }'
- '{ "date": "2024-10-29T12:12:12.987Z" }'

- do:
search:
rest_total_hits_as_int: true
index: test
body:
query:
range: {
date: {
gte: "2018-10-29T12:12:12.987Z"
},
}

- match: { hits.total: 3 }

- do:
search:
rest_total_hits_as_int: true
index: test
body:
sort: [{ date: asc }]
query:
range: {
date: {
gte: "2018-10-29T12:12:12.987Z"
},
}


- match: { hits.total: 3 }
- match: { hits.hits.0._id: "1" }

- do:
search:
rest_total_hits_as_int: true
index: test
body:
sort: [{ date: desc }]
query:
range: {
date: {
gte: "2018-10-29T12:12:12.987Z",
lte: "2020-10-29T12:12:12.987Z"
},
}

- match: { hits.total: 2 }
- match: { hits.hits.0._id: "2" }
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,16 @@ public class FeatureFlags {
Property.NodeScope
);

/**
* Gates the functionality of ApproximatePointRangeQuery where we approximate query results.
*/
public static final String APPROXIMATE_POINT_RANGE_QUERY = "opensearch.experimental.feature.approximate_point_range_query.enabled";
public static final Setting<Boolean> APPROXIMATE_POINT_RANGE_QUERY_SETTING = Setting.boolSetting(
APPROXIMATE_POINT_RANGE_QUERY,
false,
Property.NodeScope
);

private static final List<Setting<Boolean>> ALL_FEATURE_FLAG_SETTINGS = List.of(
REMOTE_STORE_MIGRATION_EXPERIMENTAL_SETTING,
EXTENSIONS_SETTING,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@
import org.opensearch.index.query.QueryRewriteContext;
import org.opensearch.index.query.QueryShardContext;
import org.opensearch.search.DocValueFormat;
import org.opensearch.search.approximate.ApproximateIndexOrDocValuesQuery;
import org.opensearch.search.approximate.ApproximatePointRangeQuery;
import org.opensearch.search.lookup.SearchLookup;

import java.io.IOException;
Expand All @@ -80,6 +82,7 @@
import java.util.function.Supplier;

import static org.opensearch.common.time.DateUtils.toLong;
import static org.apache.lucene.document.LongPoint.pack;

/**
* A {@link FieldMapper} for dates.
Expand Down Expand Up @@ -108,6 +111,21 @@
: LEGACY_DEFAULT_DATE_TIME_FORMATTER;
}

public static Query getDefaultQuery(Query pointRangeQuery, Query dvQuery, String name, long l, long u) {
return FeatureFlags.isEnabled(FeatureFlags.APPROXIMATE_POINT_RANGE_QUERY_SETTING)
? new ApproximateIndexOrDocValuesQuery(

Check warning on line 116 in server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L116

Added line #L116 was not covered by tests
pointRangeQuery,
new ApproximatePointRangeQuery(name, pack(new long[] { l }).bytes, pack(new long[] { u }).bytes, new long[] { l }.length) {

Check warning on line 118 in server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L118

Added line #L118 was not covered by tests
@Override
protected String toString(int dimension, byte[] value) {
return Long.toString(LongPoint.decodeDimension(value, 0));

Check warning on line 121 in server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L121

Added line #L121 was not covered by tests
}
},
dvQuery
)
: new IndexOrDocValuesQuery(pointRangeQuery, dvQuery);
}

/**
* Resolution of the date time
*
Expand Down Expand Up @@ -463,24 +481,22 @@
}
DateMathParser parser = forcedDateParser == null ? dateMathParser : forcedDateParser;
return dateRangeQuery(lowerTerm, upperTerm, includeLower, includeUpper, timeZone, parser, context, resolution, (l, u) -> {
Query pointRangeQuery = isSearchable() ? LongPoint.newRangeQuery(name(), l, u) : null;
Query dvQuery = hasDocValues() ? SortedNumericDocValuesField.newSlowRangeQuery(name(), l, u) : null;
if (isSearchable() && hasDocValues()) {
Query query = LongPoint.newRangeQuery(name(), l, u);
Query dvQuery = SortedNumericDocValuesField.newSlowRangeQuery(name(), l, u);
query = new IndexOrDocValuesQuery(query, dvQuery);

Query query = getDefaultQuery(pointRangeQuery, dvQuery, name(), l, u);
if (context.indexSortedOnField(name())) {
query = new IndexSortSortedNumericDocValuesRangeQuery(name(), l, u, query);
}
return query;
}
if (hasDocValues()) {
Query query = SortedNumericDocValuesField.newSlowRangeQuery(name(), l, u);
if (context.indexSortedOnField(name())) {
query = new IndexSortSortedNumericDocValuesRangeQuery(name(), l, u, query);
dvQuery = new IndexSortSortedNumericDocValuesRangeQuery(name(), l, u, dvQuery);

Check warning on line 495 in server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L495

Added line #L495 was not covered by tests
}
return query;
return dvQuery;

Check warning on line 497 in server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L497

Added line #L497 was not covered by tests
}
return LongPoint.newRangeQuery(name(), l, u);
return pointRangeQuery;

Check warning on line 499 in server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L499

Added line #L499 was not covered by tests
});
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.opensearch.common.lucene.search.function.FunctionScoreQuery;
import org.opensearch.index.mapper.DateFieldMapper;
import org.opensearch.index.query.DateRangeIncludingNowQuery;
import org.opensearch.search.approximate.ApproximateIndexOrDocValuesQuery;
import org.opensearch.search.internal.SearchContext;

import java.io.IOException;
Expand Down Expand Up @@ -54,6 +55,7 @@ private Helper() {}
queryWrappers.put(FunctionScoreQuery.class, q -> ((FunctionScoreQuery) q).getSubQuery());
queryWrappers.put(DateRangeIncludingNowQuery.class, q -> ((DateRangeIncludingNowQuery) q).getQuery());
queryWrappers.put(IndexOrDocValuesQuery.class, q -> ((IndexOrDocValuesQuery) q).getIndexQuery());
queryWrappers.put(ApproximateIndexOrDocValuesQuery.class, q -> ((ApproximateIndexOrDocValuesQuery) q).getOriginalQuery());
}

/**
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.search.approximate;

import org.apache.lucene.search.IndexOrDocValuesQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.QueryVisitor;

/**
* A wrapper around {@link IndexOrDocValuesQuery} that can be used to run approximate queries.
* It delegates to either {@link ApproximateQuery} or {@link IndexOrDocValuesQuery} based on whether the query can be approximated or not.
* @see ApproximateQuery
*/
public final class ApproximateIndexOrDocValuesQuery extends ApproximateScoreQuery {

private final ApproximateQuery approximateIndexQuery;
private final IndexOrDocValuesQuery indexOrDocValuesQuery;

public ApproximateIndexOrDocValuesQuery(Query indexQuery, ApproximateQuery approximateIndexQuery, Query dvQuery) {
super(new IndexOrDocValuesQuery(indexQuery, dvQuery), approximateIndexQuery);
this.approximateIndexQuery = approximateIndexQuery;
this.indexOrDocValuesQuery = new IndexOrDocValuesQuery(indexQuery, dvQuery);
}

@Override
public String toString(String field) {
return "ApproximateIndexOrDocValuesQuery(indexQuery="
+ indexOrDocValuesQuery.getIndexQuery().toString(field)

Check warning on line 34 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L33-L34

Added lines #L33 - L34 were not covered by tests
+ ", approximateIndexQuery="
+ approximateIndexQuery.toString(field)

Check warning on line 36 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L36

Added line #L36 was not covered by tests
+ ", dvQuery="
+ indexOrDocValuesQuery.getRandomAccessQuery().toString(field)

Check warning on line 38 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L38

Added line #L38 was not covered by tests
+ ")";
}

@Override
public void visit(QueryVisitor visitor) {
indexOrDocValuesQuery.visit(visitor);
}

Check warning on line 45 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L44-L45

Added lines #L44 - L45 were not covered by tests

@Override
public boolean equals(Object obj) {
if (sameClassAs(obj) == false) {
return false;

Check warning on line 50 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L50

Added line #L50 was not covered by tests
}
return true;

Check warning on line 52 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L52

Added line #L52 was not covered by tests
}

@Override
public int hashCode() {
int h = classHash();
h = 31 * h + indexOrDocValuesQuery.getIndexQuery().hashCode();
h = 31 * h + indexOrDocValuesQuery.getRandomAccessQuery().hashCode();
return h;
}

Check warning on line 61 in server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/search/approximate/ApproximateIndexOrDocValuesQuery.java#L57-L61

Added lines #L57 - L61 were not covered by tests
}
Loading
Loading