Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend: Remove _doc from sort array in query building, Remove limit from response processing, Prepare release v2.13.1 #278

Merged
merged 12 commits into from
Oct 12, 2023

Conversation

fridgepoet
Copy link
Member

@fridgepoet fridgepoet commented Oct 5, 2023

What this PR does / why we need it:
This PR addresses any difference between the frontend and backend query building and response processing to Grafana data frames for Lucene logs.

The only change in this PR is in bold text.

Difference list:
Query building to OpenSearch:

  • Backend has "docvalue_fields":["timestamp"]. Frontend code does not add this value, but I am reluctant to remove it as it seems like some kind of optimization (1, 2). It does not seem to negatively affect the behavior. For what it's worth Elasticsearch also has this in the query input.
  • Frontend has an extra empty nested "aggs" in the "aggs" object which is not present in the backend. It doesn't seem to serve a purpose, so not adding it to the backend. This is not present in Elasticsearch in an equivalent query.
  • Backend code has an extra object "fields" with timeField "format": "strict_date_optional_time_nanos", which was a fix based on Elasticsearch: Fix processing of logs with not-recognized time format grafana#67767. It was included in Add raw_data query support to backend #203.
  • Backend's sort object is an array which was changed from a single object as it is in the frontend in Change query sort to respect sort order #211
  • Backend's sort has {"_doc":{"order":"desc"}} which is not present in the frontend. This came from the equivalent backend migration PR of Elasticsearch, but I don't see evidence of _doc anymore in the latest OpenSearch documentation. In any case since this is not present in the frontend, this was removed.

Grafana data frame response:

  • Backend has an extra "limit" object which is not in frontend. This came from the equivalent backend migration PR of Elasticsearch, but the UI includes a "limit" option which OpenSearch does not currently have. This was removed.

Which issue(s) this PR fixes:

Contributes to #199

Special notes for reviewers:
Testing was done locally by including the changes from this branch: https://github.com/grafana/opensearch-datasource/tree/backend-flow
This redirects all Lucene queries to the backend (not just from the Explore view).

@fridgepoet fridgepoet changed the title Clean logs Backend: Harmonize Lucene logs to better match frontend Oct 5, 2023
@fridgepoet fridgepoet changed the title Backend: Harmonize Lucene logs to better match frontend Backend: Harmonize Lucene logs query building and response processing with frontend's Oct 5, 2023
Base automatically changed from fix-lucene-logs to main October 6, 2023 09:39
@fridgepoet fridgepoet changed the title Backend: Harmonize Lucene logs query building and response processing with frontend's Backend: Harmonize Lucene logs query building and response processing with frontend's, Prepare release v2.13.1 Oct 6, 2023
@fridgepoet fridgepoet changed the title Backend: Harmonize Lucene logs query building and response processing with frontend's, Prepare release v2.13.1 Backend: Remove extra _doc from sort array in query building, Prepare release v2.13.1 Oct 9, 2023
@fridgepoet fridgepoet changed the title Backend: Remove extra _doc from sort array in query building, Prepare release v2.13.1 Backend: Remove extra _doc from sort array in query building, Remove limit from response processing, Prepare release v2.13.1 Oct 9, 2023
@fridgepoet fridgepoet marked this pull request as ready for review October 9, 2023 10:54
@fridgepoet fridgepoet requested a review from a team as a code owner October 9, 2023 10:54
@fridgepoet fridgepoet requested review from iwysiu and sarahzinger and removed request for a team October 9, 2023 10:54
@fridgepoet fridgepoet changed the title Backend: Remove extra _doc from sort array in query building, Remove limit from response processing, Prepare release v2.13.1 Backend: Remove _doc from sort array in query building, Remove limit from response processing, Prepare release v2.13.1 Oct 9, 2023
@fridgepoet fridgepoet merged commit ae81111 into main Oct 12, 2023
@fridgepoet fridgepoet deleted the clean-logs branch October 12, 2023 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants