Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend: Implement Lucene Logs queries in the backend #199

Closed
Tracked by #192
idastambuk opened this issue Jun 12, 2023 · 0 comments · Fixed by #249
Closed
Tracked by #192

Backend: Implement Lucene Logs queries in the backend #199

idastambuk opened this issue Jun 12, 2023 · 0 comments · Fixed by #249

Comments

@idastambuk
Copy link
Contributor

idastambuk commented Jun 12, 2023

Note: Blocked by #224

Currently, there is no separate flow to process Lucene Logs on the backend. They are processed as time series in processAggregationDocs.
On the FE logs are processed in processResponseToDataFrames().
The output should be compared and separated out (from raw_data or in general) if necessary. We might be able to combine common logic with raw_data processing: For comparison, the ElasticSearch PR is here.

Some general guidelines are:

  • Create visualization to compare between frontend and backend flows, preferably in a shared Cloud dashboard e.g. clouddatasources.grafana.net
  • Unit tests in the frontend should be copied or updated in the backend to assert the same behavior and add unit tests when there are none.
    • Check Elasticsearch repo for related tests – helpful when there are no OpenSearch tests in the frontend or backend
  • We should match OpenSearch's backend behavior to the current frontend behavior, but be aware it may not always make sense. Elasticsearch's current behavior is also another resource for how OpenSearch should behave.
    • Let's be pragmatic about this – our foremost goal is the migration of OpenSearch's frontend behavior. We can aim for some Elasticsearch feature parity, but only within reason.
  • Often there is a PR in the Elasticsearch backend data source migration Elasticsearch: Convert the datasource to be a backend datasource grafana#54011 which is 75% of what OpenSearch will need
  • Migrate or implement: i.e. make any changes to code in backend
  • Use the dashboard created above to compare visualization results (data frame comparison)
    • Can also inspect data frame on a more detailed basis

Note: It seems like we have a problem, at least on the FE, with detecting field types, for example here the Query inspector shows AvgTicketPrice as string and not int, which isn't what I would expect. Just something to keep in mind when comparing Logs.

Screenshot 2023-06-12 at 1 26 02 PM Screenshot 2023-06-12 at 1 28 30 PM
@idastambuk idastambuk moved this from Incoming to Backlog in AWS Datasources Jun 12, 2023
@sarahzinger sarahzinger moved this from Backlog to Next in AWS Datasources Jun 12, 2023
@fridgepoet fridgepoet self-assigned this Jul 13, 2023
@fridgepoet fridgepoet moved this from Next to In Progress in AWS Datasources Jul 13, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in AWS Datasources Sep 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants