Migration to a Backend Data Source

[fridgepoet]

Before Node Graph

Preview Give feedback

1. https://github.com/grafana/oss-plugin-partnerships/issues/192
2. https://github.com/grafana/oss-plugin-partnerships/issues/195
3. move non-reg dashboard to cloudatasource Cloud instance
4. Frontend: Refactor PPL and Lucene time series so that they return data frames #185
   datasource/OpenSearch effort/small type/chore +3
   
5. Backend: Tables need to be converted to data frames #175
   datasource/OpenSearch effort/medium type/chore +3
   
6. Backend: Implement Lucene raw_data queries in the backend #196
   datasource/OpenSearch effort/large +2
   
7. Prerequisite for Lucene work below: ad hoc filters not applied to raw_data queries #224
8. Backend: Implement Lucene raw_document query #195
   datasource/OpenSearch effort/medium +2
   
9. Backend: Implement Lucene Logs queries in the backend #199
   datasource/OpenSearch effort/large +2
   
10. [stretch goal but the sooner the better] Investigate non-regression snapshot tests #248
    effort/small +1
    
11. Frontend: Pre-requisite for PPL migrations - refactor adhoc filters on frontend before sending query to backend #226
    effort/small +1
    
12. https://github.com/grafana/oss-plugin-partnerships/issues/199
13. Add snapshot tests for Lucene raw_data, raw_document, logs #255
    effort/medium type/task +2
    
14. Backend: Migrate Lucene metrics to the backend #197
    datasource/OpenSearch effort/medium +2
    
15. Backend: Migrate PPL Logs queries to the backend #200
    datasource/OpenSearch effort/medium +2
    
16. Backend: Migrate PPL Table Format to the backend #201
    datasource/OpenSearch effort/medium +2
    
17. Backend: Re-write "Convey" tests to use testify #156
    datasource/OpenSearch effort/small good first issue type/chore +4
    
18. Backend: Implement Lucene Traces in the backend #223
    3 of 3
    effort/none +1
    

After Node Graph

Preview Give feedback

1. Backend Migration: Migrate PPL Time Series to the backend #296
   datasource/OpenSearch effort/none +2
   
2. Backend Migration: Remove Lucene Logs Explore-only restriction (condition is in frontend) #275
   datasource/OpenSearch effort/none +2
3. Backend Migration: Remove Lucene Metrics Explore-only restriction (condition is in frontend) #290
   datasource/OpenSearch effort/none +2
4. Backend Migration: Remove PPL Logs Explore-only restriction (condition is in frontend) #372
   datasource/OpenSearch effort/none +2
5. Backend Migration: Remove PPL Table Explore-only restriction (condition is in frontend) #373
   datasource/OpenSearch effort/small +2
6. Backend Migration: Remove PPL Time Series Explore-only restriction (condition is in frontend) #374
   datasource/OpenSearch effort/none +2
7. Backend Migration: (FE) Investigate any refactoring of ad hoc filter logic like https://github.com/grafana/grafana/pull/65274 #379
   effort/medium +1
   
8. Make sure that the plugin returns a Grafana-readable timestamp format #161 (Investigate which queries this change may need to be applied to Elasticsearch: Fix processing of logs with not-recognized time format grafana#67767)
9. Add and rollout openSearchBackendFlowEnabled feature toggle to Grafana #407
   effort/none +1
   
10. Remove openSearchBackendFlowEnabled feature toggle #436
    +0
    

Related bugs

Preview Give feedback

1. Inconsistent results between alerts and queries #176
   datasource/OpenSearch effort/small type/bug +3
   
2. alerts fail to evaluate queries and expressions #173
   datasource/OpenSearch effort/none type/bug +3
3. Backend PPL queries should support more than 2 fields #181
   datasource/OpenSearch type/bug +2

[katebrenner]

Project Update

Highlight
All the queries are running in the backend in both explore and dashboards (explore-only restriction has been removed), besides PPL time series format, which is just in explore.

Trending

🟢 on track

Next

• release Prepare 2.15.0, add docs for Service Map #377
• Backend Migration: (FE) Investigate any refactoring of ad hoc filter logic like https://github.com/grafana/grafana/pull/65274 #379

[katebrenner]

Project Update

Highlight
Reverting due to 2 bugs - #389 and #388

Trending

🟢 on track

Next

Fix the 2 bugs related to the migration, so we can revert the revert

[kevinwcyu]

Project Update

Highlight
Found and fixed one additional bug #391

Trending

🟢 on track

Next

#379 and #389 are in progress

One more bug fix #388 needs to be completed and then we can route all queries to the backend

[kevinwcyu]

Project Update

Highlight
#379 and #389 are in review
#388 under investigation

Trending

🟢 on track

Next

Waiting for remaining bug fixes to be completed and then we will start enabling different query types (one by one) to route all queries to the backend.

[kevinwcyu]

Project Update

Highlight
#379 and #389 are completed
#388 closed due to lack of reproduction instructions
#405 pr opened

Trending

🟢 on track

Next

#407 will add feature flag to start executing queries from dashboards through the backend path

[kevinwcyu]

Project Update

Highlight
PRs opened and ready to merge for adding feature toggle to roll out dashboard queries through the backend.
PR to add feature toggle in Grafana
PR to use feature toggle in OpenSearch data source

Trending

🟢 on track

Next

Will start rolling out the feature toggle up to canary once the above PRs are merged.

[kevinwcyu]

Project Update

Highlight
Rolled out feature toggle to send dashboard queries to the backend in dev, staging, and canary.

Trending

🟢 on track

Next

Will starting rolling it out to prod for 30% of users and let it bake for a bit.

[kevinwcyu]

Project Update

Highlight
Rolled out feature toggle to prod for 30% of users.

Trending

🟢 on track

Next

Roll out to 60% of users in prod and let it bake for a bit before rolling it out to 100% of users.

[kevinwcyu]

Project Update

Highlight
Rolled out feature toggle to prod for 65% of users.

Trending

🟢 on track

Next

Roll out to 100% of users in prod.

[kevinwcyu]

Project Update

Highlight
Rolled out feature toggle to prod for 100% of users.

Trending

🟢 on track

Next

This epic is done