OurDNA: Dashboard Backend (#750)

* first commit, and initial code for aggregating the data

* added additional logic for aggregation, tested initial schema on gql

* added a mapping against participants for all samples

* added missing return field

* moved the ourdna field to the project type in the gql schema

* Update processing_time_by_site to support a histogram of hour buckets, rather than dict that keys on samples

* Add processing time statistics - average, min and max.

* Rename variables, - rather than _

* OurDNA Dashboard Field returns a single JSON rather than a list
as we are only supporting one project at a time

* Updated codecov to v4

* added helper function to allow hyphen and underscore in meta props

* Fix lint on project filter

* In Progress: Initial commit for tests

* In Progress: Testing Scaffold

* Refactor setup to improve and generalise data acess, add test_collection_to_process_end_time

* added some test cases

* added some more tests

* added final tests, fixed odd logic

* 🔨 Add generate_ourdna_data script.

* ✏️ Linting.

* reformatted dashboard with black

* refactored and optimised participant queries

* removed some TODO stubs

* code clean up, added ourdna model

* updated queries, revamped tests, added models

* fixed broken test

* refactored tests to use unittest methods

* refactored tests to use unittest methods

* cleaned up the layer logic, simplified

* encoded key accesses into fields

* OurDNA: Dashboard Frontend (#766)

* added skeleton for ourdna dashboard

* better dashboard skeleton

* added chakra ui and integrated with backend

* Fix colours on histogram

* put package files in sync

* added a grid layout

* added borders and fixed tabletile

* added height attr to bar chart

* Apply Jakarta Sans font, overwrite chakra default without using theme - as it messe up other pages

* Add all the icons to the dashboard

* Update badge colours to match icons

* Heading redundant

* Fix fontsize and weight of tile headings

* Create three stat tile and adjust formatting so all tiles are the same height

* added responsive fonts

* responsive grid, fixed spacing on doughnut chart and tested on ipad pro resolution

* fixed table tiles

* added some refactoring

* trying a grid layout on semantic

* removed chakra provider

* fixed the grid, fonts, labels

* Fix pie chart size, leave 10% for title

* Add type hint for Dashboard Data + move project to env variable

* Add colours to CSS, call vars instead of hardcoded rgbs

* added docs on env variable for vite

* switched bar chart to d3

* fixed some linting

* moved chart colour to var references

* refactored icons into components, changed vars to camelCase

* 😎 Dark mode compatibility

* fixed legend not showing properly on chart

* Add ourdna identity svg

* Create react component for logo

* Remove unused logos

* converted donut to d3

* fixed table tile

* cleaned up chakra/chartjs remnants

* updated package-lock, minor ourdna fix

* updated gql to use generated types

---------

Co-authored-by: vivbak <[email protected]>
Co-authored-by: Vivian Bakiris <[email protected]>

---------

Co-authored-by: vivbak <[email protected]>
Co-authored-by: Milo Hyben <[email protected]>
Co-authored-by: Vivian Bakiris <[email protected]>

.github/workflows/test.yaml

@@ -72,7 +72,7 @@ jobs:
           echo "rc=$rc" >> $GITHUB_OUTPUT
 
       - name: "Upload coverage report"
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v4
         with:
           files: ./coverage.xml
           token: ${{ secrets.CODECOV_TOKEN }}

api/graphql/schema.py

@@ -26,6 +26,7 @@
     AssayLayer,
     CohortLayer,
     FamilyLayer,
+    OurDnaDashboardLayer,
     SampleLayer,
     SequencingGroupLayer,
 )
@@ -53,6 +54,7 @@
 )
 from models.models.analysis_runner import AnalysisRunnerInternal
 from models.models.family import PedRowInternal
+from models.models.ourdna import OurDNADashboard, OurDNALostSample
 from models.models.project import ProjectId
 from models.models.sample import sample_id_transform_to_raw
 from models.utils.cohort_id_format import cohort_id_format, cohort_id_transform_to_raw
@@ -89,6 +91,28 @@ async def m(info: Info) -> list[str]:
 GraphQLAnalysisStatus = strawberry.enum(AnalysisStatus)
 
 
+@strawberry.experimental.pydantic.type(model=OurDNALostSample, all_fields=True)  # type: ignore
+class GraphQLOurDNALostSample:
+    """OurDNA Lost Sample GraphQL model to be used in OurDNA Dashboard"""
+
+    pass  # pylint: disable=unnecessary-pass
+
+
+@strawberry.experimental.pydantic.type(model=OurDNADashboard)  # type: ignore
+class GraphQLOurDNADashboard:
+    """OurDNA Dashboard model"""
+
+    collection_to_process_end_time: strawberry.scalars.JSON
+    collection_to_process_end_time_statistics: strawberry.scalars.JSON
+    collection_to_process_end_time_24h: strawberry.scalars.JSON
+    processing_times_by_site: strawberry.scalars.JSON
+    total_samples_by_collection_event_name: strawberry.scalars.JSON
+    samples_lost_after_collection: list[GraphQLOurDNALostSample]
+    samples_concentration_gt_1ug: strawberry.scalars.JSON
+    participants_consented_not_collected: list[int]
+    participants_signed_not_consented: list[int]
+
+
 # Create cohort GraphQL model
 @strawberry.type
 class GraphQLCohort:
@@ -244,6 +268,18 @@ async def analysis_runner(
         analysis_runners = await alayer.query(filter_)
         return [GraphQLAnalysisRunner.from_internal(ar) for ar in analysis_runners]
 
+    @strawberry.field
+    async def ourdna_dashboard(
+        self, info: Info, root: 'Project'
+    ) -> 'GraphQLOurDNADashboard':
+        connection = info.context['connection']
+        ourdna_layer = OurDnaDashboardLayer(connection)
+        if not root.id:
+            raise ValueError('Project must have an id')
+        ourdna_dashboard = await ourdna_layer.query(project_id=root.id)
+        # pylint: disable=no-member
+        return GraphQLOurDNADashboard.from_pydantic(ourdna_dashboard)
+
     @strawberry.field()
     async def pedigree(
         self,

db/python/layers/__init__.py

@@ -6,6 +6,7 @@
 from db.python.layers.billing import BillingLayer
 from db.python.layers.cohort import CohortLayer
 from db.python.layers.family import FamilyLayer
+from db.python.layers.ourdna.dashboard import OurDnaDashboardLayer
 from db.python.layers.participant import ParticipantLayer
 from db.python.layers.sample import SampleLayer
 from db.python.layers.search import SearchLayer