feat: Data upload: upload a CSV data set #352
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
eatyourgreens
commented
Feb 27, 2024
•
eatyourgreens
commented
- towards upload data #347.
- upload a CSV file and map headers.
- map administration ID to dosing compartment.
- map observations to model variables.
- data stratification UI.
- data stratification in the Django models (subject groups.)
- generate groups of simulations from datasets.
- data visualisation.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## development #352 +/- ##
===============================================
+ Coverage 76.35% 76.82% +0.46%
===============================================
Files 95 101 +6
Lines 5379 5492 +113
===============================================
+ Hits 4107 4219 +112
- Misses 1272 1273 +1 ☔ View full report in Codecov by Sentry. |
eatyourgreens
force-pushed
the
data-upload
branch
4 times, most recently
from
b572be0 to
084892c
Compare
eatyourgreens
changed the title
eatyourgreens
force-pushed
the
data-upload
branch
3 times, most recently
from
9abe775 to
89673da
Compare
eatyourgreens
force-pushed
the
development
branch
from
1752f70 to
f41756b
Compare
eatyourgreens
force-pushed
the
data-upload
branch
6 times, most recently
from
ebedbe8 to
7d8edea
Compare
Fix type errors for data sub-page tabs Case insensitive header normalisation Trim blank lines from CSV Add Administration ID to the CSV data parser Allow time_unit as a header
* Map data to model variables Show a table of dose amount data (and units), with select menus to map these to model inputs. Show a table of observation variables (and units), with select menus to map those to model outputs. * Add a mapped qname to biomarkers Store the mapped variable qname on biomarker types. Read it from OBSERVATION_VARIABLE in a dataset. * Support ES2015 * Allow for multiple variable mappings Map each Administration ID to a dosing compartment. Map each Observation ID to a model output. Add new columns to the CSV data, with mappings and optional units. * Preview the final dataset Add a final step to the upload, which will preview the final CSV before saving it. * Save modified dataset to backend - load or create a dataset when we start an upload. - save the dataset when we finish an upload. - modify the `/datasets/:dataset_id:/csv` endpoint to accept a JSON string. - update the dataset API to allow filtering by project ID. * Allow for a single unit column - When there's a single unit column, use that column for both dosing and observations. - Split the CSV data into dosing rows and observation rows. - Add administration route to the Map Dosing screen. - Allow for dimensionless observation units. - Filter mapped observation variables for compatibility with the observation unit value.
- Replace the tabbed interface for data uploads with a stepper. - Add utilities to group subjects by protocol. - Add stratification to the stepper. - Stratify by dose protocols for the time being. - Display each protocol group as a MUI data grid. - Display protocol groups as tabs in the trial design view. - Add the dosing compartment qname to the protocol model. - Add a 'cohort' field to the CSV.
- always use the global dataset object rather than local state. - remove a debugging log.
- expand the lists of expected column headers for units and observations. - set non-numeric observations to 0.0.
- break `create_myokit_simulator` up into smaller methods. - break `simulate` up into smaller methods. - add optional `dosing_protocols` to `create_myokit_simulator`. If set, it overrides the default dosing variable protocols.
* feat: subject groups Add a subject group model to the Django app and the API. Subject groups have a list of subjects and a list of protocols associated with those subjects. Datasets have a list of subject groups in `dataset.groups`. Refactor the Data and Trial Design views to use `dataset.groups` for tabbed views. * Change GROUP to GROUP_ID
Refactor the Myokit model mixin to allow multiple simulations to be run. Each dataset subject group has its own simulation, with subject dosing protocols and dosing events. Update the simulate API to return multiple simulations. Show multiple plots in the Simulations tab Test simulations with a project and dataset
`SimulateResponseSerializer` doesn't need a separate `ListSerializer`. It can use `many=True` to return a list of simulations.
- add biomarker data to the dataset serialiser. - add biomarker data to the `useDataset` hook. - display the dataset as data grids in the Data tab. - show biomarkers as scatter plots on simulation plots.
eatyourgreens
force-pushed
the
data-upload
branch
from
42e714a to
d928d86
Compare
When the uploaded CSV doesn't have an observation unit column, allow the user to pick a unit from a list of units, then pick a compatible variable for the selected unit.
Better handling of missing unit columns in the CSV. - When time units are missing, prompt to choose a time unit. - Change the column headings in the submitted CSV to match headings that are expected by the Python data parser utility. - Add a preclinical flag to the dosing step during uploads.
Makes sure that observation units are valid symbols. Symbols loaded from a CSV sometimes have encoding issues. If the symbol isn't valid, show the unit menu instead, so that we can pick a valid unit and correct the uploading error.
|
That latest commit should fix a bug where you couldn't upload a CSV if there are encoding errors in the unit symbols. Now you get a dropdown menu so that you can fix the error. 
|
Update the Group ID column when the selected cohort changes. Use the group column values as the group ID for each row eg. (1, 2, 3) or (Male, Female.)
When there's no selected unit yet, the observation variables menu should show all possible model variables.
Choose a mapped variable, then pick a unit for that variable (if required.)
Without this, saving a protocol from the Trial Design tab will trigger a loop of failed PUT requests.
|
Swtiched the Unit and Variable selections around in the previous screenshot. Choosing a variable, then a unit, feels like a more natural interaction (and reduces the number of choices for unit.) 
There is a bug in this particular case, in that once you've selected a unit, it stops being editable. EDIT: fixed by f90611b. |
When there's only observation values in the uploaded CSV, we should always be able to pick from all available model variables.
Map ADDL and II columns, in the CSV, to repeat doses in the model and simulations.
In certain cases, the observation units menu could disappear after selecting a unit. This should fix that by allowing its visible state to change from false to true, but not from true to false.
In certain cases, the observation units menu could disappear after selecting a unit. This should fix that by allowing its visible state to change from false to true, but not from true to false.
eatyourgreens
force-pushed
the
data-upload
branch
from
06168fc to
2cb0248
Compare
Add checkboxes to control which subject groups are displayed in the simulations view.
eatyourgreens
force-pushed
the
data-upload
branch
from
4dddc44 to
63c6be9
Compare
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.