Merge branch 'master' of github.com:beacon-biosignals/OndaEDF.jl into…

… dfk/export
beacon-biosignals · Jul 17, 2023 · dc33c3f · dc33c3f
2 parents b3d72ba + 8c3ac67
commit dc33c3f
Show file tree

Hide file tree

Showing 8 changed files with 414 additions and 197 deletions.
diff --git a/Project.toml b/Project.toml
@@ -1,7 +1,7 @@
 name = "OndaEDF"
 uuid = "e3ed2cd1-99bf-415e-bb8f-38f4b42a544e"
 authors = ["Beacon Biosignals, Inc."]
-version = "0.11.6"
+version = "0.11.9"
 
 [deps]
 Compat = "34da2185-b29b-5c13-b0c7-acf172513d20"

diff --git a/README.md b/README.md
@@ -4,120 +4,4 @@
 [![codecov](https://codecov.io/gh/beacon-biosignals/OndaEDF.jl/branch/master/graph/badge.svg?token=7oZhx7P9kq)](https://codecov.io/gh/beacon-biosignals/OndaEDF.jl)
 [![](https://img.shields.io/badge/docs-stable-blue.svg)](https://beacon-biosignals.github.io/OndaEDF.jl/stable)
 
-OndaEDF provides functionality to convert/import/export EDF files to/from Onda recordings; see the `edf_to_onda_samples`, `edf_to_onda_annotations`, and `onda_to_edf` docs/tests for details.
-
-## EDF Formatting Expectations
-
-While OndaEDF attempts to be somewhat robust to more common nonstandard/noncompliant quirks that often appear in EDF files "in the wild", the package generally expects the caller to perform any necessary preprocessing to their EDFs to ensure they comply with the EDF/EDF+ standards/specifications, as well as a few other expectations to facilitate conversion to Onda.
-
-These expectations are as follows:
-
-- `EDF.Signal` labels follow the standard "$TYPE $SPECIFICATION" structure defined by [the EDF standards](https://www.edfplus.info/specs/edftexts.html), and signal types documented by the aforementioned standard (EEG, EKG, etc.) are labeled in compliance with naming conventions defined by the standard.
-- `EDF.Signal`s that are matched as channels to a common `Onda.Signal` must have the same `physical_dimension`, sample rate, and sample count.
-- The `physical_dimension` field for any given `EDF.Signal` is a value supported by `OndaEDF.STANDARD_UNITS`.
-
-Note that callers can additionally use the `labels` argument to `edf_to_onda_signals` to workaround some of these expectations; see the `plan_edf_to_onda_samples` docstring for more details.
-
-## Fine-grained control over `Signal` processing: `plan_edf_to_onda_samples` and `edf_to_onda_samples(edf, plan)`
-
-Because the default labels do not always match EDF files as seen in the wild, OndaEDF provides additional tools for creating, inspecting, manipulating, and recording the `EDF.Signal`-to-`Onda.Samples` mapping.
-In fact, the high-level function `edf_to_onda_samples` contains very few lines of code:
-```julia
-function edf_to_onda_samples(edf::EDF.File; kwargs...)
-    signals_plan = plan_edf_to_onda_samples(edf; kwargs...)
-    EDF.read!(edf)
-    samples, exec_plan = edf_to_onda_samples(edf, signals_plan)
-    return samples, exec_plan
-end
-```
-The executed plan as returned is a [Tables.jl](https://github.com/JuliaData/Tables.jl)-compatible table, with one row per `EDF.Signal` and columns for
-- the fields of the original `EDF.SignalHeader`
-- the fields of the generated `Onda.SamplesInfoV2`, including
-  - `:sensor_type`, the extracted sensor type
-  - `:channel`, the extracted channel label (instead of `:channels`, since each `EDF.Signal` is exactly one channel in `Onda.Samples`)
-- `:edf_signal_index`, the 1-based numerical index of the source signal in `edf.signals`
-- `:onda_signal_index`, the ordinal index of the resulting samples (not necessarily the index into `samples`, since some groups might be skipped)
-- `:error`, any errors that were caught during planning and/or execution.
-
-This table could, for instance, be recorded somewhere during ingest of large or complex datasets, as a record of how the `Onda.Samples` were generated.
-OndaEDF includes the OndaEDFSchemas sub-package, which provides [Legolas.jl Schemas](https://beacon-biosignals.github.io/Legolas.jl/stable/#Legolas-Schemas-and-Rows-1) for this purpose: `Plan` (`"ondaedf.plan@1"`) which corresponds to the columns for a single EDF signal-to-Onda channel conversion, and `FilePlan` (`"ondaedf.file-plan@1"`) which includes the additional file-level linkage columns `:edf_signal_index` and `:onda_signal_index`.
-The `write_plan(io_or_path, plan_table)` provides a wrapper around [`Legolas.write`](https://beacon-biosignals.github.io/Legolas.jl/stable/#Legolas.write) which writes a table following the `"ondaedf.file-plan@1"` schema to a generic path-like destination.
-If you are including the plan tables in a dataset, you can add a dependency on OndaEDFSchemas to make sure the relevant schemas are defined without the full OndaEDF dependency.
-
-It can also be manipulated programmatically, by manually or semi-automatically modifying the `:sensor_type`, `:channel`, or other columns to correct for missed signals by the default labels (for which `:sensor_type` and `:channel` will be `missing`).
-We give two examples of how such a workflow might work here: one where the plan is modified before being executed, and another where EDF signal headers are be _preprocessed_ before the plan is constructed.
-
-### Modification of a plan
-
-For instance, some EEG datasets have the physical units set to millivolts, but the signals are usually better measured in microvolts.
-During import, you want to correct this by adjusting the encoding settings used by Onda to store samples, by scaling the sample offset and resolution by 1000 and setting the physical units.
-This can be accomplished by modifying the rows of the plan like so:
-
-```julia
-edf = EDF.File(my_edf_file_path)
-plans = plan_edf_to_onda_samples(edf; label=my_labels)
-
-function fix_millivolts(plan)
-    if plan.sample_unit == "millivolt" && plan.sensor_type == "eeg"
-        sample_resolution_in_unit = plan.sample_resolution_in_unit * 1000
-        sample_offset_in_unit = plan.sample_offset_in_unit * 1000
-        return Tables.rowmerge(plan; sample_unit="microvolt",
-                               sample_resolution_in_unit,
-                               sample_offset_in_unit)
-    else
-        return plan
-    end
-end
-
-new_plan = map(fix_millivolts, Tables.rows(plans))
-samples, plan_executed = edf_to_onda_samples(edf, new_plan)
-```
-
-As another, similar example, sometimes EMG channels get recorded with different physical units.
-In such a case, OndaEDF cannot merge these channels and will create multiple separate `Samples` objects which each have `sensor_type = "emg"`.
-This can be corrected in a similar way, for exmaple by converting millivolts to microvolts (adjusting of course depending on the nature of your dataset) and re-grouping into Onda samples:
-```julia
-edf = EDF.File(my_edf_file_path)
-plans = plan_edf_to_onda_samples(edf; label=my_labels)
-
-function fix_emg(plan)
-    if plan.sensor_type == "emg"
-        if plan.sample_unit == "millivolt"
-            sample_resolution_in_unit = plan.sample_resolution_in_unit * 1000
-            sample_offset_in_unit = plan.sample_offset_in_unit * 1000
-            plan = Tables.rowmerge(plan; sample_unit="microvolt",
-                                   sample_resolution_in_unit,
-                                   sample_offset_in_unit)
-        end
-        return plan
-    else
-        return plan
-    end
-end
-
-new_plan = map(fix_emg, Tables.rows(plans))
-# re-compute the grouping of EDF signals into Onda signals:
-new_plan = plan_edf_to_onda_samples_groups(new_plan)
-samples, plan_executed = edf_to_onda_samples(edf, new_plan)
-```
-
-### Pre-processing signal headers
-
-Sometimes non-standard usage of the label and transducer type fields makes automatic matching difficult.
-In such cases, you can _preprocess_ the signal headers before generating a plan.
-For example, in a situation where the transducer type and labels are switched, you can switch them back before planning:
-
-```julia
-edf = EDF.File(my_edf_file_path)
-
-function corrected_header(signal::EDF.Signal)
-    header = signal.header
-    return Tables.rowmerge(header; 
-                           label=header.transducer_type, 
-                           transducer_type=header.label)
-end
-
-plans = map(plan_edf_to_onda_samples ∘ corrected_header, edf.signals)
-grouped_plans = plan_edf_to_onda_samples_groups(plans)
-samples, plan_executed = edf_to_onda_samples(edf, grouped_plans)
-```
+OndaEDF provides functionality to convert/import/export EDF files to/from [Onda format](https://github.com/beacon-biosignals/Onda.jl).
diff --git a/docs/make.jl b/docs/make.jl
@@ -1,11 +1,12 @@
 using OndaEDF
-using OndaEDFSchemas
 using Documenter
 
-makedocs(modules=[OndaEDF, OndaEDFSchemas],
+makedocs(modules=[OndaEDF, OndaEDF.OndaEDFSchemas],
          sitename="OndaEDF",
          authors="Beacon Biosignals and other contributors",
-         pages=["API Documentation" => "index.md"])
+         pages=["OndaEDF" => "index.md",
+                "Converting from EDF" => "convert-to-onda.md",
+                "API Documentation" => "api.md"])
 
 deploydocs(repo="github.com/beacon-biosignals/OndaEDF.jl.git",
            push_preview=true)
diff --git a/docs/src/api.md b/docs/src/api.md
@@ -0,0 +1,75 @@
+# API Documentation
+
+```@meta
+CurrentModule = OndaEDF
+```
+
+## Import EDF to Onda
+
+OndaEDF.jl prefers "self-service" import over "automagic", and provides
+functionality to extract
+[`Onda.Samples`](https://beacon-biosignals.github.io/Onda.jl/stable/#Samples-1)
+and [`EDFAnnotationV1`](@ref)s (which extend 
+[`Onda.AnnotationV1`](https://beacon-biosignals.github.io/Onda.jl/stable/#Onda.AnnotationV1)s)
+from an `EDF.File`.  These can be written to disk (with
+[`Onda.store`](https://beacon-biosignals.github.io/Onda.jl/stable/#Onda.store) /
+[`Legolas.write`](https://beacon-biosignals.github.io/Legolas.jl/stable/#Legolas.write)
+or manipulated in memory as desired.
+
+### Import signal data as `Samples`
+
+```@docs
+edf_to_onda_samples
+plan_edf_to_onda_samples
+plan_edf_to_onda_samples_groups
+```
+
+### Import annotations
+
+```@docs
+edf_to_onda_annotations
+EDFAnnotationV1
+```
+
+### Import plan table schemas
+
+```@docs
+PlanV2
+FilePlanV2
+write_plan
+```
+
+### Full-service import
+
+For a more "full-service" experience, OndaEDF.jl also provides functionality to
+extract `Onda.Samples` and `EDFAnnotationV1`s and then write them to disk:
+
+```@docs
+store_edf_as_onda
+```
+
+### Internal import utilities
+
+```@docs
+OndaEDF.match_edf_label
+OndaEDF.merge_samples_info
+OndaEDF.onda_samples_from_edf_signals
+OndaEDF.promote_encodings
+```
+
+## Export EDF from Onda
+
+```@docs
+onda_to_edf
+```
+
+## Deprecations
+
+To support deserializing plan tables generated with old versions of OndaEDF +
+Onda, the following schemas are provided.  These are deprecated and will be
+removed in a future release.
+
+```@docs
+PlanV1
+FilePlanV1
+```