Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix sample encoding for export #76

Merged
merged 16 commits into from
Jul 18, 2023
Merged

fix sample encoding for export #76

merged 16 commits into from
Jul 18, 2023

Conversation

kleinschmidt
Copy link
Member

@kleinschmidt kleinschmidt commented Jun 26, 2023

fixes #75 and cleans up a lot of crufty old code that's gotten out of sync with both Onda and EDF.

It adds a new internal function reencode_samples that takes Onda.Samples and returns new samples that are encoded in an EDF-friendly way (either directly as Int16/the requested type, or as something that can be converted losslessly to that type). It tries very hard to not change the encoding if it's possible, based on the typemin/max of the input sample_type, adn failing that, the acutal min/max of the encoded values (for <:Integer sample types). If it's not possible to losslessly represent the encoded values as Int16, it picks a new quantization setting based on the min/max of the decoded values (fitting those to typemin/max of Int16 or whatever teh desired type is) and returning the re-encoded samples.

It also adds much more thorough tests of the export behavior, testing every type that Onda declares is allowed in Onda.LPCM_SAMPLE_TYPE_UNION.

While adding tests, I found that it's not possible for EDF.jl to decode things where the physical min/max are anywhere near the Float32 min/max finite values, see beacon-biosignals/EDF.jl#67 As a result, some of the re-encoding tests do not hit the true typemin/max of the Onda sample type, but I think that's okay. We're still in much better shape here than we were.

I'm putting this on hold for now but want to get a draft PR up


@kleinschmidt kleinschmidt marked this pull request as ready for review June 28, 2023 21:23
@kleinschmidt kleinschmidt requested a review from palday June 28, 2023 21:23
src/export_edf.jl Outdated Show resolved Hide resolved
Co-authored-by: Phillip Alday <[email protected]>
@kleinschmidt kleinschmidt requested a review from palday July 17, 2023 16:21
src/export_edf.jl Outdated Show resolved Hide resolved
src/export_edf.jl Outdated Show resolved Hide resolved
src/export_edf.jl Outdated Show resolved Hide resolved
# converting everything to Int16 in the actual export.
samples = encode(samples)
new_info = SamplesInfoV2(Tables.rowmerge(samples.info; sample_type))
return Samples(samples.data, new_info, true; validate=false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mildly cursed, but makes sense

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, hence the XXX comment above, wanted to make sure this was known to be cursed

src/export_edf.jl Outdated Show resolved Hide resolved
src/export_edf.jl Outdated Show resolved Hide resolved
sample_data = vec(samples[channel_name, :].data)
# manually convert here in case we have input samples whose encoded
# values are convertible losslessly to Int16:
sample_data = Int16.(vec(samples[channel_name, :].data))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems unlikely but I guess it doesn't hurt 🤷

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, this is actually necessary when you have input samples with Int8/UInt8 encoding (which is the "mildly cursed" bit you called out above)

@palday
Copy link
Member

palday commented Jul 17, 2023

I kinda think that this should be a method of Onda.encode_sample. To that end, you might want to add a comment to the docstring that this is an internal function and could disappear in the future (e.g. if moved into Onda itself).

@kleinschmidt
Copy link
Member Author

I kinda think that this should be a method of Onda.encode_sample. To that end, you might want to add a comment to the docstring that this is an internal function and could disappear in the future (e.g. if moved into Onda itself).

Hm, I'm not sure about this...it's not part of the Onda documented API so I'm a bit reluctant, and it's also not really doing the same thing (IIRC, Onda.encode_sample is actually doing the encoding given specified parameters; we're choosing parameters here and encoding a whole Samples, not just a single sample...)

@kleinschmidt kleinschmidt merged commit 7f34243 into master Jul 18, 2023
@kleinschmidt kleinschmidt deleted the dfk/export branch July 18, 2023 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cannot export Onda Samples with sample_type = UInt16
2 participants