Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large mixed type dataset logged to the console can be missleading #917

Closed
t20100 opened this issue Jan 4, 2022 · 3 comments · Fixed by #1462
Closed

Large mixed type dataset logged to the console can be missleading #917

t20100 opened this issue Jan 4, 2022 · 3 comments · Fixed by #1462

Comments

@t20100
Copy link
Member

t20100 commented Jan 4, 2022

Describe the bug

Too large to display mixed type dataset leads to the message: The dataset is too big to be displayed and was logged to the console instead.
This seems missleading to me, since not everyone will look into the dev tools to open the console.

To Reproduce

  1. Go to https://h5web.panosc.eu/?file=epics.h5
  2. Click on arrays -> Vdata with mixed types
  3. Message is: The dataset is too big to be displayed and was logged to the console instead.

Expected behaviour

To avoid missleading message:

  • the data could not be displayed at all: The dataset is too big to be displayed., or
  • the data could be displayed on-demand: This dataset is big, display it, if possible with a foldable json viewer.

Context

  • H5Web context: demo
@axelboc
Copy link
Contributor

axelboc commented Jan 6, 2022

The initial problem was that the browser would get stuck while parsing/displaying large JSON outputs. So we could also look for a way to solve this performance issue with an external library that parses JSON asynchronously and can render it in a virtualized way.

@axelboc
Copy link
Contributor

axelboc commented Oct 26, 2022

After some more research, I couldn't find a library for stringifying JSON asynchronously in the browser. I got decent results using greenlet to move the JSON.stringify call to a web worker, but transfering the output string back and displaying it remains an issue. It basically just shifts the bottleneck... I think that stringifying a very large dataset is just a bad idea, full stop.

Perhaps we should consider removing the RawVis altogether and trying to handle more complex compound types with the MatrixVis instead? I'm thinking that for array fields (regardless of shape), the visualization could display the first few numbers - e.g. [1.2, 34.5, 678.9, ...], and for other non-printable dtypes, we could just show <COMPOUND>, <REGION>, etc. or something along those lines.

@axelboc
Copy link
Contributor

axelboc commented Sep 1, 2023

Better fix in #1485: instead of logging the JSON string to the browser console, users can now export the dataset to JSON via the toolbar:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants