Extension to visualize metadata provided externally

[leman31]

No description provided.

[lutzroeder]

@leman31 can you share a model and metadata files to test this implementation. How are the metadata files generated? Can you also start reworking the code from directly modifying the DOM to align with the underlying object model. Security related removals need to be reverted. Assuming you read the threads attached to #1240 and #1241?

[mciprian13]

@lutzroeder Hello, I am the tech lead of a team in NXP and I decided to ask our student (@leman31) to implement this feature which helps us a lot in model debugging purposes.
A reminder: we discussed extensively about this feature here.
I will not go again through all the details, just a summary:

• This feature allows to add extra annotations on top of a model that is provided through a separate file (not the model itself).
• This allows to add annotations on both tensors and operators, such as:
  • Metadata as key-value pairs.
  • Actions (button) that trigger specific shell commands or javascripts.
  • Opening a text or image when hovering over the tensor/operator with the mouse.
    With these very simple, generic but versatile commands a lot of functionalities can be obtained.

To be noted that this mechanism was tested only with TFLite models (which was our usecase). This has to do with how we reference tensor/operator IDs in the metadata file using the integer identifier field which is TFLite specific. Maybe this can be generalized easily to other formats as well.

Please find in the attached archive what you are looking for:
Package.zip
You can find:

• A readme which describes how the feature works
• Some tests including a TFLite model and some metadata text files (some of them might contain some invalid stuff for negative testing or some invalid paths ... but you should get a quick feel how it all works and change it). These are simple examples to understand how it works.
  Maybe you can advise us where is the best place to put that document (Readme).

The metadata text file in our specific case is generated by our testing infrastructure (which is custom/internal). In our specific case it is extremly useful to export information from the runtime related to the errors we get at each tensor (tensor metrics as text and tensor error heatmaps as images). Thus we annotate the graph and we can see extra text and images for each tensor to know whether our runtime produces extra errors relative to vanilla TFLite runtime. This is super cool and very useful and I am really confident that others will consider this useful for their own purposes.

Again, I would not insist on how these files are produces and trying to impose a standard because, in principle, the metadata text file syntax is very simple and the features are very generic and thus anyone who likes these features and find them useful can use them easily in their own infrastructure. We are open to changing the syntax of the metadata file if you want although we chose the simplest syntax possible.

We would love to contribute this feature to the community and even improve it based on feedback with the only request that the functionality is maintained (afterall this is the philosophy of open-source).

Once we decide the strategy, we can discuss the technical parts (go on with the code review, update based on comments, etc).
What do you think?

[lutzroeder]

@mciprian13 @leman31 thank you for sharing the sample files. Let's start by getting to a point where I can try out the feature. 1. Fix the build errors 2. starting the app currently does not show the feature (on macOS) and 3. opening via browser is showing a black screen.

The pull request also removed multiple security checks and these changes need to get reverted.

As mentioned earlier, modifying the DOM tree directly and side stepping the object model might need to be addressed differently, let's discuss once the pull request is in a working state.

[mciprian13]

@lutzroeder Indeed, we tested only the application executable and also the web application on Windows. We will try to get familiar with the CI/CD jobs for the other environments.
We can send you a windows exe installer if you want to try out the features (or a video) :)
We removed some of the code because the features were not working in the browser. I totally agree, we need to revisit that.
Thanks for your availability. For now I need to wait for the student to return from the exam session to continue.
Will keep you posted.