Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Neural network support webgpu #138

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ziyuan-linn
Copy link
Member

This PR adds webgpu support for neuralNetwork so it can be used without called ml5.setBackend.

Changes:

  • bump TensorFlow.js dependencies to a version that supports readSync as @lindapaiste suggested.
  • add await tf.ready() to the init function of DiyNeuralNetwork @lindapaiste suggested.
  • change copy and crossover to async, support both promise and callback interface.
  • Update example sketches accordingly

The ml5.neuralNetwork function is technically an async operation like with other models. It should be called with a callback function or placed within p5 preload function. Our examples, however, have been treating ml5.neuralNetwork like a synchronous function since it sometimes (when not loading data) behaves synchronously. The changes from this PR adds await tf.ready() thus making it definitively asynchronous.

The functions copy and crossover also instantiate the neuralNetwork object, which is why they have to be async.

I updated all example sketches, removing ml5.setBackend and moving the ml5.nerualNetwork function to preload for non neuroevolution examples.

The neuroevolution examples are kept on cpu backend for performance. Adapting for async copy and crossover slightly complicates those examples but I think they are still fairly readable. The ml5.neuralNetwork calls in neuroevolution examples are still synchronous, but they seem to not cause any error for now. I will try to come up with a clean way to adapt them to async.

@ziyuan-linn ziyuan-linn linked an issue Jun 15, 2024 that may be closed by this pull request
@shiffman
Copy link
Member

Hi @ziyuan-linn, maybe we can meet to discuss these! I would very much prefer to keep the synchronous calls for neural network especially for the neuroevolution examples. I originally made asychrnous versions of these examples for teaching the class and the students really struggled with them along managing the draw() loop and other aspects of working with neural networks inside animated systems. The examples are also going to be printed with the synchronous code in the nature of code book in chapters 10 and 11 (along with an explanation.)

I realize it's not typical to use synchronous calls for ML models, but since these are tiny neural networks running on CPU for these p5.js examples I think it's worth making an exception and hacking our way around it!

Later, when p5.js adopts promises I think we could reconsider!

@jchwenger
Copy link

I stumbled across this, as @ziyuan-linn mentioned, that explanation was nice and clear, @shiffman, thanks!

Out of curiosity, is the adoption of promises in p5 in the works / planned?

@ziyuan-linn
Copy link
Member Author

Out of curiosity, is the adoption of promises in p5 in the works / planned?

Looks like it is! https://github.com/processing/p5.js/blob/dev-2.0/rfc_p5js_2.md#async

@jchwenger
Copy link

Awesome, thanks @ziyuan-linn !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

setBackend() required for ml5.neuralNetwork() bug
3 participants