README.md

Presentation: Slides

Coding session:

• HandPose

Examples

• New ml5js next gen handpose (ml5 next gen, All examples)

  • handpose start and stop
  • handpose keypoints
  • handpose parts
  • handpost image
  • New body pose blaze pose Blazepose-keypoints-ml5-next-gen (more points, more stable)

• Handpose_Image

• Handpose_Webcam

• Training hand pose model

• Hand pose tracking + KNN Classification, p5 web editor code, demo video

• Hand pose tracking + Neural Network, demo video

  • Collect data
  • Train the model
  • Run the model

• Multiple hands detection:

• Handpose multiple hands with mediapipe If the webcam video is not loading, try open a incognito window, or set the camera access for the browser.

• KNN Classifier on multiple hands: Change HAND_NUM = 2 if you would like to detect other hand numbers. Once clicking on "Add example" button, it will start collecting data in 5 seconds for 5 seconds.

Resource

• ml5js - handPose source
• tfjs - handPose

Projects

• fingerspelling

Assignment 5

Reading & Video

• ml5.js: Pose Classification with PoseNet and ml5.neuralNetwork()
• Face and hand tracking in the browser with MediaPipe and TensorFlow.js

Coding exercise

• ml5js-next-gen template
• handpose keypoints
• handpose-rect
• handpost-particles

Use the HandPose model, and following the examples and ml5.js documentation experiment with controlling elements of a p5.js sketch (color, geometry, sound, text) with the output of the model. (You may also choose a ml5.js model not covered here if you like!)

Document your p5.js sketch in a blog post and add a link to the post / your p5 sketch on the homework wiki. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.