04-face-and-hand-models

Face and Hand Models

Slides

Objectives

• Learn about TensorFlow Face Landmarks Detection model and how it works.
• Learn about TensorFlow Hand Pose Detection model and how it works.
• Learn to work with ml5.faceMesh and ml5.handPose.
• Learn to work with 3D graphics in p5.js WEBGL mode.

Lecture Notes

Tools

• ml5.faceMesh Reference
  • ml5.faceMesh Source Code
  • TensorFlow.js Face Landmarks Detection Source Code
  • MediaPipe FaceMesh Model Card
• ml5.handPose Reference
  • ml5.handPose Souce Code
  • TensorFlow.js Hand Pose Detection Source Code
  • Hand Detection/Tracking Model Card

Related Projects

• Keyboard by Use All Five & Google Creative Lab.
• Fingerspelling by Hello Monday.
• Projection Mapping by Nahuel Gerth.
• Customizable AR face masks - Made with TensorFlow.js by Samarth Gulati and Praveen Sinha.
• DeepPrivacy2 - A Toolbox for Realistic Image Anonymization by Hukkelås, Håkon and Lindseth, Frank.

Music

• LipSync by YouTube by Google PI & bit.studio.
• Eye Conductor by Andreas Refsgaard.
• Sampler by Use All Five & Google Creative Lab.
• Mouth-Controlled Synthesizer with FaceMesh by Jack B. Du. [ Live Demo ]
• Melody Painter with HandPose by Jack B. Du. [ Live Demo ]

Playful

• Finger Talk by Future Sketches.
• Look At You by Alan Ren & Kyrie Yang.
• Bubbles by Nahuel Gerth.
• Finger Numerals by Nahuel Gerth.

Code Examples

In class examples

• HandPose - pinch painting
• FaceMesh - image texture
• FaceMesh - pixel colors
• FaceMesh - lip contours

Core Functionalities

• FaceMesh - Single Image
• FaceMesh - Face Keypoints
• FaceMesh - Face Bounding Box
• FaceMesh - Parts Keypoints
• FaceMesh - Parts Bounding Box
• FaceMesh - Shapes from Parts
• HandPose - Single Image
• HandPose - Keypoints
• HandPose - Keypoints 3D
• HandPose - Parts
• HandPose - Detect Start and Stop

Creative Applications

• FaceMesh - Emoji Face by Jack Du
• FaceMesh - Triangulation by Jack Du
• FaceMesh - Face Masks by Jack Du
• HandPose - Particles by Jack Du
• HandPose - Quadrilateral by Jack Du
• HandPose - Quadrilateral with Texture by Jack Du

Technical Blog Posts

• Face and hand tracking in the browser with MediaPipe and TensorFlow.js by Ann Yuan and Andrey Vakunov. [ Original Paper ]
• On-Device, Real-Time Hand Tracking with MediaPipe by Valentin Bazarevsky and Fan Zhang.
• 3D Hand Pose with MediaPipe and TensorFlow.js by Valentin Bazarevsky, Ivan Grishchenko, Eduard Gabriel Bazavan, Andrei Zanfir, Mihai Zanfir, Jiuqiang Tang, Jason Mayes, Ahmed Sabie.

Video Tutorials

• ml5.js FaceMesh Creative Sketch - video tutorial with Patt Vira.

Assignment 4

1. Research and find a project (experiments, websites, art installations, games, etc) that utilizes machine learning in a creative way. Consider the following:
   • What type of machine learning models did the creator use?
   • What data might have been used to train the machine learning model?
   • Why did the creator of the project choose to use this machine learning model?
2. Pick one of the models above (FaceMesh or HandPose), following the examples above and the ml5.js documentation, experiment with controlling elements of a p5.js sketch (color, geometry, sound, text, etc) with the output of the model. Try to create an interaction that is surprising or one that is inspired by the project you find.
3. Document your research, your response, and your p5.js sketch in a blog post. Add a link to the post and your p5.js sketch on the Assignment 4 Wiki page. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.