03_ml_models

Applications of Machine Learning models

Session A: Other pre-trained models

Objectives:

• Understanding how to use pre-trained models other than image classification.
• Ability to work with PoseNet and ml5.js
• Ability to work with Object Detection Models (CoCoSSD).
• Ability to work with Image Segmentation models (UNet and BodyPix) and ml5.js.
• Week 3 Slides

Models

• PoseNet
  • Read Real-Time Human Pose Estimation in the Browser with TensorFlow.js by Dan Oved, with editing and illustrations by Irene Alvarado and Alexis Gallo.
• Object Detection (documentation in progress)
• UNet
  • Trained by Zaid Alyafeai using mut1ny - Face/Head segmentation dataset
• BodyPix
  • Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js

ml5 video tutorials

• ml5.js Pose Estimation with PoseNet, code

ml5 code editor examples

• PoseNet Webcam Part Selection
• PoseNet Webcam Full Skeleton
• PoseNet Single Image
• Object Detection - CoCoSSD
• UNet Image Segmentation
• BodyPix Image Segmentation

Related Projects

• Maya Man's PoseNet sketchbook, Body, Movement, Language: A.I. Sketches with Bill T. Jones
• PoseNet Whack a Mole by Vibert Thio
• Sidewalk Orchestra by Cristóbal Valenzuela
• Pose Music by Tero Parviainen
• The Treachery of Sanctuary by Chris Milk
• Gait Analysis from runnersneed

Session B

Data and Model Biographies

• Slides from Ellen Nickles
• The Future Is Here!, a commission for The Photographers’ Gallery which examines the process of dataset creation by Mimi Ọnụọha
• Feminist Data Set tool kit to interrogate every step of the AI process by Caroline Sinders
• Ted Talk: The era of blind faith in big data must end and Cathy O'Neil's Risk Consulting & Algorithmic Auditing firm

Assignment 3, due Monday Sept 28:

2. Read Mixing movement and machine by Maya Man
3. Read Humans of AI by Philipp Schmitt
4. Pick one of the models described in Ellen Nickles slides (PoseNet, UNET, BodyPix, CoCoSSD) and following the examples above and the ml5.js documentation and experiment with controlling elements of a p5.js sketch (color, geometry, sound, text) with the output of the model. (You may also choose a ml5.js model not covered in Ellen's slides)
5. Read the Model and Data Biography for the model you picked and consider the following:
   • What questions do you still have about the model and the associated data? Are there elements you would propose including in the biography?
   • How does understanding the provenance of the model and its data inform your creative process?
6. Document your response to the Data and Model biographies as well as your p5.js sketch in a blog post and add a link to the post / your p5 sketch on the Assignment 3 Wiki. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.