09_runwayML

Introduction to RunwayML

Session A: RunwayML 101

Objectives

• Become familiar with RunwayML and understand how to install and run new models locally.

Background on RunwayML

• Machine Learning En Plein Air: Building accessible tools for artists by Cristóbal Valenzuela
• Runway: Adding artificial intelligence capabilities to design and creative platforms by Cristóbal Valenzuela, Alejandro Matamala, and Anastasis Germanidis

Resources

• RunwayML.com
• Installation
• Introductory Guide to Using Models
• Model Lists:
  • Detect Objects
  • Transform Content
  • Generate Media
• How to Chain Models Together
• Videos
  • Getting Started with RunwayML
  • The Coding Train: Introduction to Runway: Machine Learning for Creators (Part 1)
  • Intro to Machine Learning with Gene Kogan

Session B: Inputs and Outputs

Objectives

• Learn how to integrate RunwayML with JavaScript and other software applications.

Resources

• Interact with models in RunwayML over the network
• Networking examples and tutorials, organized by software application

Code Examples

• p5.js Send text to RunwayML: Generate an image with AttnGAN
• p5.js Send webcam image to RunwayML: Photo booth with Fast Style Transfer
• p5.js Send uploaded image file to RunwayML: Generate a caption with im2txt
• Multiple Networking Examples from RunwayML

Tip: Before running your p5.js sketch, choose the model checkpoint in RunwayML (if applicable) and then start the model.

Related Projects

• Projects Made with RunwayML
• No Machine is an Island by Yuxi Liu and Larissa Pschetz
• Introspectons by Philipp Schmitt

Assignment (Due Sunday, Nov 3, 6pm):

Reading

• Machine Learning En Plein Air: Building accessible tools for artists by Cristóbal Valenzuela
• Runway: Adding artificial intelligence capabilities to design and creative platforms by Cristóbal Valenzuela, Alejandro Matamala, and Anastasis Germanidis

Instructions

1. Try a model available in RunwayML that was not demonstrated in class.
2. Create a p5.js sketch that receives data from RunwayML (using any model).
3. Optionally (instead of or in addition to 2), send data to RunwayML or use another programming environment or software tool besides p5.js.
4. Document your experience working with RunwayML in a blog post. Include screenshots and screen captures of your workflow. Compare and constrast working with RunwayML as a tool for machine learning as related to ml5.js, python, and any other tools explored this semester.