Introduction to Machine Learning for the Arts

OUTLINE

1: Intro!

• Session 1 (W 9/2): Introduction to Machine Learning
• Session 2 (W 9/9): Image Classification

2: Transfer Learning

• Session 1 (M 9/14): Transfer Learning with Teachable Machine
• Session 2 (W 9/16): Transfer learning with ml5.js

3: Machine Learning Models

• Session 1 (M 9/21): Pre-trained models 1
• Session 2 (W 9/23): Pre-trained models 2

4: Training a Model

• Session 1 (M 9/28): Real-Time Data Collection
• Session 2 (W 9/30): Real-Time Model Training

5: Datasets and Training

• Session 1 (M 10/5): CSV and JSON Data
• Session 2 (W 10/7): More Model Training

6: Convolutional Neural Networks

• Session 1 (M 10/12): Image Data
• Session 2 (W 10/14): Model Training

7: Recurrent Neural Networks

• Session 1 (T 10/19): SketchRNN (Drawing)
• Session 2 (W 10/21): charRNN (Text)

8: Pre-Trained Models with RunwayML

• Session 1 (M 10/28): Introduction to RunwayML
• Session 2 (W 10/30): RunwayML Model Workshop

9: Generative Models with Runway

• Session 1 (M 11/2): Hosted Models and Networking
• Session 2 (W 11/4): Generative Adversarial Networks, Interactive Image Synthesis

10: Model Training with RunwayML

• Session 1 (M 11/9): StyleGAN and Object Detection
• Session 2 (W 11/11): GPT-2

11: Final Project Proposals

• Session 1 (M 11/16): Project Proposals 1
• Session 2 (W 11/18): Project Proposals 2

12: Final Project Development

• Session 1 (M 11/23): Individual Meetings
• Session 2 (M 11/25): Individual Meetings

13: Final Project Development

• Session 1 (M 11/30): TBD 1
• Session 2 (W 12/2): TBD 2

14: Final Project Presentations

• Session 1 (M 12/7): Group 1
• Session 2 (W 12/9): Group 2

CODE OF CONDUCT

Please read and review the ITP/IMA Code of Conduct. The Code of Conduct will be reviewed and discussed as part of the course introduction.

The ITP/IMA Code of Conduct is an evolving work-in-progress document that establishes and communicates the commitment of the ITP/IMA community to uphold a key set of standards and obligations that aim to make ITP/IMA an inclusive and welcoming environment.

COURSE DESCRIPTION

An introductory course designed to provide students with hands-on experience developing creative coding projects with machine learning. The history, theory, and application of machine learning algorithms and related datasets are explored in a laboratory context of experimentation and discussion. Examples and exercises will be demonstrated in JavaScript using the p5.js, ml5.js, and TensorFlow.js libraries. In addition, students will learn to work with open source pre-trained models in the cloud using Runway. Principles of data collection and ethics are introduced. Weekly assignments, team and independent projects, and project reports are required.

COURSE OBJECTIVES

At the completion of this course, the student will:

• Develop an intuition for and high level understanding of core machine learning concepts and algorithms, including supervised learning, unsupervised learning, reinforcement learning, transfer learning, classification, and regression.
• Be able to apply machine learning algorithms to real-time interaction in media art projects using pre-trained models and “transfer learning” in JavaScript and related tools.
• Learn how to collect a custom dataset to train a machine learning model and
• Develop a vocabulary for critical discussions around the social impact and ethics of data collection and application of machine learning algorithms.
• Become familiar with the current landscape of new media art generated from machine learning algorithms. Understand how to use a machine learning model to generate media: words, sound, and images.

EQUIPMENT

You will need a modern laptop (4 years old or younger is a good rule of thumb). Most required software is freely available. The department has all required commercial software installed on laptops available for checkout.

COURSE TEXTS

There is no textbook for the class. Readings and videos will be assigned on the individual session notes pages.

GRADING AND ATTENDANCE

Grades for the course will follow the standard A through F letter grading system and will be determined by the following breakdown:

• 25% Participation
• 50% Assignments (including reading responses and other written work)
• 25% Final project

At most two (2) unexcused absences will be tolerated without effect to your grade. Any more than two (2) unexcused absences will result a lowering of your final grade by one whole grade for each unexcused absence. For example, three (3) unexcused absences will result in your highest possible grade being a B instead of an A. Four (4) unexcused absences will result in your highest possible grade being a C and so on. Six (6) unexcused absences would result in an automatic F for the course. Two (2) late arrivals will count for one (1) absence.

PARTICIPATION:

This class will be highly participatory. You are expected to contribute to discussions, engage in group work, give feedback to your peers, and otherwise fully participate in class.

PHONE, TABLET, LAPTOP, AND OTHER ELECTRONICS USE

Recreational use of phones, music players, video game systems and other devices are forbidden. Laptops and tablets are allowed for note taking, in class work, as well as relevant research only. Activities not related to the class, such as recreational use of the internet, including all social media websites, email and instant messaging, game playing, and work for other classes, will not be permitted. Such activities are disrespectful to the instructor and distracting to others. Your laptop should always be closed whenever a fellow student is presenting.

TEACHING STYLE

Classes will be a mixture of lecture, discussion, hands-on tutorials, homework review, presentations, and group work. You will need to come to class prepared with a laptop and any other supplies specified for that class.

COURSE SCHEDULE

The course will be two (2) times per week for one hour and thirty minutes (1:30) for a total of 14 weeks.

ASSIGNMENTS

There will be regular assignments that are relevant the class material. These assignments must be documented (written description, photos, screenshots, screen recording, code, and video all qualify based on the assignment) on a web platform such as a blog or google doc. You are required to link to your assignment from the course repo (you may choose to use a privately shared google doc or password protected website if you prefer.) The due dates are specified on the assignment page.

It is expected that you will spend 6 to 8 hours a week on the class outside of class itself. This will include reviewing material, reading, watching video, completing assignments and so on. Please budget your time accordingly.

Each assignment will be marked as complete (full credit), partially complete (half credit), or incomplete (no credit). To be complete an assignment should meet the criteria specified in the syllabus including documentation. If significant portions are simply not attempted or the assignment is turned in late (up to 1 week) then it may be marked partially complete. If it is more than a week late, not turned in, or an attempt isn’t made to meet the criteria specified it will be marked incomplete.

Responses to reading and other written assignments are also due in class one week after they are assigned and must also be submitted via the class website. Written assignments are expected to be 200 to 500 words in length unless otherwise specified. Grading will follow the same guidelines as above; on time and meeting the criteria specified will be marked as complete. Late (up to 1 week) or partially completed work will be given half credit. Work that is more than a week late, not turned in, or fails to meet the criteria specified will be given no credit.

STATEMENT OF ACADEMIC INTEGRITY

Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else. More information can be found on Tisch’s page regarding Academic Integrity (http://tisch.nyu.edu/faculty/academic-integ).

USE OF FREE AND OPEN SOURCE CODE EXAMPLES

(The following is adapted from Golan Levin’s Interactivity and Computation Course (Fall 2018) at Carnegie Mellon University.)

You must cite the source of any code you use. Please note the following additional expectations and guidelines:

1. Check the License. When using others' code, pay attention to the license under which it has been released, and be certain to fulfill the terms and requirements of those licenses. Descriptions of common licenses, and their requirements, can be found at choosealicense.com. Some licenses may require permission. If you are confused or aren’t sure how to credit code, ask one of the course instructors and make your best good faith effort. Not properly citing code sources is grounds for a 0 on an assignment.

2. Use Libraries. The use of general, repurposable libraries is strongly encouraged. The people who developed and contributed these components to the community worked hard, often for no pay; acknowledge them by citing their name and linking to their repository.

3. Be Careful. It sometimes happens that an artist places the entire source code for their sketch or artwork online, as a resource from which others can learn. Assignments professors give in new-media arts courses are often similar (e.g. "Clock"); you may also discover the work of a student in some other class or school, who has posted code for a project which responds to a similar assignment. You should probably avoid this code. At the very least, you should be careful about approaching such code for possible re-use. If it is necessary to do so, it is best to extract components that solve a specific technical problem, rather than those parts which operate to create a poetic experience. Your challenge, if and/or when you work with others' code, is to make it your own. It should be clear that downloading an artwork from someone's GitHub and simply changing the colors would be disgracefully lazy. And doing so without proper citation would be outright plagiarism.

ACCESSIBILITY

Academic accommodations are available for students with documented disabilities. Please contact the Moses Center for Students with Disabilities at 212 998-4980 for further information.

WELLNESS

Your health and safety are a priority at NYU. If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYUWellness Exchange 212-443-9999.

TITLE IX

Tisch School of the Arts to dedicated to providing its students with a learning environment that is rigorous, respectful, supportive and nurturing so that they can engage in the free exchange of ideas and commit themselves fully to the study of their discipline. To that end Tisch is committed to enforcing University policies prohibiting all forms of sexual misconduct as well as discrimination on the basis of sex and gender. Detailed information regarding these policies and the resources that are available to students through the Title IX office can be found by using the following link. https://www.nyu.edu/about/policies-guidelines-compliance/equal-opportunity/title9.html