Skip to content

Files

Latest commit

calaldeescalaldees
Updated history doc with new directions
Nov 6, 2019
71b99ef · Nov 6, 2019

History

History
174 lines (147 loc) · 9.21 KB

development_history.md

File metadata and controls

174 lines (147 loc) · 9.21 KB

Development History

These tools evolved from a few simple requirements for a small hobby band project. The tools have continually evolved into a fairly complex tech stack. This document chronicles the changes and thought process's that have lead to current system.

Stage 0 - Initial problem - 2013

Problem

  • A live music performance with interactive video game/music
  • Requirement to trigger a video in sync with our live music (A holographic singer)

Solution

  • Spacebar at the same time on two computers
    • Cuebase
    • vlc

Performance

Stage 1 - October 2014 to September 2015

Problems with Stage 0

  • Don't trigger things by human interaction. Trigger them automatically.
  • Could we trigger/drive stage lights in time to our music without a lighting engineer?

If we could setup a virtual midi port that then trigger's a message to be sent over a network to display a video on another machine that is listening.

Solution

  • displayTrigger
    • Commandline pygame MIDI trigger listened to a midi port and send a json message
    • display was a HTML5/javascript webapp
  • lightingAutomation v0.1
    • Send UDP Artnet3 DMX packets
    • Lights described in yaml
    • Lights visualized by pygame DMXSimulator abstract representation that rendered DMX packets
      • display played an audio file with seek bar to send absolute time to lightingAutomation across websocket bridge
    • Real-time control of lights with physical MIDI-Mixer controller with HTML5/js app
  • pentatonicHero HTML5/js visuals and real-time control of lights

Use

  • Gulbenkian September 2015
    • We could trigger videos, images, sliding text animations, iframes (for other interactive content)
    • Basic DMX lights
      • pentatonicHero realtime lights

Stage 1.5 - October 2015 to July 2016

Problem with Stage 1

  • pygame MIDI trigger was a pain to constantly deploy to the music (Cuebase) machine (as it had no internet connection)
  • Single message bus for all displays. All messages went to every connected device.
  • If connections dropped we had to have access to computers/terminals to fix it

Solution

  • Deprecate pygame MIDI trigger -> javascript web trigger through Chrome MIDI
  • multisocketClient self-healing network connections for websocket and tcp
  • multisocketServer - SubscriptionServer
    • Network bus routing to route messages to specific/multiple listeners/displays
  • Floor prompt subtitle display
    • RaspberyPi 2 with WiFi + Floor Screen

Use

  • Minami March 2016
  • Amecon July 2016
    • Ultra short throw projector mounted on lighting rig

Stage 2 - July 2016 to November 2019

Problems with Stage 1.5

  • lightingAutomation scripting extremely limited
    • Could only tween entire lighting rig state to another state
    • Overly complex nested anonymous function complexity every frame (too complex to test/debug properly)
  • lightingAutomation:DMXSimulator visualization was insufficient
    • Visualizing what was on the display and what pygame lights were doing was an exercise in abstract thinking
    • Clunky, fragile and unusable by anyone other than a linux wizard
  • Needed each display to have a separate browser instance.
    • Could we have multiple displays in a single instance of a browser?
  • Cubase/music machine had lots of MIDI triggers that took a long time to setup.
    • Some tracks had 10 + triggers. Total we had 60+ triggers
    • Constant Cuebase manual maintenance was not committable to repo

Solution

  • stageOrchestration - rename of lightingAutomation
    • Lighting is scripted in python and compiled down to binary frames on disk
      • Live reload/render on change
    • Media triggers (eventline)
      • media trigger in same script as lighting
    • Separate realtime-renderer process takes binary frames off disk + send over network
      • Testable segmented architecture
    • ObjectRelationalMapper for lights
    • Generic python timeline.py - Powerful programmatic animation framework
      • Inspired by other animation frameworks
      • Overload mathematical operators
        • +
        • &
        • *
        • / (split in progress)
  • displayTrigger
    • rewritten in es6, webpack
    • Multiple displays in one browser instance
      • display be bound to arbitrary divs
    • Greensock Animation Project engine for animations
      • Allows timeline seek to set animation state
    • Pixi.js Particle Engine
      • Realtime, Rain, Fire, Firework effects
        • Can control state with GSAP to allow rain to intensify
  • stageViewer - (two es6 components)
    • 3D Stage representation
      • three.js css3d component
      • 3D stage, lights, screens
        • Uses display library bound to screens/divs in 3D scene
    • timeline
      • react component
      • Visual representation of lights
  • superLimitBreakSetup

Use

Stage 3 - November 2019 to present

Problems with Stage 2.0

  • Syncing with the cuebase music machine is still fiddly
    • Individual triggers are difficult to time
      • We could not trigger accurately with a 2 bar count-in as we have overlapping tracks where the clicktrack was just an audio sample
      • The assumption of 2 bar countin was now invalid and defeated the purpose of automatic precaching of videos
  • Difference in production single_triggers to development continuous_triggers (for visualizer)
  • No ability to seek in cuebase project to visualize timing
  • Calibrating sync offsets for visulisations is fiddley, complex and time consuming
  • No live control of lights
    • We did not implement that feature from lightingAutomation
  • multisocetServer was saturated with messagegs at 30fps with text floating point number number in plain text json
    • each frame was hundreds of kb
  • multisocetServer was unreliable and messages would get lost
    • The json message system manually joined multiple messages/packets strings and scanned for }} to denote end of messages. It was a mess.

Solution

  • The cuebase machine broadcasts timecode state continuously
    • SMTPE (protocol designed to sync with analog gear)
    • We can seek in cuebase project
  • timecode_to_track_router translates timecode to a track position *
  • Move to msgpack for messaging
  • Move to a stable profetional message bus

Use

  • unknown