scratchpad.txt

##  Workshop of 2 hours  
  Michal loves the idea 
    and also thinks that DeepDream is a really interesting topic
  Thumbs up from organiser on DeepLearning idea
    Not too worried about track selection, etc - all a work-in-progress

Plan
  -- hand out VMs on sticks
  
  NN intro
  chain rule
    demo with convnet.js cat picture
  supervised learning
    demo with convnet.js MNIST
  
  network configurations
    MLP
    CNN
    RNN
  
  merits of various software frameworks 
    direct programming
    meta-programming
    meta-meta-programming

  Concurrent with workshop hands-on:
    explain frontiers (eg: Futurists talk)
  
  Concurrent with workshop hands-on:
    merits of various software frameworks 
      convnet.js (!)
      caffe (C++)
        python
      torch
        lua
      theano (Python)
        python, GPU + CPU
      tensorflow (C++)
        python and C++, GPUs + CPUs
      
      ease-of-use layers
        keras
        lasagne
        tensorflow-slim
        
    TODO demo of tensorboard 
      IDEA : show tensorboard for MNIST ++ ?
  
    demo of Lasagne for MNIST MLP
      DONE
    
    demo of Lasagne for MNIST CNN
      DONE
    
    demo of VM for ImageNet recognition
      WORKS: lasagne can load+run Inception3 in Jupyter inside 2Gb RAM 
        Picture of inception3 network included in : https://github.com/google/inception/blob/master/inception.ipynb
          Added to presentation
      
      
    Commerce section
      demo of VM with ImageNet / cutoff == 'commercial'
        WORKS : logistic regression of several car images
  
  
    Art section : Intro : http://fastml.com/deep-nets-generating-stuff/
      demo of VM with style-transfer == 'art' 
        Lasagne :  (theano ~ secs per iteration)  ~600Mb peak memory
        https://github.com/Lasagne/Recipes/blob/master/examples/styletransfer/Art%20Style%20Transfer.ipynb
        ==?== (lasagne ~deep-forger): http://www.kdnuggets.com/2015/12/deep-learning-art-forgery.html
        TEST : Can this be run in reasonable time?
        TEST : Does this look Ok with Inception3?

        http://ostagram.ru/static_pages/lenta?last_days=30            
          ->  https://www.facebook.com/ostagram/


      ?demo of VM with deep dream == 'art'
        Original Google article (Caffe) :
          https://github.com/google/deepdream/blob/master/dream.ipynb
        ?TEST : Can this be run in reasonable time 
        Keras :  (tensorflow backend ~ 70secs, ~1.1Gb peak RAM,  theano ~ 570secs per iteration, ~?350Mb peak RAM)
          https://github.com/fchollet/keras/blob/master/examples/deep_dream.py
            :: http://i.imgur.com/FX6ROg9.jpg
            
        ?TEST : Does this look Ok with Inception3?
        Lasagne :
          https://github.com/317070/Twitch-plays-LSD-neural-net 
          == https://317070.github.io/Dream/


      demo of LSTM word thingy
        IMPLEMENT : Need suitable text data
        TEST : Can this be run in reasonable time?
    
        https://github.com/ebenolson/pydata2015/blob/master/4%20-%20Recurrent%20Networks/RNN%20Character%20Model%20-%202%20Layer.ipynb
        xgit : Recipes : examples : lstm_text_generation.py
        
        Problem : Takes too long to train...

      Alternative RNN ideas:        
        http://jiwonkim.org/awesome-rnn/

    Word Embedding
      Arguable whether it's really Deep Learning
      OTOH, it is a 'learned layer' that has arisen from DL research
      Either : Need word corpus (and patience)
      Or     : Precalculated vectors
    
    Deep Q-Learning
      https://cs.stanford.edu/people/karpathy/convnetjs/demo/rldemo.html
      http://maciejjaskowski.github.io/2016/03/09/space-invaders.html
      https://github.com/maciejjaskowski/deep-q-learning
        -- theano / lasagne
        -- needs non-free Atari ROM image
        
      https://github.com/yandexdataschool/AgentNet
    
    
### Topics within Deep Learning     
#### ConvNets:

  Pros 
    More materials available
    Different angles : Art, Commerce
    Can do some simple 'real' training (MNIST)
    Can also use pre-built model (ImageNet / Inception-3)
  Cons
    'Simple stuff'
    Overlap with potential customers?
    Google DeepDreaming may not be ready : CHECK THIS
  Ideas for progression :
    Stanford Seminar - Oriol Vinyals of Google
      https://youtu.be/UAq961jQjYg

#### RNNs / NLP:

  Pros 
    More similar to current work
  Cons
    Difficult to set up satisfying motivation / experiments

#### Reinforcement Learning:

  Pros 
    Go = VerySexy
    
  Cons
    Not so clearly 'deep learning'
    Very Difficult to set up satisfying motivation / experiments
    Someone else might do it...





More generally    

  JS in the browser :
    Pros
      CPU-only
      Single-threaded
      Avoids thumb-drive distribution
    Cons
      Not Python
      No GPU or multi-core - though this is (basically) same as with VM or docker
      Not iPython 
    Code
      https://github.com/karpathy/convnetjs
      https://github.com/karpathy/reinforcejs
        There is a new paper in regards to deep reinforcements learning in continous spaces by deepmind. 
        Continuous control with deep reinforcements learning. Is there plans to add this in code form. Many thanks Andrew.

      http://cs.stanford.edu/people/karpathy/convnetjs/demo/image_regression.html
        Cat picture reconstruction

      http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html      
        MNIST



  iPython as a GUI
    https://github.com/donnemartin/data-science-ipython-notebooks
    Deep Dream (Caffe)
      https://github.com/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/deep-dream/dream.ipynb
        http://googleresearch.blogspot.sg/2015/07/deepdream-code-example-for-visualizing.html
      http://www.pyimagesearch.com/2015/07/06/bat-country-an-extendible-lightweight-python-package-for-deep-dreaming-with-caffe-and-convolutional-neural-networks/#show_and_tell  
        https://github.com/jrosebr1/bat-country/blob/master/demo.py


  Tensorflow in iPython notebook :
    https://github.com/pkmital/tensorflow_tutorials
    MNIST:
      https://github.com/pkmital/tensorflow_tutorials/blob/master/notebooks/05_basic_convnet.ipynb
    TensorFlow
      https://github.com/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/5_word2vec.ipynb


  Google Inception network notebooks :
    https://github.com/google/inception
    
  Keras on TensorFlow (descriptions)
    *  http://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
    *  TensorFlow backend for Keras is pretty functional : 
       *  http://blog.keras.io/keras-now-running-on-tensorflow.html

  Keras MNIST:
    *  98.40 : https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py  
    *  99.25 : https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py

  Keras MNIST transfer learning :
    *  Learn features on {0,1,2,3,4} and then fine tune final dense layers against {5,6,7,8,9} 
       *  https://github.com/fchollet/keras/blob/master/examples/mnist_transfer_cnn.py

  Keras may have problems importing VGG16 into TensorFlow:
    *  https://github.com/fchollet/keras/issues/1135

  Show VGG16 layers :
    *  https://github.com/fchollet/keras/blob/master/examples/conv_filter_visualization.py


  Something akin to DeepDream using Keras on TensorFlow :: Keras
    *  https://github.com/fchollet/keras/blob/master/examples/deep_dream.py

  Neural Style transfer :
    *  https://github.com/fchollet/keras/blob/master/examples/neural_style_transfer.py  
  
  MAY BE PROBLEMATIC :  Code in C++
    How to Classify Images with TensorFlow 
      *  http://googleresearch.blogspot.sg/2015/12/how-to-classify-images-with-tensorflow.html
        *  C++ Source :: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/label_image


  How to Classify Images with TensorFlow (Python - but may be memory intensive, see below)
     *  Python : https://github.com/tensorflow/models/tree/master/inception
       *  Includes their library 'tf-slim'
       *  Also : https://github.com/tensorflow/models/blob/master/inception/inception/inception_eval.py

  
  MAY BE PROBLEMATIC :  Involves retraining Inception
    Train your own image classifier with Inception in TensorFlow 
      *  http://googleresearch.blogspot.sg/2016/03/train-your-own-image-classifier-with.html
         *  Python : https://github.com/tensorflow/models/tree/master/inception
           *  Includes their library 'tf-slim'
           *  Also : https://github.com/tensorflow/models/blob/master/inception/inception/inception_eval.py
      *  Note that the entire model architecture is predicated on a 299x299 image, 
         *  thus if you wish to change the input image size, then you may need to redesign the entire model architecture.
      *  We have tuned the model to employ about ~40GB of CPU memory. Thus, having access to 64 or 128 GB of CPU memory would be ideal.
      *  We targeted a desktop with 128GB of CPU ram connected to 8 NVIDIA Tesla K40 GPU cards 
         *  but we have run this on desktops with 32GB of CPU ram and 1 NVIDIA Tesla K40. 
    
    

  MAY BE PROBLEMATIC :
    Image Analogies using Neural Networks (github.com) :: Keras
      *  HN : https://news.ycombinator.com/item?id=11234793
         *   Paper : https://github.com/awentzonline/image-analogies/
      *  Similiar code : https://github.com/fchollet/keras/blob/master/examples/neural_style_transfer.py
      *  http://ostagram.ru/static_pages/lenta?last_days=30
      *  http://deepart.io/
         *   Our algorithm uses a neural network based on the 19-layer VGG network by Karen Simonyan and Andrew Zisserman.
         *   University of Tübingen has a pending patent application for the Neural Art technology.

      *   There's also this: https://http://deepart.io/ - Not sure if it's the same thing but makes interesting pictures.
         *   According to their about page, that site is using this algorithm http://arxiv.org/pdf/1508.06576.pdf 
             *   whereas neural-image-analogies is a combination of http://www.mrl.nyu.edu/projects/image-analogies/index.html and http://arxiv.org/pdf/1601.04589v1.pdf
         *   Basically you can do a few other things with image analogies, such as texture-by-numbers, that the neural style algo doesn't cover. 
         *   In addition, the matching and blending covered in the Li and Wand paper provides more accurate style details.


      *  Problem with image patches in image-analogies :: 
         *   The current code uses images2neibs from theano to generate the image patches.
             *   https://github.com/awentzonline/image-analogies/issues/3
         *   This could be hugely simplified if patch_size==1

  MAY BE PROBLEMATIC :
    Turn your two-bit doodles into fine artworks with deep neural networks
      *  https://news.ycombinator.com/item?id=11257566
         *  https://github.com/alexjc/neural-doodle
      *  For details, the research paper is linked on the GitHub page: http://arxiv.org/abs/1603.01768
      *  For a video and higher-level overview see my article from yesterday: http://nucl.ai/blog/neural-doodles/
      *  It's a pre-trained network on image classification dataset from 2014 called ImageNet. 
         *  The network is called VGG, paper is here: http://arxiv.org/abs/1409.1556
      *  There's no additional training apart from that. 
         *  The neural network is used to extract patterns (grain/texture/style) 
         *  and a separate optimization tries to reproduce them as appropriate.

      *  Also uses theano-only function:
         *  patches = theano.tensor.nnet.neighbours.images2neibs(f, (size, size), (stride, stride), mode='valid')

            
  neural-style :: Torch
    *  https://github.com/jcjohnson/neural-style


  AlphaGo : reimplementation in Kerase / theano (/ tensorflow?)
    *  https://github.com/Rochester-NRT/AlphaGo  


Models

  Model Zoo ideas:
    https://groups.google.com/a/tensorflow.org/forum/m/#!topic/discuss/BKOiAgKDxqM
    But decent ImageNet model is ~60Mb => need ThumbDrives
  Inception 3 :
    https://www.tensorflow.org/versions/master/tutorials/image_recognition/index.html
  Inception 3 / Deep Dream = Coming soon (bottom of page) :
    https://www.tensorflow.org/versions/master/tutorials/index.html
  Caffe :
    http://caffe.berkeleyvision.org/gathered/examples/imagenet.html
  Lasagne :
    https://github.com/Lasagne/Recipes/tree/master/modelzoo
    https://github.com/alexjc/neural-doodle
      download this pre-trained neural network (VGG19, 80Mb)

  Microsoft Deep Residual Networks (with models, and Lasagne version link)
    https://github.com/KaimingHe/deep-residual-networks
    https://github.com/Lasagne/Recipes/tree/master/papers/deep_residual_learning
  

  Using Caffe models in Lasagne
    https://github.com/Lasagne/Lasagne/issues/157

  VGG16 model for Keras
    https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3
    
  VGG16 model for Lasagne
    model   : https://github.com/Lasagne/Recipes/blob/master/modelzoo/vgg16.py
    weights :  https://s3.amazonaws.com/lasagne/recipes/pretrained/imagenet/vgg16.pkl
    See also : https://github.com/Lasagne/Recipes/issues/20
    
  VGG16 model in Tensorflow-slim
    https://github.com/tensorflow/models/blob/master/inception/inception/slim/README.md


Lasagne Recipes Examples :
  drwxrwxr-x. 2 andrewsm andrewsm    4096 Mar 14 17:53 styletransfer
    VGG19 80Mb network  (this must be a cut-off version)
    
  -rw-rw-r--. 1 andrewsm andrewsm  970738 Mar 14 17:53 ImageNet Pretrained Network (VGG_S).ipynb
    393Mb network (VGG_CNN_S)  HUGE
    IMPLEMENT : Do the same with Inception3?
    
  -rw-rw-r--. 1 andrewsm andrewsm   68760 Mar 14 17:53 Using a Caffe Pretrained Network - CIFAR10.ipynb
    Small CIFAR10 network (pickled version is ~10Mb only, and workbook includes model definition)
  
  -rw-rw-r--. 1 andrewsm andrewsm   10805 Mar 14 17:53 lstm_text_generation.py
    Like Karpathy example
    
  
  drwxrwxr-x. 2 andrewsm andrewsm    4096 Mar 14 17:53 variational_autoencoder
  -rw-rw-r--. 1 andrewsm andrewsm  165138 Mar 14 17:53 Video features with C3D.ipynb
  -rw-rw-r--. 1 andrewsm andrewsm 1043040 Mar 14 17:53 Saliency Maps and Guided Backpropagation.ipynb
  -rw-rw-r--. 1 andrewsm andrewsm   72414 Mar 14 17:53 spatial_transformer_network.ipynb


Lasagne Recipes ModelZoo :
  -rw-rw-r--. 1 andrewsm andrewsm 4807 Mar 14 17:53 googlenet.py        ::   27Mb for full model  (unrestricted)
  -rw-rw-r--. 1 andrewsm andrewsm 7308 Mar 14 17:53 inception_v3.py     ::   91Mb for full model  (Apache)  (299x299 RGB uint8 input)
  -rw-rw-r--. 1 andrewsm andrewsm 2544 Mar 14 17:53 vgg16.py            ::  528Mb for full model  (non-commercial)
  -rw-rw-r--. 1 andrewsm andrewsm 2817 Mar 14 17:53 vgg19.py            ::  548Mb for full model  (non-commercial)
  -rw-rw-r--. 1 andrewsm andrewsm 2898 Mar 14 17:53 vgg_cnn_s.py        ::  393Mb for full model  (non-commercial)

  -rw-rw-r--. 1 andrewsm andrewsm 6382 Mar 14 17:53 c3d.py              ::  AlexNet for 3d processing (video)
  -rw-rw-r--. 1 andrewsm andrewsm 9779 Mar 14 17:53 caffe_reference.py 
  -rw-rw-r--. 1 andrewsm andrewsm 2572 Mar 14 17:53 cifar10_nin.py      ::   11Mb for full model  (unknown)


How about building the tensorflow inception-3 graph and saving it for viewing by TensorBoard?


Other courses

  Udacity course from Google 
    https://www.udacity.com/course/deep-learning--ud730
      Lesson 1: From Machine Learning to Deep Learning
      Lesson 2: Deep Neural Networks
      Lesson 3: Convolutional Neural Networks
      Lesson 4: Deep Models for Text and Sequences

  
  Other peoples'
    http://www.slideshare.net/roelofp/deep-learning-as-a-catdog-detector











### Foundation : Python (based in Jupyter)

Potential Toolsets : 

*  TensorFlow + Keras?  (requires more work)
*  Theano + Lasagne (standing on head, but doesn't have TensorBoard, which would be sexy)

Cannot rely on people having appropriate GPU installation (TensorFlow is particularly fussy version-wise).
Therefore, must resign ourselves to CPU-based training.

Decision : Run VirtualBox VMs





## How to implement workshop practicals

Running iPython / Jupyter

*  http://blog.ouseful.info/2014/12/12/seven-ways-of-running-ipython-notebooks/
   *  http://blog.ouseful.info/2015/05/18/kiteflying-around-containers-a-better-alternative-to-course-vms/
      *  https://github.com/docker/kitematic  (docker container manager for VirtualBox VM - but Not For Linux => disqualified)

Host the VM on a central server?

*  http://lambdaops.com/ipythonjupyter-tmpnb-debuts/
  *  Problem : CPU usage is high - so 10 people logged in would probably bring a single host to its knees

Alternative : AWS GPU instance
  *  How to install Theano on Amazon EC2 GPU instances for deep learning 
     *  Has prebuilt AMI too  : The current one is ami-b141a2f5 (the updated one at the top of the post). You can always search Theano - CUDA 7 as its name.
        *  NOw : ami-1f3e225a
     *  http://markus.com/install-theano-on-aws/

  *  Requirement : g2.2xlarge On-Demand Instances
     *  g2.2xlarge : GRID K520 gpu 
     *  Pricing : 
        *             vCPU  ECU  Memory (GiB) Instance Storage (GB)  Linux/UNIX Usage
        * g2.2xlarge  8      26           15                 60 SSD    $0.65 per Hour
        * g2.8xlarge  32    104           60            2 x 120 SSD    $2.60 per Hour
        
  *  Import of existing VMs
     *  Verify that your VM satisfies the prerequisites for VM Import, prepare it for import, 
        and export it from its current environment as an OVA file (or VMDK, VHD, or RAW).
        


## Vagrant

This project provides a Ubuntu (14.04) Vagrant Virtual Machine (VM) with the TensorFlow library from Google and IPython (now know as Jupyter) notebooks using vagrant :
*  https://github.com/gavinln/tensorflow-ipy

Python-dev vagrant configuration :
*  https://github.com/AnthonyNystrom/python-dev-bootstrap

*  https://datascience.berkeley.edu/data-science-vagrant-ipython-notebook/
   *  Except that :
      *   vagrant still requires people to have VirtualBox installed.
      *   vagrant downloads 320Mb disk images - which would over-burden the network, and take 20mins
      *   ... So : Much more reasonable to distribute VirtualBox VM images on USB ...

*  Alternative : Vagrant
   *   https://developer.fedoraproject.org/tools/vagrant/vagrant-virtualbox.html
   *   https://www.vagrantup.com/        
        *   This was how the Spark environment for the edX course was done...  Perhaps the vagrant(?) files exist?


*  Explicit instructions for building fedora environment that runs self-contained:
   *   https://rwmj.wordpress.com/2013/10/27/using-virt-builder-to-build-packages/
   


## VirtualBox images

VirtualBox images are pretty portable...

*  If we only need iPython with all the imports, don't need to worry about editor, etc
   *  could also do it without installing X - and running the browser session in the host machine...
  
Background :
*  Fedora minimal VM building : 
   *  https://developer.fedoraproject.org/tools/virt-builder/about.html
   *  http://libguestfs.org/virt-builder.1.html
      *   https://kashyapc.fedorapeople.org/virt/virt-builder-usage.txt

*  https://spin.atomicobject.com/2013/06/03/ovf-virtual-machine/
  *  qemu-img – Part of the QEMU project, qemu-img is the holy grail, Swiss Army knife of disk image conversion.  If you need to convert a disk image, check here first.
     *  http://docs.openstack.org/image-guide/convert-images.html#qemu-img-convert-raw-qcow2-qed-vdi-vmdk-vhd
     
   
*  Creating something bootable by VirtualBox
  *  https://github.com/devrandom/gitian-builder#virtualbox-1
     * (not yet...)
  *  https://getfedora.org/en/cloud/download/index.html
     * "Base Cloud Images for Vagrant" : "VirtualBox Image"
  *  https://www.packer.io/intro/index.html
     *  OVF exports for VirtualBox?
     *  VirtualBox VDI file?
  *  http://virtuallyhyper.com/2013/06/migrate-from-libvirt-kvm-to-virtualbox/
     *  Sadly, the virt-convert tool no longer supports the -o vmx option. 
        *   It looks like you might have to just use the old version:
     *  Might want to state right at the beginning that this requires VMware installed and isn't any use if you don't!
        *   ovftool is free, you just have to create a myVMware account. 
        *   The registration for a MyVMWare account is also free, you just need a valid email address. 
        *   Have you tried registering for a MyVMware account?
  *  http://it-ovid.blogspot.sg/2012/10/virtual-box-headless-cheatsheet.html
     *  Import VMWare disk images
        *   VBoxManage convertfromraw --format vmdk --variant Standard <source>.vmdk <destination>.vdi
     *  This was suggested by Kai Hendry as being very do-able
        *   Also ... VirtualBox is capable of importing and exporting virtual machines in the standard Open Virtualization Format (OVF). 
   *  [How To] Headless VirtualBox on Headless Server
      *   https://forums.virtualbox.org/viewtopic.php?f=7&t=60681
         *  Just a note: exporting a VM will change internal IDs for both the VM virtual motherboard and the disks.
         *  This will typically break a grub/fstab config in Linux and possibly trigger again activation on Windows. 
            *  Copying the VM directly really is the proper way of doing it.
         *  Exporting should be a last resort thing.





  
Successful installers could also open up their 'server' to access via WiFi, without risk :

*  https://nsrc.org/workshops/2014/btnog/raw-attachment/wiki/Track2Agenda/ex-virtualbox-portforward-ssh.htm
*  https://www.reddit.com/r/IPython/comments/3gw3e5/how_to_access_to_the_ipython_server_from_a/
  

GPU?

*  unfortunately VM solution probably doesn't allow for GPU anyway : 
   *  https://github.com/BVLC/caffe/issues/2749
  


## What environment to use?  

VPS?  expense...

  TensorFlow + iPython + VM
    https://github.com/gavinln/tensorflow-ipy
    http://stackoverflow.com/questions/33784214/how-to-test-tensorflow-cifar10-cnn-tutorial-model?rq=1

  Thumb drive ideas:
    http://www.aliexpress.com/item/USB-3-0-3000-style-Cartoon-Usb-Flash-Drive-4GB-8GB-16GB-32GB-Pen-Drive-100pcs/32500540806.html

  Utilitarian:
    http://www.aliexpress.com/item/Usb-flash-drive-Smart-Phone-OTG-Pendrive-Pen-Drive-32GB-16GB-8GB-4GB-Memory-stick-mini/32462761490.html
    
  Cat-themed:
    Claw:
      http://www.aliexpress.com/item/Cute-Cat-Claw-Usb-Flash-Drive-USB-2-0-pen-drive-64GB-32GB-16GB-Pendrive-8GB/32593071579.html
      http://www.aliexpress.com/item/Usb-flash-drive-4gb-8gb-16gb-32gb-cat-s-claw-usb-pendrive-gift-u-disk-memory/32424298881.html
    Metal:
      http://www.aliexpress.com/item/Wholesale-and-custom-Cat-model-of-metal-gift-art-creative-personality-key-usb-flash-drive-16gb/32323841251.html
    3D: 
      http://www.aliexpress.com/item/Free-Shipping-cartoon-pen-drive-kitty-cat-flash-memory-stick-pendrive-1GB-64GB/2038130430.html
      http://www.aliexpress.com/item/Plastic-cartoon-baby-hello-kitty-cat-USB-Flash-Drives-thumb-pen-drives-memory-stick-2G-4GB/1024940593.html
    Flat:
      http://www.aliexpress.com/item/Cartoon-Animal-Flat-Cat-Style-4GB-8GB-16GB-32GB-USB-2-0-Flash-Drive-Pen-Drive/32372039038.html

  Engraved
    http://www.aliexpress.com/item/Creative-Crystal-Transparent-LED-With-Volkswagen-s-VW-Logo-2GB-32GB-USB-Flash-2-0-Memory/1406199398.html
    http://www.aliexpress.com/item/10PCS-LOT-Wooden-USB-with-gift-box-2GB-32GB-USB-2-0-Flash-Memory-USB-flash/32221529820.html
    http://www.aliexpress.com/item/Brand-New-Wooden-Maple-Personalized-LOGO-USB-2-0-Memory-flash-stick-pen-drive-with-metal/32412131258.html
    http://www.aliexpress.com/item/Wooden-USB-Flash-Drive-32GB-16GB-8GB-4GB-Custom-Logo-Engraving-USB-2-0-flash-disk/32573215831.html
    
docker images?  

      GPU would be tricky to use, and there's no OS-independence
        http://stackoverflow.com/questions/25185405/using-gpu-from-a-docker-container
        https://github.com/NVIDIA/nvidia-docker
        
      Can run on windows (it essentially builds on top of a Virtualized Linux environment)
        https://docs.docker.com/engine/installation/windows/
    
USB Live environments?  


    

Squid caching of RPM downloads?      
  http://serverascode.com/2014/03/29/squid-cache-yum.html

virt-builder cache location
  https://rwmj.wordpress.com/2014/09/11/creating-a-local-virt-builder-repository/