video_to_bvh.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "video_to_bvh.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "P__hA1g6HzpZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Initial imports, install, initializations \n",
        "####Be patient it'll take some time :)"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "j_YTs3eEFXjW",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/Dene33/keras_Realtime_Multi-Person_Pose_Estimation.git\n",
        "!pip install configobj\n",
        "!apt install ffmpeg\n",
        "!apt-get install blender\n",
        "\n",
        "!apt install python2.7 python-pip\n",
        "!git clone https://github.com/Dene33/hmr.git\n",
        "!wget https://people.eecs.berkeley.edu/~kanazawa/cachedir/hmr/models.tar.gz && tar -xf models.tar.gz\n",
        "!mv models hmr/\n",
        "!pip2 install -r hmr/requirements.txt\n",
        "!mkdir hmr/output\n",
        "!mkdir hmr/output/csv\n",
        "!mkdir hmr/output/images\n",
        "!mkdir hmr/output/csv_joined\n",
        "!mkdir hmr/output/bvh_animation\n",
        "\n",
        "import os\n",
        "os.chdir('keras_Realtime_Multi-Person_Pose_Estimation')\n",
        "!bash model/get_keras_model.sh\n",
        "!mkdir sample_jsons\n",
        "!mkdir sample_videos\n",
        "!mkdir sample_images\n",
        "\n",
        "exec(open('model_load.py').read())\n",
        "!pip2 install opendr==0.77"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XNg9osekfr9o",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Upload video"
      ]
    },
    {
      "metadata": {
        "id": "20WUpg6ffraY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#upload video\n",
        "exec(open('upload_videos.py').read())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "gwTyO3683c08"
      },
      "cell_type": "markdown",
      "source": [
        "##Process the video\n",
        "####<font color='green'>`24`</font> in `video_to_images.sh` stands for desired fps rate at which you want to convert video to images"
      ]
    },
    {
      "metadata": {
        "id": "UIAyAmBJ3bJf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#convert to images, specify fps rate\n",
        "!bash video_to_images.sh 24\n",
        "\n",
        "#2d pose estimation. For each image creates corresponding .json file with format\n",
        "#similar to output .json format of openpose (https://github.com/CMU-Perceptual-Computing-Lab/openpose)\n",
        "exec(open('2d_pose_estimation.py').read())\n",
        "\n",
        "#3d pose estimation\n",
        "os.chdir('..')\n",
        "!bash hmr/3dpose_estimate.sh\n",
        "\n",
        "#convert estimated .csv files to bvh\n",
        "!blender --background hmr/csv_to_bvh.blend -noaudio -P hmr/csv_to_bvh.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nTjDp_WpOK2r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Download .bvh"
      ]
    },
    {
      "metadata": {
        "id": "GE3TrWCq6jP_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "files.download('hmr/output/bvh_animation/estimated_animation.bvh')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "e3mfmjKlWJxS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Display results of 3d pose estimation (optional)"
      ]
    },
    {
      "metadata": {
        "id": "sLFtmdXWVztJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from IPython.display import Image, display\n",
        "\n",
        "images = os.listdir('hmr/output/images')\n",
        "extensions = {\".jpg\", \".png\", \".gif\", \".bmp\", \".jpeg\"}\n",
        "\n",
        "for image in sorted(images):\n",
        "  for ext in extensions:\n",
        "    if image.endswith(ext):\n",
        "      display(Image('hmr/output/images/'+image, width=512))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3b7kJVtZfMj6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Clear all the generated data if you want to process new video"
      ]
    },
    {
      "metadata": {
        "id": "jIWBMk8rfMuB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!rm keras_Realtime_Multi-Person_Pose_Estimation/sample_images/*\n",
        "!rm keras_Realtime_Multi-Person_Pose_Estimation/sample_jsons/*\n",
        "!rm keras_Realtime_Multi-Person_Pose_Estimation/sample_videos/*\n",
        "!rm hmr/output/bvh_animation/*\n",
        "!rm hmr/output/csv/*\n",
        "!rm hmr/output/csv_joined/*\n",
        "!rm hmr/output/images/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4o7jqjHAW7pA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "####Based on:\n",
        "\n",
        "\n",
        "1.   **End-to-end Recovery of Human Shape and Pose** by *Angjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik *\n",
        "https://github.com/akanazawa/hmr  \n",
        "\n",
        "  Corresponding fork by *Denis Cera*: https://github.com/Dene33/hmr\n",
        "\n",
        "2.   **Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields** by *Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh* \\\n",
        "Keras implementation by *Michal Faber*: https://github.com/michalfaber/keras_Realtime_Multi-Person_Pose_Estimation\n",
        "\n",
        "  Corresponding fork by *Denis Cera*: https://github.com/Dene33/keras_Realtime_Multi-Person_Pose_Estimation\n",
        "\n",
        "\n"
      ]
    }
  ]
}