live_braille_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **0. Setup**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Imports**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from utils import *\n",
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Pybraille**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pybraille performs grade-1 Braille translation. Eventual goal is [grade-2 translation](https://www.brailletranslator.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installation:\n",
    "```\n",
    "pip install pybraille\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⠓⠑⠇⠇⠕\n",
      "⠋⠊⠇⠑⠝⠁⠍⠑⠲⠞⠭⠞\n"
     ]
    }
   ],
   "source": [
    "from pybraille import convertText\n",
    "\n",
    "print(convertText(\"hello\"))\n",
    "\n",
    "print(convertText(\"filename.txt\")) #eg: tests/sample.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **EasyOCR**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a number of competing packages which offer scene text detection out-of-the-box. EasyOCR has a simple setup process, and offers nice performance when GPU-accelerated (but not ideally fast — i.e., not suitable for real-time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installation\n",
    "```\n",
    "pip install git+https://github.com/jaidedai/easyocr.git\n",
    "```\n",
    "Note: using vanilla pip, it may cause a reinstall of PyTorch. Reinstall with appropriate CUDA version [here](https://pytorch.org).  \n",
    "Check CUDA version using `nvcc --version`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\hocbu\\AppData\\Roaming\\Python\\Python39\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import easyocr\n",
    "reader = easyocr.Reader(['en'], gpu=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **PaddleOCR**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Offers multiple models, including pruned and quantized models for mobile deployment, which is exactly what we need.  \n",
    "It comes with *layout analysis* feature (auto-detects title/picture/table/etc), could prove useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installation\n",
    "```\n",
    "pip install paddlepaddle-gpu -i https://mirror.baidu.com/pypi/simple\n",
    "pip install paddleocr\n",
    "```\n",
    "Note: Windows: you may need to install shapely (`pip install shapely`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2022/05/27 14:39:41] ppocr DEBUG: Namespace(help='==SUPPRESS==', use_gpu=True, ir_optim=True, use_tensorrt=False, min_subgraph_size=15, precision='fp32', gpu_mem=500, image_dir=None, det_algorithm='DB', det_model_dir='C:\\\\Users\\\\hocbu/.paddleocr/whl\\\\det\\\\en\\\\en_PP-OCRv3_det_infer', det_limit_side_len=960, det_limit_type='max', det_db_thresh=0.3, det_db_box_thresh=0.6, det_db_unclip_ratio=1.5, max_batch_size=10, use_dilation=False, det_db_score_mode='fast', det_east_score_thresh=0.8, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_sast_score_thresh=0.5, det_sast_nms_thresh=0.2, det_sast_polygon=False, det_pse_thresh=0, det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_box_type='quad', det_pse_scale=1, scales=[8, 16, 32], alpha=1.0, beta=1.0, fourier_degree=5, det_fce_box_type='poly', rec_algorithm='SVTR_LCNet', rec_model_dir='C:\\\\Users\\\\hocbu/.paddleocr/whl\\\\rec\\\\en\\\\en_PP-OCRv3_rec_infer', rec_image_shape='3, 48, 320', rec_batch_num=6, max_text_length=25, rec_char_dict_path='c:\\\\Users\\\\hocbu\\\\AppData\\\\Local\\\\Programs\\\\Python\\\\Python39\\\\lib\\\\site-packages\\\\paddleocr\\\\ppocr\\\\utils\\\\en_dict.txt', use_space_char=True, vis_font_path='./doc/fonts/simfang.ttf', drop_score=0.5, e2e_algorithm='PGNet', e2e_model_dir=None, e2e_limit_side_len=768, e2e_limit_type='max', e2e_pgnet_score_thresh=0.5, e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_pgnet_valid_set='totaltext', e2e_pgnet_mode='fast', use_angle_cls=False, cls_model_dir='C:\\\\Users\\\\hocbu/.paddleocr/whl\\\\cls\\\\ch_ppocr_mobile_v2.0_cls_infer', cls_image_shape='3, 48, 192', label_list=['0', '180'], cls_batch_num=6, cls_thresh=0.9, enable_mkldnn=False, cpu_threads=10, use_pdserving=False, warmup=False, draw_img_save_dir='./inference_results', save_crop_res=False, crop_res_save_dir='./output', use_mp=False, total_process_num=1, process_id=0, benchmark=False, save_log_path='./log_output/', show_log=True, use_onnx=False, output='./output', table_max_len=488, table_model_dir=None, table_char_dict_path=None, layout_path_model='lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config', layout_label_map=None, mode='structure', layout=True, table=True, ocr=True, lang='en', det=True, rec=True, type='ocr', ocr_version='PP-OCRv3', structure_version='PP-STRUCTURE')\n"
     ]
    }
   ],
   "source": [
    "from paddleocr import PaddleOCR, draw_ocr\n",
    "ocr = PaddleOCR(lang=\"en\")  # need to run only once to download and load model into memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation Log (tells default model) (can delete later)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```download https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_ppocr_mobile_v2.0_det_infer.tar to C:\\Users\\hocbu/.paddleocr/whl\\det\\en\\en_ppocr_mobile_v2.0_det_infer\\en_ppocr_mobile_v2.0_det_infer.tar\n",
    "100%|██████████| 3.16M/3.16M [00:10<00:00, 313kiB/s] \n",
    "download https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_number_mobile_v2.0_rec_infer.tar to C:\\Users\\hocbu/.paddleocr/whl\\rec\\en\\en_number_mobile_v2.0_rec_infer\\en_number_mobile_v2.0_rec_infer.tar\n",
    "100%|██████████| 2.70M/2.70M [00:09<00:00, 277kiB/s] \n",
    "download https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar to C:\\Users\\hocbu/.paddleocr/whl\\cls\\ch_ppocr_mobile_v2.0_cls_infer\\ch_ppocr_mobile_v2.0_cls_infer.tar\n",
    "100%|██████████| 1.45M/1.45M [00:08<00:00, 175kiB/s] [2022/04/10 17:03:35] ppocr DEBUG: Namespace(help='==SUPPRESS==', use_gpu=True, ir_optim=True, use_tensorrt=False, min_subgraph_size=15, precision='fp32', gpu_mem=500, image_dir=None, det_algorithm='DB', det_model_dir='C:\\\\Users\\\\hocbu/.paddleocr/whl\\\\det\\\\en\\\\en_ppocr_mobile_v2.0_det_infer', det_limit_side_len=960, det_limit_type='max', det_db_thresh=0.3, det_db_box_thresh=0.6, det_db_unclip_ratio=1.5, max_batch_size=10, use_dilation=False, det_db_score_mode='fast', det_east_score_thresh=0.8, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_sast_score_thresh=0.5, det_sast_nms_thresh=0.2, det_sast_polygon=False, det_pse_thresh=0, det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_box_type='quad', det_pse_scale=1, scales=[8, 16, 32], alpha=1.0, beta=1.0, fourier_degree=5, det_fce_box_type='poly', rec_algorithm='CRNN', rec_model_dir='C:\\\\Users\\\\hocbu/.paddleocr/whl\\\\rec\\\\en\\\\en_number_mobile_v2.0_rec_infer', rec_image_shape='3, 32, 320', rec_batch_num=6, max_text_length=25, rec_char_dict_path='C:\\\\Users\\\\hocbu\\\\AppData\\\\Roaming\\\\Python\\\\Python39\\\\site-packages\\\\paddleocr\\\\ppocr\\\\utils\\\\en_dict.txt', use_space_char=True, vis_font_path='./doc/fonts/simfang.ttf', drop_score=0.5, e2e_algorithm='PGNet', e2e_model_dir=None, e2e_limit_side_len=768, e2e_limit_type='max', e2e_pgnet_score_thresh=0.5, e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_pgnet_valid_set='totaltext', e2e_pgnet_mode='fast', use_angle_cls=True, cls_model_dir='C:\\\\Users\\\\hocbu/.paddleocr/whl\\\\cls\\\\ch_ppocr_mobile_v2.0_cls_infer', cls_image_shape='3, 48, 192', label_list=['0', '180'], cls_batch_num=6, cls_thresh=0.9, enable_mkldnn=False, cpu_threads=10, use_pdserving=False, warmup=False, draw_img_save_dir='./inference_results', save_crop_res=False, crop_res_save_dir='./output', use_mp=False, total_process_num=1, process_id=0, benchmark=False, save_log_path='./log_output/', show_log=True, use_onnx=False, output='./output', table_max_len=488, table_model_dir=None, table_char_type='en', table_char_dict_path=None, layout_path_model='lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config', layout_label_map=None, mode='structure', lang='en', det=True, rec=True, type='ocr', ocr_version='PP-OCRv2', structure_version='STRUCTURE')```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **1. Single Image**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "- Scene text detection\n",
    "- Braille translation\n",
    "- Text highlight and braille caption"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = './demo_image/'\n",
    "filename = 'demodemo2.jpg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = cv2.imread(folder+filename)\n",
    "img_d, sf = imresize(img) # downscale, remember scaling factor\n",
    "# resize를 하고 돌리면 integer coordinate가 나옴 (정확한 이유 불명)\n",
    "\n",
    "img_ = img.copy()  # backup\n",
    "img_d_ = img_d.copy() # backup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize images\n",
    "# print(img_d.shape)\n",
    "# print(img.shape)\n",
    "# plt.imshow(img)\n",
    "# plt.imshow(img_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scene Text Detection (EasyOCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may have a built-in resolution reduction (& recovery) feature. A little annoying to repeatedly scale points afterwards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = reader.readtext(img_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아직 좀 느린데, quantization, treeshake 해볼까?  \n",
    "\n",
    "[Medium 글](https://medium.com/swlh/ocr-engine-comparison-tesseract-vs-easyocr-729be893d3ae)에 따르면 0.07s 라는데, 아마 scene text detection은 그보다 느릴 듯. text detection 열고있어도 웹캠 윈도우 업데이트 되나?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scene Text Detection (PaddleOCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# result2 = ocr.ocr(img_d, cls=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Highlight Text Regions\n",
    "✅"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Region Selection\n",
    "- make a selection, save it\n",
    "  - there should be single selection. if multiple texts fit within tolerance, choose the closest one\n",
    "- apply a stronger highlight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "center = (int(img.shape[1]/2), int(img.shape[0]/2))\n",
    "gaze_point = (999,999) # change this. possibly interactive. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "([[90, 146], [156, 146], [156, 172], [90, 172]], 'AUDIO &', 0.9374154164120676)\n",
      "([[101, 162], [193, 162], [193, 190], [101, 190]], 'RECORDING', 0.9990951954757775)\n",
      "([[113, 179], [182, 179], [182, 206], [113, 206]], 'STUDiOS', 0.6264606800384376)\n",
      "([[155.23076923076923, 149.15384615384616], [202.90531842243738, 154.3919196524136], [199.76923076923077, 176.84615384615384], [152.09468157756262, 170.6080803475864]], 'VIDEO', 0.9853656861861788)\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "argument for rectangle() given by name ('thickness') and position (4)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_14076/2947279168.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     43\u001b[0m \u001b[0mpt1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpt2\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mn\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0msf\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mn\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mselection\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mn\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0msf\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mn\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mselection\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     44\u001b[0m \u001b[0mstrong\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0madd_mask\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstrong\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpt1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpt2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 45\u001b[1;33m \u001b[0mimg\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcv2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrectangle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mimg\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpt1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpt2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;36m255\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m255\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m255\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mthickness\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m20\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# selection outline\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     46\u001b[0m \u001b[0mimg\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcv2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrectangle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mimg\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcenter\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcenter\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m50\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mthickness\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m50\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# visualize center\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     47\u001b[0m \u001b[0mhighlighted\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mapply_highlight\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mimg\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mweak\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstrong\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: argument for rectangle() given by name ('thickness') and position (4)"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1080x1080 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# matplotlib details\n",
    "fontsize = 0.025\n",
    "plt.figure(figsize=(15, 15))\n",
    "\n",
    "# highlight\n",
    "weak = np.zeros(shape=img.shape)\n",
    "strong = np.zeros(shape=img.shape)\n",
    "\n",
    "# selection\n",
    "min_dist = 200\n",
    "selection = None\n",
    "\n",
    "# iterate through each text\n",
    "for i, r in enumerate(result): \n",
    "        \n",
    "    # debug\n",
    "    print(r)\n",
    "\n",
    "    # points\n",
    "    pt1 = [int(n*sf) for n in r[0][0]]\n",
    "    pt2 = [int(n*sf) for n in r[0][2]]\n",
    "\n",
    "    # add highlight\n",
    "    try: weak = add_mask(weak, pt1, pt2)\n",
    "    except Exception: print('highlight error')\n",
    "\n",
    "    # add caption\n",
    "    try:\n",
    "        br = convertText(r[1])\n",
    "        plt.figtext(0.5, 0.065+fontsize-fontsize*2*i, r[1], ha=\"center\", fontsize=18)\n",
    "        plt.figtext(0.5, 0.065+0.005-fontsize*2*i, br, ha=\"center\", fontsize=18, bbox={\"facecolor\":\"orange\", \"alpha\":0.5, \"pad\":1})\n",
    "    except Exception: \n",
    "        plt.figtext(0.5, 0.065+fontsize-fontsize*2*i, r[1], ha=\"center\", fontsize=18)\n",
    "        plt.figtext(0.5, 0.065+0.005-fontsize*2*i, \"(error: contains special symbols)\", ha=\"center\", fontsize=18, bbox={\"facecolor\":\"orange\", \"alpha\":0.5, \"pad\":1})\n",
    "\n",
    "    # distance\n",
    "    dist = distance([pt1, pt2], center)\n",
    "    # print(dist, r[1])\n",
    "    if dist < min_dist: \n",
    "        min_dist = dist\n",
    "        selection = r\n",
    "\n",
    "pt1, pt2 = [int(n*sf) for n in selection[0][0]], [int(n*sf) for n in selection[0][2]]\n",
    "strong = add_mask(strong, pt1, pt2)\n",
    "img = cv2.rectangle(img, pt1, pt2, (255,255,255), thickness=20) # selection outline\n",
    "img = cv2.rectangle(img, center, center, (0,50,0), thickness=50) # visualize center\n",
    "highlighted = apply_highlight(img, weak, strong)\n",
    "\n",
    "print('Selected: ', selection[1])\n",
    "\n",
    "plt.imshow(highlighted) # this line has to be in this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **2. Interactive Video**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objetives\n",
    "- Receive webcam stream ✅\n",
    "    - DroidCam prototype\n",
    "- Interactive selection \n",
    "    - Intuitive selection behavior: hysteresis 🆇\n",
    "    - Braille output on bottom (or as tooltip) ✅\n",
    "- Real-time optimization "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pygame \n",
    "- rationale: more flexible, powerful graphics than OpenCV GUI\n",
    "  - possible addition of Dot Display module\n",
    "  - possible addition of text presence animation (propagating/expanding dot wave)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pygame\n",
    "import cv2\n",
    "import numpy as np\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process(img, texts, sf):\n",
    "    # selection\n",
    "    min_dist = 200\n",
    "    selection = None\n",
    "    center = (int(img.shape[1]/2), int(img.shape[0]/2))\n",
    "\n",
    "\n",
    "    # highlight mask\n",
    "    weak = np.zeros(shape=img.shape)\n",
    "    strong = np.zeros(shape=img.shape)\n",
    "\n",
    "    # iterate through each text\n",
    "    for i, r in enumerate(texts): \n",
    "        # print(r)\n",
    "\n",
    "        # points\n",
    "        pt1 = [int(n*sf) for n in r[0][0]]\n",
    "        pt2 = [int(n*sf) for n in r[0][2]]\n",
    "\n",
    "        # add highlight\n",
    "        try: weak = add_mask(weak, pt1, pt2)\n",
    "        except Exception: print('highlight error')\n",
    "\n",
    "        # select\n",
    "        dist = distance([pt1, pt2], center)\n",
    "        if dist < min_dist: \n",
    "            min_dist = dist\n",
    "            selection = r\n",
    "\n",
    "    # selected text\n",
    "    # print('Selected text: ', selection)\n",
    "    if selection:\n",
    "        pt1, pt2 = [int(n*sf) for n in selection[0][0]], [int(n*sf) for n in selection[0][2]]\n",
    "        pt1, pt2 = tuple(pt1), tuple(pt2) # OpenCV's rectangle doesn't like coordinates given in list\n",
    "        strong = add_mask(strong, pt1, pt2)\n",
    "        img = cv2.rectangle(img, pt1, pt2, (255,255,255), 15) # outline\n",
    "\n",
    "    # visualization\n",
    "    img = cv2.rectangle(img, center, center, (0,50,0), 20) # center\n",
    "    highlighted = apply_highlight(img, weak, strong)\n",
    "\n",
    "    return highlighted, selection\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "pygame.init()\n",
    "pygame.display.set_caption(\"Camera Braille Translation\")\n",
    "surface = pygame.display.set_mode([1280, 720])\n",
    "\n",
    "# OpenCV\n",
    "cap = cv2.VideoCapture(1)\n",
    "\n",
    "# OpenCV FPS\n",
    "fps = cap.get(cv2.CAP_PROP_FPS)\n",
    "cap.set(cv2.CAP_PROP_FPS, 60)\n",
    "\n",
    "# pygame\n",
    "pygame.font.init()\n",
    "main_font = pygame.font.SysFont('segoeuisymbol', 30) # for Braille\n",
    "side_font = pygame.font.SysFont('segoeuisymbol', 15) # for Braille\n",
    "\n",
    "background_color = (30,30,30)\n",
    "\n",
    "# pygame FPS\n",
    "clock = pygame.time.Clock()\n",
    "\n",
    "initialized = False\n",
    "stop = False\n",
    "\n",
    "\n",
    "while not initialized:\n",
    "    success, frame = cap.read()\n",
    "    if success: break\n",
    "\n",
    "while not stop:\n",
    "    # frames per second \n",
    "    clock.tick()\n",
    "    # profiling\n",
    "    ts_base = time.time() # timestamp base\n",
    "\n",
    "    # read cam\n",
    "    success, frame = cap.read()\n",
    "    if not success: print(\"not success\")\n",
    "    ts_read = time.time() - ts_base # time: reading from cam\n",
    "\n",
    "    # scene text detection\n",
    "    frame_d, sf = imresize(frame)\n",
    "    texts = reader.readtext(frame_d)\n",
    "    # print(texts)\n",
    "    ts_ocr = time.time()-ts_base - ts_read # time: running scene text detection\n",
    "\n",
    "    # ANNOTATION\n",
    "    # frame processing\n",
    "    frame, sel = process(frame, texts, sf)\n",
    "    ts_hili = time.time()-ts_base - ts_ocr # time: highlighting\n",
    "    # pre-pygame-processing\n",
    "    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "    frame = np.rot90(np.fliplr(frame))\n",
    "\n",
    "    # RENDERING\n",
    "    # background\n",
    "    surface.fill(background_color)\n",
    "    # frame\n",
    "    frame_surface = pygame.surfarray.make_surface(frame)\n",
    "    surface.blit(frame_surface, (0,0))\n",
    "    ts_game = time.time()-ts_base - ts_ocr\n",
    "\n",
    "    # DASHBOARD\n",
    "    # selected text (in English and Braille)\n",
    "    try: \n",
    "        selected_surf = main_font.render(sel[1], False, (255,255,255))\n",
    "        surface.blit(selected_surf, (30,550))\n",
    "\n",
    "        br = convertText(sel[1])\n",
    "        b = main_font.render(br, False, (255,255,255))\n",
    "        surface.blit(b, (30,610))\n",
    "        # print(sel[1], br)\n",
    "    except TypeError as e: pass\n",
    "    ts_text = time.time()-ts_base - ts_game\n",
    "        \n",
    "    # labels\n",
    "    # selected\n",
    "    selected_label_surf = side_font.render('English', False, (230,230,230))\n",
    "    surface.blit(selected_label_surf, (30,535))\n",
    "    # braille\n",
    "    braille_label_surf = side_font.render('Grade I Braille', False, (230,230,230))\n",
    "    surface.blit(braille_label_surf, (30,590))\n",
    "    # framerate\n",
    "    fps_text_surface = side_font.render('FPS:'+str(round(clock.get_fps(), 2)), False, (255,255,255))\n",
    "    surface.blit(fps_text_surface, (950, 680))\n",
    "    ts_label = time.time()-ts_base - ts_text\n",
    "\n",
    "    # profiling\n",
    "    ts_sum = sum([ts_read, ts_ocr, ts_hili, ts_game, ts_text, ts_label])\n",
    "    percentages = [round(ts_read/ts_sum*100),round(ts_ocr/ts_sum*100), round(ts_hili/ts_sum*100), round(ts_game/ts_sum*100), round(ts_text/ts_sum*100), round(ts_label/ts_sum*100)] # make this mapped\n",
    "    percentages_label = ['Reading from camera', 'Running OCR', 'Highlighting', 'Rendering frame', 'Rendering text', 'Rendering label']\n",
    "\n",
    "    # render title\n",
    "    profile_surf = main_font.render('Profiler', False, (255,255,255))\n",
    "    surface.blit(profile_surf, (950,490))\n",
    "    # render stats\n",
    "    for index, percentage in enumerate(percentages):\n",
    "        surface.blit(side_font.render(str(percentage)+'%   '+percentages_label[index], False, (255,255,255)), (950, 540+(index*20)))\n",
    "\n",
    "    # pygame essentials\n",
    "    pygame.display.flip()\n",
    "    # end management\n",
    "    for event in pygame.event.get():\n",
    "        if event.type == pygame.QUIT: \n",
    "            pygame.quit()\n",
    "            stop = True\n",
    "\n",
    "    # except TypeError as te: \n",
    "    #     print(te) # occurs in the beginning, when capture has not finished initialization\n",
    "\n",
    "cap.release()\n",
    "\n",
    "# calculate mismatch between time.time based vs. pygame's estimated FPS\n",
    "# so we can check if we're missing something from profiling\n",
    "\n",
    "# https://stackoverflow.com/questions/59948996/how-to-use-webcam-as-a-screen-of-pygame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "General Observation Notes\n",
    "- Selection outline too thick\n",
    "- Framerate is usable, but worsens as the amount of text increases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactivity & UX\n",
    "- DroidCam image size meter\n",
    "- selection change event listener"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ed14edbca5228d46ee727e3895a84b669fc2394a68d3f600b4cef4b26033d3d1"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}