resnet算法分析文档.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基本数据结构：</br>\n",
    "struct Matrix </br>\n",
    "{</br>\n",
    "    (int) channelSize ：描述该特征矩阵的通道数</br>\n",
    "    (int) rowSize ： 描述该特征矩阵的行数</br>\n",
    "    (int) columnSize ： 描述该特征矩阵的列数</br>\n",
    "    (float ***) feature : 存储该特征矩阵的值</br>\n",
    "}</br>\n",
    "定义：文中所示 num 为批次大小 ； i 为特征矩阵的通道数 ； j 为特征矩阵的行数 ； k 为特征矩阵的例数 ；x 为卷积核数量\n",
    "    ；y为卷积核行数 ； z为卷积核例数"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>以下皆为 特征矩阵基本计算时使用的函数算法分析</br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "class Matrix:\n",
    "    channelSize = 0\n",
    "    rowSize = 0\n",
    "    columnSize = 0\n",
    "    feature =[[[]]]\n",
    "\n",
    "    def __init__(self,func,channelSize, rowSize , columnSize):\n",
    "        self.channelSize = channelSize\n",
    "        self.rowSize = rowSize\n",
    "        self.columnSize = columnSize\n",
    "        if(func == \"Zero\"):\n",
    "            self.feature = Matrix.Zero(channelSize, rowSize , columnSize)\n",
    "        if(func == \"Random\"):\n",
    "            self.feature = Matrix.Random(channelSize, rowSize , columnSize)\n",
    "\n",
    "    def Zero(channel:int ,row:int , column:int):\n",
    "        output = [[[0 for k in range(column)] for j in range(row)] for i in range(channel)]\n",
    "        return output\n",
    "\n",
    "    def Random(channel:int ,row:int , column:int):\n",
    "        output = [[[random.random() for k in range(column)] for j in range(row)] for i in range(channel)]\n",
    "        return output\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "struct Matrix *Matrix_Init(struct Matrix *(*func)(int ,int,int),int channel ,int row,int column);\n",
    "\n",
    "功能 ：该函数为特征矩阵的初始化操作，由*func(Zero/Random)选取初始化模式并通过申请一个Matrix数据结构的空间，再将其指针地址返回。其中Zero初始化的矩阵值全为0，而Random函数初始化的值则全为0至1的浮点数。\n",
    "\n",
    "时间复杂度O(N) ：2 * i * j * k\n",
    "\n",
    "调用例子 ： struct Matrix *feature = Matrix_Init(Zero,3,6,4)\n",
    "如上所示将初始化一个大小为(3X6X4)的特侦矩阵并将其指针地址存储到feature变量中。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Matrix_Print(struct Matrix *input)<br/>\n",
    "<br/>\n",
    "功能 ：该函数为输出代入的特征矩阵<br/>\n",
    "其格式为：<br/>\n",
    "channel size = %d , size = %d * %d<br/> \n",
    "并依次输出每个特征矩阵通道的二维矩阵<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k<br/>\n",
    "\n",
    "调用例子 ： Matrix_Print(feature)<br/>\n",
    "如上所示将输出该变量的特征矩阵大小并依次输出feature变量中的值。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Matrix_Print(input:Matrix):\n",
    "    print(\"channel size =\",input.channelSize,\"size =\", input.rowSize,\"* columnSize\" , input.columnSize)\n",
    "    for i in range(input.channelSize):\n",
    "        print(\"channel =\",i)\n",
    "        for j in range(input.rowSize):\n",
    "            for k in range(input.columnSize):\n",
    "                print(\"{:.6f}\".format(input.feature[i][j][k]),end = \" \")\n",
    "            print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "channel size = 2 size = 4 * columnSize 4\n",
      "channel = 0\n",
      "0.463846 0.055004 0.396108 0.269861 \n",
      "0.957129 0.925554 0.287401 0.154724 \n",
      "0.495547 0.501303 0.220264 0.879485 \n",
      "0.763517 0.488933 0.313242 0.296595 \n",
      "channel = 1\n",
      "0.981330 0.191742 0.749812 0.960439 \n",
      "0.534702 0.090567 0.222822 0.280778 \n",
      "0.069963 0.583073 0.444843 0.235924 \n",
      "0.061166 0.516728 0.363048 0.215814 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",2,4,4)\n",
    "Matrix_Print(feature)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Matrix_Convolution(struct Matrix *input,struct Matrix **kernel,struct Matrix *output ,int paddingAmt,int stride)\n",
    "<br/>\n",
    "变量：   (Matrix *)input = 待卷积的特征矩阵<br/>\n",
    "        (Matrix **)kernel = 为卷积核 (通常卷积核不止一个，所以输入为一列的卷积核)<br/>\n",
    "        (Matrix *)output = 卷积后的特征矩阵<br/>\n",
    "        (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "        (int) stride = 步长<br/>\n",
    "功能：计算input与kernel的卷积答案并将答案存储在output中，paddingAmt为其填充大小，stride为其步长<br/>\n",
    "output大小为:<br/>\n",
    "kernel的数量 X 取整(input的行数 -2 * 填充大小 - 卷积核的行数)/步长）+1 X 取整(input的例数 -2 * 填充大小 - 卷积核的例数)/步长）+1<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k * x * y * z<br/>\n",
    "\n",
    "调用例子 ： Matrix_Convolution(feature,kernel,ans,1,2)<br/>\n",
    "如上所示将用kernel对feature进行填充为1，步长为2的卷积并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Matrix_Convolution(input:Matrix,kernel: list[Matrix], output:Matrix , paddingAmt:int , stride:int):\n",
    "    channel = input.channelSize\n",
    "    outputSize = output.rowSize\n",
    "    kernelSize = kernel[0].rowSize\n",
    "    kernelAmt = output.channelSize\n",
    "    for x in range (kernelAmt):\n",
    "        for j in range (outputSize):\n",
    "            for k in range (outputSize):\n",
    "                temp = 0\n",
    "                for i in range (channel):\n",
    "                    for y in range (kernelSize):\n",
    "                        for z in range (kernelSize):\n",
    "                            rowLoc = j * stride + y - paddingAmt #用于确定其在input内的行索引\n",
    "                            columnLoc = k * stride + z - paddingAmt #用于确定其在input内的例索引\n",
    "                            #用于判断是否为padding区域若为padding区域不需要计算\n",
    "                            if ((rowLoc < 0 or rowLoc >= input.rowSize ) or (columnLoc < 0 or columnLoc >= input.columnSize )):\n",
    "                                continue\n",
    "                            #若位于该位置进行点乘并叠加到暂时变量\n",
    "                            temp = temp + input.feature[i][rowLoc][columnLoc] * kernel[x].feature[i][y][z]\n",
    "                output.feature[x][j][k] = temp #将暂时变量的值存到答案中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "channel size = 12 size = 3 * columnSize 3\n",
      "channel = 0\n",
      "2.861483 5.484993 3.656394 \n",
      "4.415974 8.690598 5.129236 \n",
      "2.826769 5.639196 3.637814 \n",
      "channel = 1\n",
      "3.691501 6.227852 4.485535 \n",
      "4.842864 8.970283 5.591837 \n",
      "3.404345 6.639315 3.907929 \n",
      "channel = 2\n",
      "3.735665 6.442918 3.365600 \n",
      "5.387335 7.883470 4.222595 \n",
      "2.708169 5.333133 2.654628 \n",
      "channel = 3\n",
      "3.221355 5.016346 2.650469 \n",
      "4.689790 8.076442 3.074036 \n",
      "3.268157 5.565744 2.594296 \n",
      "channel = 4\n",
      "3.730161 6.556230 3.147093 \n",
      "4.424768 9.722346 4.706911 \n",
      "3.257290 6.638313 3.617179 \n",
      "channel = 5\n",
      "2.032599 5.177394 3.587141 \n",
      "3.252410 7.495872 4.062761 \n",
      "3.174226 5.053252 2.668802 \n",
      "channel = 6\n",
      "3.372055 6.106624 4.086923 \n",
      "4.384296 9.456740 5.234183 \n",
      "3.328302 6.694340 4.280468 \n",
      "channel = 7\n",
      "2.790497 4.003550 3.275084 \n",
      "4.312906 7.639253 4.060403 \n",
      "2.813390 4.683752 3.054389 \n",
      "channel = 8\n",
      "3.083744 5.498510 2.109720 \n",
      "4.160560 9.005725 4.122452 \n",
      "3.075448 5.891340 3.246418 \n",
      "channel = 9\n",
      "3.528448 5.040360 3.766196 \n",
      "3.742920 7.944966 3.699381 \n",
      "2.816266 5.198250 2.808086 \n",
      "channel = 10\n",
      "2.731141 4.216626 2.830907 \n",
      "4.810425 6.810141 3.166104 \n",
      "2.225406 4.119238 1.910646 \n",
      "channel = 11\n",
      "3.443496 5.715486 3.724571 \n",
      "4.191918 7.058447 4.722563 \n",
      "2.104012 5.099679 3.118845 \n"
     ]
    }
   ],
   "source": [
    "input = Matrix(\"Random\",3,5,5)\n",
    "kernel = [Matrix(\"Random\",3,3,3) for kernelAmt in range (12)]\n",
    "output = Matrix(\"Random\",12,3,3)\n",
    "paddingAmt = 1\n",
    "stride = 2\n",
    "Matrix_Convolution(input ,kernel, output , paddingAmt , stride)\n",
    "Matrix_Print(output)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Matrix_Sum(struct Matrix *input,struct Matrix *addTerm,struct Matrix *output);\n",
    "<br/>\n",
    "变量：   (Matrix *)input = 待加的特征矩阵<br/>\n",
    "        (Matrix *)addTerm = 被加的特征矩阵<br/> \n",
    "        (Matrix *)output = 相加后的特征矩阵<br/>\n",
    "功能：计算input与addTerm的向加答案并将答案存储在output中，由于都是指针所以output如果填入input的话则将答案存储回input中<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k <br/>\n",
    "\n",
    "调用例子 ： Matrix_Sum(feature,addTerm,ans)<br/>\n",
    "如上所示将feature和addTerm进行相加并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Matrix_Sum(input : Matrix ,addTerm : Matrix ,output : Matrix ):\n",
    "    channel = input.channelSize\n",
    "    row = input.rowSize\n",
    "    column = input.columnSize\n",
    "    for i in range (channel):\n",
    "        for j in range (row):\n",
    "            for k in range (column):\n",
    "                #简单的相加，遍历整个特征矩阵\n",
    "                output.feature[i][j][k] = input.feature[i][j][k] + addTerm.feature[i][j][k]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "channel size = 2 size = 3 * columnSize 3\n",
      "channel = 0\n",
      "0.527471 0.007150 0.684503 \n",
      "0.342191 0.798709 0.917567 \n",
      "0.810716 0.386779 0.229733 \n",
      "channel = 1\n",
      "0.010114 0.449184 0.753212 \n",
      "0.296020 0.841618 0.994437 \n",
      "0.809964 0.189660 0.155757 \n"
     ]
    }
   ],
   "source": [
    "input = Matrix(\"Zero\",2,3,3)\n",
    "addTerm = Matrix(\"Random\",2,3,3)\n",
    "Matrix_Sum(input,addTerm,input)\n",
    "Matrix_Print(input)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Matrix_Multiply(struct Matrix *input,struct Matrix *mulTerm,struct Matrix *output);\n",
    "<br/>\n",
    "变量：   (Matrix *)input = 待乘的特征矩阵<br/>\n",
    "        (Matrix *)mulTerm = 被乘的特征矩阵<br/> \n",
    "        (Matrix *)output = 相乘后的特征矩阵<br/>\n",
    "功能：计算input与addTerm的叉乘答案并将答案存储在output中<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k * z<br/>\n",
    "\n",
    "调用例子 ： Matrix_Multiply(feature,mulTerm,ans)<br/>\n",
    "如上所示将feature和addTerm进行叉乘并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Matrix_Multiply(input : Matrix ,mulTerm : Matrix ,output : Matrix ):\n",
    "    channel = input.channelSize\n",
    "    row = input.rowSize\n",
    "    column = input.columnSize\n",
    "    mulSize = mulTerm.columnSize\n",
    "    for i in range (channel):\n",
    "        for z in range (mulSize):\n",
    "            for j in range (row):\n",
    "                temp = 0 #存储行X列相加的答案\n",
    "                for k in range (column):\n",
    "                    temp = temp + input.feature[i][j][k] * mulTerm.feature[i][k][z]\n",
    "                output.feature[i][j][z] = temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "0.958077 1.160086 \n",
      "0.577696 0.585037 \n",
      "channel = 1\n",
      "0.357706 1.168849 \n",
      "0.446968 1.405853 \n"
     ]
    }
   ],
   "source": [
    "input = Matrix(\"Random\",2,2,3)\n",
    "mulTerm = Matrix(\"Random\",2,3,2)\n",
    "ans = Matrix(\"Zero\",2,2,2)\n",
    "Matrix_Multiply(input,mulTerm,ans)\n",
    "Matrix_Print(ans)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Matrix_ToZero(struct Matrix *input)<br/>\n",
    "<br/>\n",
    "功能 ：将特征矩阵所有特征值变为0<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k<br/>\n",
    "\n",
    "调用例子 ： Matrix_ToZero(feature)<br/>\n",
    "如上所示将特征矩阵feature变量中的值全变为0。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Matrix_ToZero(input:Matrix):\n",
    "    for i in range(input.channelSize):\n",
    "        for j in range(input.rowSize):\n",
    "            for k in range(input.columnSize):\n",
    "                input.feature[i][j][k] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before to zero ...\n",
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "0.833939 0.873763 \n",
      "0.824797 0.466952 \n",
      "channel = 1\n",
      "0.864338 0.784714 \n",
      "0.622560 0.446398 \n",
      "after to zero ...\n",
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "0.000000 0.000000 \n",
      "0.000000 0.000000 \n",
      "channel = 1\n",
      "0.000000 0.000000 \n",
      "0.000000 0.000000 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",2,2,2)\n",
    "print(\"before to zero ...\")\n",
    "Matrix_Print(feature)\n",
    "Matrix_ToZero(feature)\n",
    "print(\"after to zero ...\")\n",
    "Matrix_Print(feature)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>以下皆为 批量归一化函数算法分析</br>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基本数据结构</br>\n",
    "struct BN</br>\n",
    "{</br>\n",
    "    int channelSize = 描述该层的通道数 </br>\n",
    "</br>\n",
    "    float *beta = 该层各通道的beta常数</br>\n",
    "    float *gamma = 该层各通道的gamma常数</br>\n",
    "    float *mean = 该层各通道的平均数</br>\n",
    "    float *variance  = 该层各通道的方差</br>\n",
    "};</br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BN:\n",
    "    channelSize = 0\n",
    "\n",
    "    beta = []\n",
    "    gamma = []\n",
    "    mean =[]\n",
    "    variance = []\n",
    "    runMean = []\n",
    "    runVar = []\n",
    "\n",
    "    def __init__(self,channelSize):\n",
    "        self.channelSize = channelSize\n",
    "        self.beta = [0 for i in range (channelSize)]\n",
    "        self.gamma = [1 for i in range (channelSize)]\n",
    "        self.mean = [0 for i in range (channelSize)]\n",
    "        self.variance = [0 for i in range (channelSize)]\n",
    "        self.runMean = [0 for i in range (channelSize)]\n",
    "        self.runVar = [1 for i in range (channelSize)]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "struct BN *BN_Init(int channel);\n",
    "\n",
    "功能 ：该函数为批标准化的初始化操作，通过申请一个BN数据结构的空间，再将其指针地址返回。其中所有beta默认为0.1，gamma为1。\n",
    "\n",
    "时间复杂度O(N) ： i\n",
    "\n",
    "调用例子 ： struct BN *coeff = BN_Init(6)\n",
    "如上所示将初始化一个通道数为6的批标准化数据结构并将其指针地址存储到coeff变量中。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void *BN_GetCoeff(struct Matrix **input , struct BN *coeff)<br/>\n",
    "</br>\n",
    "功能 ：获取该层的批标准化的平均数以及方差<br/>\n",
    "\n",
    "时间复杂度O(N) ：2 * (num * i * j * k)<br/>\n",
    "\n",
    "调用例子 ： BN_GetCoeff(feature,coeff)<br/>\n",
    "如上所示将整批特征矩阵feature代入求出该层的平均值以及方差。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def BN_GetCoeff(input:list[Matrix],coeff:BN,momentum:float):\n",
    "    BATCH = 5 #批次大小建议宏定义\n",
    "    channel = coeff.channelSize\n",
    "    row = input[0].rowSize\n",
    "    column = input[0].columnSize\n",
    "    amount = BATCH * row * column\n",
    "    for i in range(channel):\n",
    "        temp = 0 #用以存储加总值\n",
    "        for num in range(BATCH):\n",
    "            for j in range (row):\n",
    "                for k in range (column):\n",
    "                    temp = temp + input[num].feature[i][j][k]\n",
    "        coeff.mean[i] = temp/amount\n",
    "        coeff.runMean[i] = (1-momentum)*coeff.runMean[i] + momentum * coeff.mean[i]\n",
    "\n",
    "    for i in range(channel):\n",
    "        temp = 0 #用以存储加总值\n",
    "        for num in range(BATCH):\n",
    "            for j in range (row):\n",
    "                for k in range (column):\n",
    "                    temp = temp + (input[i].feature[i][j][k] - coeff.mean[i])**2\n",
    "        coeff.variance[i] = temp/amount\n",
    "        coeff.runVar[i] = (1-momentum)*coeff.runVar[i] + momentum*temp/(amount-1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after get coeff [0.5565143744309039, 0.4912936417996365, 0.4296305643353902]\n",
      "after get coeff [0.08590840663990257, 0.06688436501182544, 0.04546311377800545]\n"
     ]
    }
   ],
   "source": [
    "BATCH = 5\n",
    "feature = [Matrix(\"Random\",3,3,3) for featureAmt in range (BATCH)]\n",
    "coeff = BN(3)\n",
    "BN_GetCoeff(feature,coeff,0.1)\n",
    "print(\"after get coeff\",coeff.mean)\n",
    "print(\"after get coeff\",coeff.variance)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void BN_BatchNorm(struct Matrix *input , struct Matrix *output , struct BN *coeff )<br/>\n",
    "</br>\n",
    "功能 ：对该层特征矩阵进行批标准化<br/>\n",
    "具体公式如下：</br>\n",
    "x_hat = x-mean/(variance+epsilon)^0.5  (epsilon用于防止分母为零)</br>\n",
    "y = x_hat*gamma + beta</br>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k<br/>\n",
    "\n",
    "调用例子 ： BN_BatchNorm(feature,ans,coeff)<br/>\n",
    "如上所示将特征矩阵feature与coeff进行批标准化并将标准化后的值输入ans。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def BN_BatchNorm(input:Matrix , output:Matrix , coeff:BN):\n",
    "    EPSILON = 0.1 #建议宏定义\n",
    "    channel = coeff.channelSize\n",
    "    row = input.rowSize\n",
    "    column = input.columnSize\n",
    "    for i in range(channel):\n",
    "        for j in range (row):\n",
    "            for k in range (column):\n",
    "                output.feature[i][j][k] = coeff.gamma[i]*(input.feature[i][j][k] -coeff.mean[i]) / (coeff.variance[i] + EPSILON)**0.5 + coeff.beta[i] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before batch normalize...\n",
      "channel size = 3 size = 3 * columnSize 3\n",
      "channel = 0\n",
      "0.323364 0.212317 0.778605 \n",
      "0.596198 0.312863 0.176205 \n",
      "0.426114 0.429056 0.786290 \n",
      "channel = 1\n",
      "0.054830 0.992671 0.922197 \n",
      "0.125630 0.188861 0.281913 \n",
      "0.682305 0.057214 0.218566 \n",
      "channel = 2\n",
      "0.596378 0.186927 0.498418 \n",
      "0.689648 0.517896 0.230297 \n",
      "0.972134 0.488334 0.151764 \n",
      "after batch normalize...\n",
      "channel size = 3 size = 3 * columnSize 3\n",
      "channel = 0\n",
      "-0.183019 -0.473806 1.009072 \n",
      "0.531423 -0.210516 -0.568369 \n",
      "0.086043 0.093745 1.029198 \n",
      "channel = 1\n",
      "-0.958501 1.426956 1.247703 \n",
      "-0.778418 -0.617585 -0.380901 \n",
      "0.637522 -0.952438 -0.542028 \n",
      "channel = 2\n",
      "0.403633 -0.555184 0.174238 \n",
      "0.622045 0.219849 -0.453625 \n",
      "1.283545 0.150623 -0.637526 \n"
     ]
    }
   ],
   "source": [
    "BATCH = 5\n",
    "featureExample = [Matrix(\"Random\",3,3,3) for featureAmt in range (BATCH)]\n",
    "feature = featureExample[0]\n",
    "coeff = BN(3)\n",
    "BN_GetCoeff(featureExample,coeff)\n",
    "print(\"before batch normalize...\")\n",
    "Matrix_Print(feature)\n",
    "BN_BatchNorm(feature,feature,coeff)\n",
    "print(\"after batch normalize...\")\n",
    "Matrix_Print(feature)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void **BNBatch_BatchNorm(struct Matrix **input ,struct Matrix **output, struct BN *coeff )<br/>\n",
    "</br>\n",
    "功能 ：对该层整批特征矩阵进行批标准化<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * i * j * k<br/>\n",
    "\n",
    "调用例子 ： BNBatch_BatchNorm(feature,ans,coeff)<br/>\n",
    "如上所示将整批特征矩阵feature与coeff进行批标准化并将标准化后的值整批输入ans。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def BNBatch_BatchNorm(input:list[Matrix] , output:list[Matrix] , coeff:BN):\n",
    "    BATCH = 5 #批次大小建议宏定义\n",
    "    for num in range (BATCH):\n",
    "        BN_BatchNorm(input[num],output[num],coeff)\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>以下皆为 前向传导时使用的函数算法分析</br>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Front_ReLU (struct Matrix *input,struct Matrix *output);</br>\n",
    "\n",
    "变量：   (Matrix *)input = 待激活的特征矩阵<br/>\n",
    "        (Matrix *)output = 激活后的特征矩阵<br/>\n",
    "功能：ReLU为激活函数即将负数为0<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k <br/>\n",
    "\n",
    "调用例子 ： Front_ReLU(feature,ans)<br/>\n",
    "如上所示将feature进行激活并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Front_ReLU(input:Matrix,output:Matrix):\n",
    "    for i in range(input.channelSize):\n",
    "        for j in range(input.rowSize):\n",
    "            for k in range(input.columnSize):\n",
    "                if(input.feature[i][j][k] < 0):\n",
    "                    output.feature[i][j][k] = 0 \n",
    "                else:\n",
    "                    output.feature[i][j][k] = input.feature[i][j][k]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before ReLU ...\n",
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "1.000000 -1.000000 \n",
      "3.000000 -4.000000 \n",
      "channel = 1\n",
      "-6.000000 7.000000 \n",
      "9.000000 -2.000000 \n",
      "after ReLU ...\n",
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "1.000000 0.000000 \n",
      "3.000000 0.000000 \n",
      "channel = 1\n",
      "0.000000 7.000000 \n",
      "9.000000 0.000000 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",2,2,2)\n",
    "feature.feature = [[[1, -1], [3, -4]], [[-6, 7], [9, -2]]]\n",
    "ans = Matrix(\"Zero\",2,2,2)\n",
    "print(\"before ReLU ...\")\n",
    "Matrix_Print(feature)\n",
    "Front_ReLU(feature,ans)\n",
    "print(\"after ReLU ...\")\n",
    "Matrix_Print(ans)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_ReLU (struct Matrix **input,struct Matrix **output);</br>\n",
    "\n",
    "变量：   (Matrix **)input = 一批待激活的特征矩阵<br/>\n",
    "        (Matrix **)output = 一批激活后的特征矩阵<br/>\n",
    "功能：往往前传导时我们会一整批的数据一起处理，此函数为整批一起做激活<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Front_ReLU(feature,ans)<br/>\n",
    "如上所示将整批feature进行激活并将结果整批地存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_ReLU(input:list[Matrix],output:list[Matrix]):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        Front_ReLU(input[num],output[num])  \n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Front_MaxPooLing(struct Matrix *input,struct Matrix *output,int poolingSize,int paddingAmt,int stride)\n",
    "<br/>\n",
    "变量：   (Matrix *)input = 待池化的特征矩阵<br/>\n",
    "        (Matrix *)output = 池化后的特征矩阵<br/>\n",
    "        (int) poolingSize = 池化大小 <br/>\n",
    "        (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "        (int) stride = 步长<br/>\n",
    "功能：选取每个input池化区域的最大值并将答案存储在output中，paddingAmt为其填充大小，stride为其步长<br/>\n",
    "output大小为:<br/>\n",
    "kernel的数量 X 取整(input的行数 -2 * 填充大小 - 池化大小)/步长）+1 X 取整(input的例数 -2 * 填充大小 - 池化大小)/步长）+1<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k * y * z<br/>\n",
    "\n",
    "调用例子 ： Front_MaxPooling(feature,ans,3,1,2)<br/>\n",
    "如上所示对feature进行填充为1，步长为2的最大池化并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Front_MaxPooling(input:Matrix, output:Matrix , poolingSize:int,  paddingAmt:int , stride:int):\n",
    "    channel = input.channelSize\n",
    "    outputSize = output.rowSize\n",
    "    for i in range (channel):\n",
    "        for j in range (outputSize):\n",
    "            for k in range (outputSize):\n",
    "                maximun = -9999 #设初始为一极小数保证池化区域内号码必大于初始数目\n",
    "                for y in range (poolingSize):\n",
    "                    for z in range (poolingSize):\n",
    "                        rowLoc = j * stride + y - paddingAmt #用于确定其在input内的行索引\n",
    "                        columnLoc = k * stride + z - paddingAmt #用于确定其在input内的例索引\n",
    "                        #用于判断是否为padding区域若为padding区域不需要计算\n",
    "                        if ((rowLoc < 0 or rowLoc >= input.rowSize ) or (columnLoc < 0 or columnLoc >= input.columnSize )):\n",
    "                            if maximun < 0:\n",
    "                                maximun = 0 #填充区域值\n",
    "                            continue\n",
    "                        #若位于该位置进行点乘并叠加到暂时变量\n",
    "                        if(input.feature[i][j][k] > maximun):\n",
    "                            maximun = input.feature[i][j][k]\n",
    "                output.feature[i][j][k] = maximun #将最大值存到答案中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before maxPooling ...\n",
      "channel size = 1 size = 5 * columnSize 5\n",
      "channel = 0\n",
      "0.574301 0.836090 0.571234 0.702342 0.956653 \n",
      "0.174107 0.110991 0.494246 0.808413 0.186453 \n",
      "0.882715 0.401943 0.700299 0.239343 0.275724 \n",
      "0.884415 0.136321 0.090593 0.097804 0.257168 \n",
      "0.025628 0.913057 0.491138 0.302521 0.388079 \n",
      "after maxPooling ...\n",
      "channel size = 1 size = 3 * columnSize 3\n",
      "channel = 0\n",
      "0.574301 0.836090 0.571234 \n",
      "0.174107 0.110991 0.494246 \n",
      "0.882715 0.401943 0.700299 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",1,5,5)\n",
    "ans = Matrix(\"Zero\",1,3,3)\n",
    "print(\"before maxPooling ...\")\n",
    "Matrix_Print(feature)\n",
    "Front_MaxPooling(feature,ans,3,1,2)\n",
    "print(\"after maxPooling ...\")\n",
    "Matrix_Print(ans)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_MaxPooLing(struct Matrix **input,struct Matrix **output,int poolingSize,int paddingAmt,int stride)\n",
    "<br/>\n",
    "变量：   (Matrix **)input = 整批待池化的特征矩阵<br/>\n",
    "        (Matrix **)output = 整批池化后的特征矩阵<br/>\n",
    "        (int) poolingSize = 池化大小 <br/>\n",
    "        (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "        (int) stride = 步长<br/>\n",
    "功能：往往前传导时我们会一整批的数据一起处理，此函数为整批一起做最大池化<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * i * j * k * y * z<br/>\n",
    "\n",
    "调用例子 ： Front_MaxPooling(feature,ans,3,1,2)<br/>\n",
    "如上所示对整批feature进行填充为1，步长为2的最大池化并将结果存储在整批ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_MaxPooling(input:list[Matrix],output:list[Matrix],poolingSize:int , paddingAmt:int , stride:int):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        Front_MaxPooling(input[num],output[num],poolingSize,paddingAmt,stride)  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_Convolution(struct Matrix **input,struct Matrix **kernel,struct Matrix **output ,int paddingAmt,int stride)\n",
    "<br/>\n",
    "变量：   (Matrix **)input = 整批待卷积的特征矩阵<br/>\n",
    "        (Matrix **)kernel = 卷积核<br/>\n",
    "        (Matrix **)output = 整批卷积后的特征矩阵<br/>\n",
    "        (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "        (int) stride = 步长<br/>\n",
    "功能：往往前传导时我们会一整批的数据一起处理，此函数为整批特征矩阵对同一批卷积核一起做卷积<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * i * j * k * x * y * z<br/>\n",
    "\n",
    "调用例子 ： FrontBatch_Convolution(feature,kernel,ans,1,2)<br/>\n",
    "如上所示将整批feature对卷积核一批kernel进行填充为1，步长为2的卷积并将结果存储在整批ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_Convolution(input:list[Matrix],kernel:list[Matrix],output:list[Matrix], paddingAmt:int , stride:int):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        Matrix_Convolution(input[num],kernel,output[num],paddingAmt,stride)  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Front_GlobalAverage(struct Matrix *input,struct Matrix *output)\n",
    "<br/>\n",
    "变量：   (Matrix *)input = 待全局平均池化的特征矩阵<br/>\n",
    "        (Matrix *)output = 全局平均池化后的特征矩阵<br/>\n",
    "功能：将每个通道内的特征值相加，并分通道求出每个通道的平均值，每个通道的平均值存入output的每一行<br/>\n",
    "\n",
    "时间复杂度O(N) ：i * j * k <br/>\n",
    "\n",
    "调用例子 ： Front_GlobalAverage(feature,ans)<br/>\n",
    "如上所示对feature进行全局平均池化并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Front_GlobalAverage(input:Matrix,output:Matrix):\n",
    "    featureSize = input.rowSize * input.columnSize\n",
    "    for i in range(input.channelSize):\n",
    "        temp = 0\n",
    "        for j in range(input.rowSize):\n",
    "            for k in range(input.columnSize):\n",
    "                temp = temp + input.feature[i][j][k]\n",
    "        output.feature[0][i][0] = temp/featureSize\n",
    "                    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before GlobalAverage ...\n",
      "channel size = 3 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "0.847114 0.003265 \n",
      "0.988987 0.152371 \n",
      "channel = 1\n",
      "0.279427 0.075017 \n",
      "0.727833 0.619991 \n",
      "channel = 2\n",
      "0.881412 0.432683 \n",
      "0.274481 0.931530 \n",
      "after GlobalAverageg ...\n",
      "channel size = 1 size = 3 * columnSize 1\n",
      "channel = 0\n",
      "0.497934 \n",
      "0.425567 \n",
      "0.630027 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",3,2,2)\n",
    "ans = Matrix(\"Zero\",1,3,1)\n",
    "print(\"before GlobalAverage ...\")\n",
    "Matrix_Print(feature)\n",
    "Front_GlobalAverage(feature,ans)\n",
    "print(\"after GlobalAverageg ...\")\n",
    "Matrix_Print(ans)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_GlobalAverage(struct Matrix **input,struct Matrix **output);</br>\n",
    "\n",
    "变量：   (Matrix **)input = 一批待全局池化的特征矩阵<br/>\n",
    "        (Matrix **)output = 一批全局池化的特征矩阵<br/>\n",
    "功能：往往前传导时我们会一整批的数据一起处理，此函数为整批一起做全局平均池化<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * i * j * k <br/>\n",
    "\n",
    "调用例子 ： FrontBatch_GlobalAverage(feature,ans)<br/>\n",
    "如上所示将整批feature进行全局平均池化并将结果整批地存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_GlobalAverage(input:list[Matrix],output:list[Matrix]):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        Front_GlobalAverage(input[num],output[num])  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Front_FullConnect(struct Matrix *input,struct Matrix *weight,struct Matrix *output,struct Matrix *bias);</br>\n",
    "\n",
    "变量：   (Matrix *)input = 待叉乘的特征矩阵<br/>\n",
    "        (Matrix *)weight = 权重特征矩阵<br/>\n",
    "        (Matrix *)output = 全连接后的特征矩阵<br/>\n",
    "        (Matrix *)bias =  每个通道的偏差量\n",
    "功能：全连接简单来说就是特征矩阵与权重矩阵的叉乘再加上偏差量<br/>\n",
    "\n",
    "时间复杂度O(N) ： i * z * j * k + i * j * k <br/>\n",
    "\n",
    "调用例子 ： Front_FullConnect(feature,weight,ans,bias)<br/>\n",
    "如上所示将feature与weight进行叉乘再加上偏差量最后将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Front_FullConnect(input:Matrix , weight:Matrix , output:Matrix , bias : Matrix):\n",
    "    #input 尺寸 (1 X 上层通道数 X 1)\n",
    "    #weight 尺寸 (1 X 需预测的分类数 X 上层通道数)\n",
    "    #output 尺寸 (1 X 需预测的分类数 X 1)\n",
    "    #bias 尺寸 (1 X 需预测的分类数 X 1)\n",
    "    Matrix_Multiply(input,weight,output) #先进行叉乘并存储在output中\n",
    "    Matrix_Sum(output,bias,output) #再加上偏差量"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_FUllConnct(struct Matrix **input,struct Matrix *weight,struct Matrix **output,struct Matrix *bias)\n",
    "</br>\n",
    "\n",
    "变量：   (Matrix **)input = 整批待叉乘的特征矩阵<br/>\n",
    "        (Matrix *)weight = 权重特征矩阵<br/>\n",
    "        (Matrix **)output = 整批全连接后的特征矩阵<br/>\n",
    "        (Matrix *)bias =  每个通道的偏差量\n",
    "功能：往往前传导时我们会一整批的数据一起处理，此函数为整批特征矩阵对同一权重与偏差量做全连接<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * （i * z * j * k + i * j * k） <br/>\n",
    "\n",
    "调用例子 ： FrontBatch_FUllConnct(feature,weight,ans,bias)<br/>\n",
    "如上所示将整批feature与weight进行叉乘再加上偏差量最后将结果存储在整批ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_FUllConnct(input:list[Matrix] , weight:Matrix , output:list[Matrix] , bias : Matrix):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        Front_FullConnect(input[num],weight,output[num],bias) "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Front_Softmax(struct Matrix *input,struct Matrix *output)\n",
    "<br/>\n",
    "变量：   (Matrix *)input = 待激活的特征矩阵<br/>\n",
    "        (Matrix *)output = 激活后的特征矩阵<br/>\n",
    "功能：将每行内的特征值变成e的指数，并分通道求出每行的平均值，每行的百分比存入output的每一行，其目的为方便求出预测值<br/>\n",
    "\n",
    "时间复杂度O(N) ： j * j  <br/>\n",
    "\n",
    "调用例子 ： Front_Softmax(feature,ans)<br/>\n",
    "如上所示对feature进行softmax激活并将结果存储在ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def Front_Softmax(input:Matrix , output:Matrix ):\n",
    "    size = input.rowSize\n",
    "    summation = 0 #用于储存所有特征的和，以计算激活后的值\n",
    "    for j in range (size):\n",
    "        temp = math.exp(input.feature[0][j][0])\n",
    "        summation = summation + temp \n",
    "        output.feature[0][j][0] = temp #待取百分比\n",
    "    for j in range (size):\n",
    "        output.feature[0][j][0] = output.feature[0][j][0]/summation #转换为百分比"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before Softmax ...\n",
      "channel size = 1 size = 5 * columnSize 1\n",
      "channel = 0\n",
      "1.000000 \n",
      "2.000000 \n",
      "3.000000 \n",
      "4.000000 \n",
      "5.000000 \n",
      "after Softmaxg ...\n",
      "channel size = 1 size = 5 * columnSize 1\n",
      "channel = 0\n",
      "0.011656 \n",
      "0.031685 \n",
      "0.086129 \n",
      "0.234122 \n",
      "0.636409 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",1,5,1)\n",
    "feature.feature = [[[1], [2], [3] , [4] , [5]]]\n",
    "ans = Matrix(\"Zero\",1,5,1)\n",
    "print(\"before Softmax ...\")\n",
    "Matrix_Print(feature)\n",
    "Front_Softmax(feature,ans)\n",
    "print(\"after Softmaxg ...\")\n",
    "Matrix_Print(ans) #其和必为1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_Softmax(struct Matrix **input,struct Matrix **output)\n",
    "</br>\n",
    "\n",
    "变量：   (Matrix **)input = 整批待激活的特征矩阵<br/>\n",
    "        (Matrix **)output = 整批激活后的特征矩阵<br/>\n",
    "功能：往往前传导时我们会一整批的数据一起处理，此函数为整批特征矩阵做softmax激活<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * j * j <br/>\n",
    "\n",
    "调用例子 ： FrontBatch_Softmax(feature,ans)<br/>\n",
    "如上所示将整批feature进行激活最后将结果存储在整批ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_Softmax(input:list[Matrix] , output:list[Matrix] ):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        Front_Softmax(input[num],output[num]) "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "int Front_Predict(struct Matrix *input)\n",
    "<br/>\n",
    "变量：   (Matrix *)input = softmax后的特征矩阵<br/>\n",
    "功能：找出特征矩阵最大值，所在的索引行则为预测答案<br/>\n",
    "\n",
    "时间复杂度O(N) ： j  <br/>\n",
    "\n",
    "调用例子 ： Front_Predict(feature)<br/>\n",
    "如上所示对feature取最大值，并返回其索引。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Front_Predict(input:Matrix):\n",
    "    maximun = 0 \n",
    "    index = 0 \n",
    "    for j in range(input.rowSize):\n",
    "        if(input.feature[0][j][0] > maximun):\n",
    "            maximun = input.feature[0][j][0]\n",
    "            index = j #最大值的索引\n",
    "    return index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before predict ...\n",
      "channel size = 1 size = 5 * columnSize 1\n",
      "channel = 0\n",
      "0.011656 \n",
      "0.031685 \n",
      "0.086129 \n",
      "0.234122 \n",
      "0.636409 \n",
      "predict value\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature = ans\n",
    "print(\"before predict ...\")\n",
    "Matrix_Print(feature)\n",
    "print(\"predict value\")\n",
    "Front_Predict(feature)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void FrontBatch_Predict(struct Matrix **input, int *output)\n",
    "<br/>\n",
    "变量：   (Matrix **)input = 整批softmax后的特征矩阵<br/>\n",
    "        (int *)output = 整批的预测答案<br/>\n",
    "功能：逐个进行预测并将每个的预测结果存储到output中<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * j  <br/>\n",
    "\n",
    "调用例子 ： FrontBatch_Predict(feature,ans)<br/>\n",
    "如上所示对整批feature分别取最大值，并分别将其索引存储到ans中。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_Predict(input:list[Matrix],output:list[int]):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    for num in range(BATCH):\n",
    "        output[num] = Front_Predict(input[num],output[num]) "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "float FrontBatch_Accurancy(int *input,int *testCaseLabel,int index)</br>\n",
    "\n",
    "<br/>\n",
    "变量： (int *)input = 整批的预测答案<br/>\n",
    "       (int *)testCaseLabel = 整批的实际答案<br/>\n",
    "       (int ) index = 实际答案的索引位置<br/>\n",
    "功能：逐个进行对比，最后反馈正确率<br/>\n",
    "\n",
    "时间复杂度O(N) ： num  <br/>\n",
    "\n",
    "调用例子 ： FrontBatch_Accurancy(predict,testCaseAns,index)<br/>\n",
    "如上所示对整批predict与index索引到的testcaseAns整批做对比，并输出其误差。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def FrontBatch_Accurancy(input:list[int],testCaseLabel:list[int],index:int):\n",
    "    BATCH = 5 #建议宏定义批次大小\n",
    "    correct = 0 #use to count the correct time\n",
    "    start = index*BATCH #索引开始位置\n",
    "    for num in range(BATCH):\n",
    "        if(input[num] != testCaseLabel[start+num]):\n",
    "            correct = correct + 1\n",
    "    return correct/BATCH"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>以下皆为 后向传导(Back propagation)时使用的函数算法分析</br>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_ToZero(float *gradient,int size)</br>\n",
    "\n",
    "<br/>\n",
    "变量： (float *)gradient = 一列待归零的梯度值<br/>\n",
    "       (int)size = 列的大小<br/>\n",
    "功能：逐列进行归零操作<br/>\n",
    "\n",
    "时间复杂度O(N) ： i  <br/>\n",
    "\n",
    "调用例子 ： Back_ToZero(gradient,size)<br/>\n",
    "如上所示对大小为size的gradient进行归零操作。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_ToZero(gradient:list[float],size:int):\n",
    "    for i in range(size):\n",
    "        gradient[i] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient before to zero ...\n",
      "[0.1, 0.2, 0.3, 0.4, -0.1]\n",
      "gradient after to zero ...\n",
      "[0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "gradient = [0.1,0.2,0.3,0.4,-0.1]\n",
    "size = 5\n",
    "print(\"gradient before to zero ...\")\n",
    "print(gradient)\n",
    "Back_ToZero(gradient,size)\n",
    "print(\"gradient after to zero ...\")\n",
    "print(gradient)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_Descent(struct Matrix *kernel,struct Matrix *gradient)</br>\n",
    "\n",
    "<br/>\n",
    "变量： (struct Matrix*)kernel = 待梯度递减的卷积核<br/>\n",
    "       (struct Matrix*)gradient = 卷积核的梯度<br/>\n",
    "功能：对该卷积核进行梯度递减即更新卷积核<br/>\n",
    "\n",
    "时间复杂度O(N) ： i * x * y  <br/>\n",
    "\n",
    "调用例子 ： Back_Descent(kernel,gradient)<br/>\n",
    "如上所示根据gradient中每个卷积核中的梯度，逐个进行递减。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_Descent(kernel:Matrix ,gradient:Matrix ):\n",
    "    learningRate = 0.001 # 学习率建议宏定义\n",
    "    decayWeight = 0.1 # 衰变权重建议宏定义\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    decay = (1-learningRate*decayWeight/BATCH) #为衰变函数直接计算并存储为一个变量减少计算时间\n",
    "    learning = (learningRate/BATCH)  #为递减函数直接计算并存储为一个变量减少计算时间\n",
    "    channel = kernel.channelSize\n",
    "    row = kernel.rowSize\n",
    "    column = kernel.columnSize\n",
    "    for i in range(channel):\n",
    "        for x in range(row):\n",
    "            for y in range(column):\n",
    "                kernel.feature[i][x][y] = kernel.feature[i][x][y]*decay - learning*gradient.feature[i][x][y]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before descent...\n",
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "1.000000 -1.000000 \n",
      "3.000000 -4.000000 \n",
      "channel = 1\n",
      "-6.000000 7.000000 \n",
      "9.000000 -2.000000 \n",
      "after descent...\n",
      "channel size = 2 size = 2 * columnSize 2\n",
      "channel = 0\n",
      "0.999901 -0.999980 \n",
      "2.999833 -4.000009 \n",
      "channel = 1\n",
      "-5.999983 6.999709 \n",
      "8.999814 -2.000032 \n"
     ]
    }
   ],
   "source": [
    "kernel = Matrix(\"Random\",2,2,2)\n",
    "kernel.feature = [[[1, -1], [3, -4]], [[-6, 7], [9, -2]]]\n",
    "gradient = Matrix(\"Random\",2,2,2)\n",
    "print(\"before descent...\")\n",
    "Matrix_Print(kernel)\n",
    "Back_Descent(kernel,gradient)\n",
    "print(\"after descent...\")\n",
    "Matrix_Print(kernel)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_BatchNorm_Descent(struct BN *weight,float *gradientBeta,float *gradientGamma)</br>\n",
    "\n",
    "<br/>\n",
    "变量： (struct BN*)weight = 待梯度递减的批标准化参数<br/>\n",
    "       (float **)gradientBeta = 各通道的Beta梯度<br/>\n",
    "       (float **)gradientGamma = 各通道的Gamma梯度<br/>\n",
    "功能：对该批标准化参数进行梯度递减即更新批标准化参数<br/>\n",
    "\n",
    "时间复杂度O(N) ： i <br/>\n",
    "\n",
    "调用例子 ： Back_BatchNorm_Descent(coeff,gradientBata，gradientGamma)<br/>\n",
    "如上所示根据gradientBeta核gradientGamma中每个通道的梯度，对coeff逐个进行递减。<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_BatchNorm_Descent( weight:BN , gradientBeta:list[float],gradientGamma:list[float]):\n",
    "    learningRate = 0.001 # 学习率建议宏定义\n",
    "    decayWeight = 0.1 # 衰变权重建议宏定义\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    decay = (1-learningRate*decayWeight/BATCH) #为衰变函数直接计算并存储为一个变量减少计算时间\n",
    "    learning = (learningRate/BATCH)  #为递减函数直接计算并存储为一个变量减少计算时间\n",
    "    channel= weight.channelSize\n",
    "    for i in range(channel):\n",
    "        weight.beta[i] = weight.beta[i]*decay - learning*gradientBeta[i]\n",
    "        weight.gamma[i] = weight.gamma[i]*decay - learning*gradientGamma[i]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before descent...\n",
      "beta: [1, 2, 3, 4, 5]\n",
      "gamma: [9, 7, 6.8, 7.2, 3.6]\n",
      "after descent...\n",
      "beta: [0.99996, 1.99992, 2.99988, 4.0, 5.0]\n",
      "gamma: [8.9998, 6.99982, 6.799803999999999, 7.199935999999999, 3.600028]\n"
     ]
    }
   ],
   "source": [
    "weight = BN(5)\n",
    "weight.beta = [1,2,3,4,5]\n",
    "weight.gamma = [9,7,6.8,7.2,3.6]\n",
    "gradientBeta = [0.1,0.2,0.3,-0.4,-0.5]\n",
    "gradientGamma = [0.1,0.2,0.3,-0.4,-0.5]\n",
    "print(\"before descent...\")\n",
    "print(\"beta:\",weight.beta)\n",
    "print(\"gamma:\",weight.gamma)\n",
    "Back_BatchNorm_Descent(weight,gradientBeta,gradientGamma)\n",
    "print(\"after descent...\")\n",
    "print(\"beta:\",weight.beta)\n",
    "print(\"gamma:\",weight.gamma)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_CostFunction(struct Matrix *input,int testCaseLabel,struct Matrix *output)</br>\n",
    "\n",
    "<br/>\n",
    "变量： (struct Matrix*)input  = softmax激活后的特征矩阵<br/>\n",
    "       (int)testCaseLabel = 该图片的实际答案<br/>\n",
    "       (struct Matrix*)output  = 全连接层的梯度<br/>\n",
    "功能：该层梯度为input且实际答案所在的行的值需要-1，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： j <br/>\n",
    "\n",
    "调用例子 ： Back_CostFunction(feature,2，gradient)<br/>\n",
    "如上所示根据将input的值赋值给gradient并在第二行进行-1的操作<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_CostFunction(input:Matrix , testCaseLabel:int , output:Matrix):\n",
    "    #input实际尺寸为 (1 X predictSize X 1)\n",
    "    #output实际尺寸为 (1 X predictSize X 1)\n",
    "    row = input.rowSize ; \n",
    "    for j in range(row):\n",
    "        output.feature[0][j][0] = input.feature[0][j][0]\n",
    "    output.feature[0][testCaseLabel][0] = output.feature[0][testCaseLabel][0] - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the feature value...\n",
      "channel size = 1 size = 5 * columnSize 1\n",
      "channel = 0\n",
      "0.337729 \n",
      "0.819477 \n",
      "0.616194 \n",
      "0.097791 \n",
      "0.940982 \n",
      "the gradient value...\n",
      "channel size = 1 size = 5 * columnSize 1\n",
      "channel = 0\n",
      "0.337729 \n",
      "0.819477 \n",
      "0.616194 \n",
      "-0.902209 \n",
      "0.940982 \n"
     ]
    }
   ],
   "source": [
    "feature = Matrix(\"Random\",1,5,1)\n",
    "ans = Matrix(\"Zero\",1,5,1)\n",
    "testCaseLabel = 3\n",
    "print(\"the feature value...\")\n",
    "Matrix_Print(feature)\n",
    "Back_CostFunction(feature,testCaseLabel,ans)\n",
    "print(\"the gradient value...\")\n",
    "Matrix_Print(ans)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_CostFunction(struct Matrix **input,int *testCaseLabel , int index , struct Matrix **output)</br>\n",
    "\n",
    "<br/>\n",
    "变量： (struct Matrix**)input  = 整批softmax激活后的特征矩阵<br/>\n",
    "       (int *)testCaseLabel = 整批的实际答案<br/>\n",
    "       (int ) index = 实际答案的索引位置<br/>\n",
    "       (struct Matrix**)output  = 整批全连接层的梯度<br/>\n",
    "功能：整批的全连接层梯度计算<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * j <br/>\n",
    "\n",
    "调用例子 ： Gradient_CostFunction(feature,testCaseLabel,2，gradient)<br/>\n",
    "如上所示根据将整批input的梯度计算后存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_CostFunction(input:list[Matrix] , testCaseLabel:list[int] ,index:int, output:list[Matrix]):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    start = index*BATCH #索引位置\n",
    "    for num in range(BATCH):\n",
    "        Back_CostFunction(input[num],testCaseLabel[start],output[num])\n",
    "        start = start + 1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_FullConnect_Bias(struct Matrix *lastTermGradeint,struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)gradeint  = 偏差值的梯度<br/>\n",
    "功能：全连接层中偏差值的梯度为上一层的梯度，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： j <br/>\n",
    "\n",
    "调用例子 ： Back_FullConnect_Bias(lastTermGradeint,gradient)<br/>\n",
    "如上所示根据将上一层的梯度存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_FullConnect_Bias(lastTermGradient:Matrix,gradient:Matrix):\n",
    "    Matrix_Sum(gradient,lastTermGradient,gradient)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient_FullConnect_Bias(struct Matrix **lastTermGradeint,struct Matrix *bias,struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)bias  = 偏差值的矩阵<br/>\n",
    "       (struct Matrix *)gradeint  = 偏差值的梯度<br/>\n",
    "功能：先计算整批的偏差值梯度再对其进行梯度递减<br/>\n",
    "\n",
    "时间复杂度O(N) ：2 * num * j <br/>\n",
    "\n",
    "调用例子 ： Gradient_FullConnect_Bias(lastTermGradeint,bias,gradient)<br/>\n",
    "如上所示根据将上一层整批的梯度存储到gradient中，并对bias进行梯度递减<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_FullConnect_Bias(lastTermGradient:list[Matrix],bias:Matrix,gradient:Matrix):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    Matrix_ToZero(gradient)\n",
    "    for num in range(BATCH):\n",
    "        Back_FullConnect_Bias(lastTermGradient[num],gradient)\n",
    "    Back_Descent(bias,gradient)\n",
    "        "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_FullConnect_Weight(struct Matrix *lastTermGradient ,struct Matrix *variable, struct Matrix *gradient)\n",
    "</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradient  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)variable  = 全连接层的特征矩阵<br/>\n",
    "       (struct Matrix *)gradient  = 全连接层权重的梯度<br/>\n",
    "功能：全连接层中权重的梯度即为上层梯度与全连接层特征矩阵的转置的叉乘，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： j * k<br/>\n",
    "\n",
    "调用例子 ： Back_FullConnect_Weight(lastTermGradient,featureFC,gradient)<br/>\n",
    "如上所示根据将上一层的梯度与featureFC的转置进行叉乘并将累加到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_FullConnect_Weight(lastTermGradient:Matrix,variable:Matrix,gradient:Matrix):\n",
    "    #lastTermGradient = (1 X predictSize X 1)\n",
    "    #variable =  (1 X rowSize X 1)\n",
    "    #gradient = (1 X predictSize X rowSize)\n",
    "    row = lastTermGradient.rowSize\n",
    "    column = variable.rowSize\n",
    "    for j in range(row):\n",
    "        for k in range(column):\n",
    "            gradient.feature[0][j][k] += (lastTermGradient.feature[0][j][0]*variable.feature[0][k][0])\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_FullConnect_Weight(struct Matrix **lastTermGradient ,struct Matrix **variable,struct Matrix *weight, struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradient  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批全连接层的特征矩阵<br/>\n",
    "       (struct Matrix *)gradient  = 全连接层权重的梯度<br/>\n",
    "功能：先计算整批的全连接层权重梯度再对其进行梯度递减<br/>\n",
    "\n",
    "时间复杂度O(N) ：2 * num * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_FullConnect_Weight(lastTermGradient,variable,weight,gradient)<br/>\n",
    "如上所示根据将上一层的梯度与featureFC的转置进行叉乘并将其加到gradient中,最后对weight进行梯度递减<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_FullConnect_Weight(lastTermGradient:list[Matrix],variable:list[Matrix],weight:Matrix,gradient:Matrix):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    Matrix_ToZero(gradient)\n",
    "    for num in range(BATCH):\n",
    "        Back_FullConnect_Weight(lastTermGradient[num],variable[num],gradient)\n",
    "    Back_Descent(weight,gradient)\n",
    "\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_FullConnect_Variable(struct Matrix *lastTermGradient ,struct Matrix *weight, struct Matrix *gradient)\n",
    "</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradient  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)variable  = 全连接层的权重<br/>\n",
    "       (struct Matrix *)gradient  = 全连接层的梯度<br/>\n",
    "功能：全连接层的梯度即为全连接层权重的转置上层梯度与上一层梯度的叉乘，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： j * k<br/>\n",
    "\n",
    "调用例子 ： Back_FullConnect_Variable(lastTermGradient,weightFC,gradient)<br/>\n",
    "如上所示根据将weightFC的转置与上一层的梯度进行叉乘并将存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_FullConnect_Variable(lastTermGradient:Matrix,weight:Matrix,gradient:Matrix): \n",
    "    #lastTermGradient = (1 X predictSize X 1)\n",
    "    #gradient =  (1 X rowSize X 1)\n",
    "    #weight = (1 X predictSize X rowSize)\n",
    "    row = lastTermGradient.rowSize   #predictsize\n",
    "    column = weight.columnSize   # rowsize\n",
    "    for j in range(row):\n",
    "        for k in range(column):\n",
    "            gradient.feature[0][k][0] += (lastTermGradient.feature[0][j][0] * weight.feature[0][j][k])\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_FullConnect_Variable(struct Matrix **lastTermGradient ,struct Matrix *weight, struct Matrix **gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradient  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix *)weight  = 全连接层的权重<br/>\n",
    "       (struct Matrix **)gradient  = 整批全连接层的梯度<br/>\n",
    "功能：计算整批的全连接层梯度<br/>\n",
    "\n",
    "时间复杂度O(N) ：num * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_FullConnect_Variable(lastTermGradient,weight,gradient)<br/>\n",
    "如上所示根据将weightFC的转置与整批上一层的梯度进行叉乘并将存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_FullConnect_Variable(lastTermGradient:list[Matrix],weight,gradient:list[Matrix]):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "            Matrix_ToZero(gradient[num])\n",
    "            Back_FullConnect_Variable(lastTermGradient[num],weight,gradient[num])\n",
    "            "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_FullConnect(struct Matrix **lastTermGradient ,struct Matrix *bias ,struct Matrix *gradientBias ,struct Matrix *weight , struct Matrix **variable , struct Matrix *gradientWeight\n",
    ",struct Matrix **gradient) </br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradient  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix *)bias  = 偏差值的矩阵<br/>\n",
    "       (struct Matrix *)gradeintBias  = 偏差值的梯度<br/>\n",
    "       (struct Matrix *)weight  = 全连接层的权重<br/>\n",
    "       (struct Matrix **)variable  = 整批全连接层的特征矩阵<br/>\n",
    "       (struct Matrix *)gradientWeight  = 全连接层权重的梯度<br/>\n",
    "       (struct Matrix **)gradient  = 整批全连接层的梯度<br/>\n",
    "功能：计算整批的全连接层梯度,以及完成偏差矩阵以及权重矩阵的梯度递减<br/>\n",
    "时间复杂度O(N) ：3 * num * j * k + 2 * num * j <br/>\n",
    "       \n",
    "\n",
    "调用例子 ： Gradient_FullConnect(lastTermGradient,bias,gradientBias,weight,variable,gradientWeight,gradient)<br/>\n",
    "如上所示根据将计算gradient，并使用gradientBias及gradientWeight对bias及weight进行梯度递减<br/>\n",
    "\n",
    "算法演示\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_FullConnectGradient_FullConnect(lastTermGradient:list[Matrix],bias:Matrix,gradientBias:Matrix,weight:Matrix,variable:list[Matrix],gradientWeight:Matrix,gradient:list[Matrix]):\n",
    "    Gradient_FullConnect_Variable(lastTermGradient,weight,gradient)\n",
    "    #先计算下一层的梯度，因为权重的更新会影响梯度的计算，反之不会\n",
    "    Gradient_FullConnect_Bias(lastTermGradient,bias,gradientBias)\n",
    "    Gradient_FullConnect_Weight(lastTermGradient,variable,weight,gradientWeight)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_GlobalAverage(struct Matrix *lastTermGradient ,struct Matrix *variable, struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)variable  = 全局平均池化前的特征矩阵<br/>\n",
    "       (struct Matrix *)gradeint  = 全局平均池化层的梯度<br/>\n",
    "功能：全局平均池化层的梯度为上一层的梯度分别除于（池化前矩阵的大小），详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： i * j * k <br/>\n",
    "\n",
    "调用例子 ： Back_GlobalAverage(lastTermGradeint,variable,gradient)<br/>\n",
    "如上所示根据将上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_GlobalAverage(lastTermGradient:Matrix,variable:Matrix,gradient:Matrix):\n",
    "    channel = lastTermGradient.rowSize\n",
    "    row = variable.rowSize\n",
    "    column = variable.columnSize\n",
    "    size = 1.0/row*column\n",
    "    for i in range(channel):\n",
    "        for j in range(row):\n",
    "            for k in range(column):\n",
    "                gradient.feature[i][j][k] = lastTermGradient.feature[0][i][0] * size"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_GlobalAverage(struct Matrix **lastTermGradient ,struct Matrix **variable, struct Matrix **gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批全局平均池化前的特征矩阵<br/>\n",
    "       (struct Matrix **)gradeint  = 整批全局平均池化层的梯度<br/>\n",
    "功能：整批的计算全局平均池化层的梯度<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_GlobalAverage(lastTermGradeint,variable,gradient)<br/>\n",
    "如上所示根据将整批上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_GlobalAverage(lastTermGradient:list[Matrix],variable:list[Matrix],gradient:list[Matrix]):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Back_GlobalAverage(lastTermGradient[num],variable[num],gradient[num])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_MaxPooling(struct Matrix *lastTermGradient ,struct Matrix *variable ,struct Matrix *output,int poolingSize,int paddingAmt,int stride,struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)variable  = 最大池化前的特征矩阵<br/>\n",
    "       (struct Matrix *)output  = 最大池化后的特征矩阵<br/>\n",
    "       (int) poolingSize = 池化大小 <br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "       (struct Matrix *)gradeint  = 最大池化层的梯度<br/>\n",
    "功能：最大池化层的梯度在最大值的格中的值为上一层的梯度其余为0，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： i * j * k * y * z<br/>\n",
    "\n",
    "调用例子 ： Back_MaxPooling(lastTermGradient ,variable ,output,poolingSize,paddingAmt, stride,gradient)<br/>\n",
    "如上所示根据将上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_MaxPooling(lastTermGradient:Matrix ,variable:Matrix ,output:Matrix,poolingSize:int,paddingAmt:int, stride:int,gradient:Matrix):\n",
    "    channel = variable.channelSize\n",
    "    size = lastTermGradient.rowSize\n",
    "    for i in range(channel):\n",
    "        for j in range(size):\n",
    "            for k in range(size):\n",
    "                maximun =  output.feature[i][j][k]\n",
    "                for y in range(poolingSize):\n",
    "                    for z in range(poolingSize):\n",
    "                        rowLoc = j*stride+y-paddingAmt\n",
    "                        columnLoc = k*stride+z-paddingAmt\n",
    "                        if ((rowLoc < 0 or rowLoc >= variable.rowSize ) or (columnLoc < 0 or columnLoc >= variable.columnSize)):\n",
    "                            continue #填充区域跳过\n",
    "                        if(variable.feature[i][rowLoc][columnLoc] == maximun):\n",
    "                            gradient.feature[i][rowLoc][columnLoc] = lastTermGradient.feature[i][j][k] #最大值区域\n",
    "                        else:\n",
    "                            gradient.feature[i][rowLoc][columnLoc] = 0 #非最大值区域"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_MaxPooling(struct Matrix **lastTermGradient ,struct Matrix **variable ,struct Matrix **output,int poolingSize,int paddingAmt,int stride,struct Matrix **gradient);</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批最大池化前的特征矩阵<br/>\n",
    "       (struct Matrix **)output  = 整批最大池化后的特征矩阵<br/>\n",
    "       (int) poolingSize = 池化大小 <br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "       (struct Matrix *)gradeint  = 整批最大池化层的梯度<br/>\n",
    "功能：整批的进行最大池化层的梯度计算<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * i * j * k * y * z<br/>\n",
    "\n",
    "调用例子 ： BGradient_MaxPooling(lastTermGradient ,variable ,output,poolingSize,paddingAmt, stride,gradient)<br/>\n",
    "如上所示根据将整批上一层的梯度进行计算并存储到gradient中<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_MaxPooling(lastTermGradient:list[Matrix] ,variable:list[Matrix] ,output:list[Matrix],poolingSize:int,paddingAmt:int, stride:int,gradient:list[Matrix]):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Back_MaxPooling(lastTermGradient[num],variable[num],output[num],poolingSize,paddingAmt,stride,gradient[num])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_ReLU(struct Matrix *lastTermGradient , struct Matrix *variable, struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)variable  = 激活前的特征矩阵<br/>\n",
    "       (struct Matrix *)gradeint  = 激活层的梯度<br/>\n",
    "功能：激活层的梯度为variable值大于0为1其余为0，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： i * j * k <br/>\n",
    "\n",
    "调用例子 ： Back_ReLU(lastTermGradient ,variable ,gradient)<br/>\n",
    "如上所示根据将上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_ReLU(lastTermGradient:Matrix ,variable:Matrix ,gradient:Matrix):\n",
    "    for i in range(variable.channelSize):\n",
    "        for j in range(variable.rowSize):\n",
    "            for k in range(variable.columnSize):\n",
    "                if(variable.feature[i][j][k] > 0):\n",
    "                    gradient.feature[i][j][k] = lastTermGradient.feature[i][j][k] \n",
    "                else:\n",
    "                    gradient.feature[i][j][k] = 0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_ReLU(struct Matrix **lastTermGradient , struct Matrix **variable, struct Matrix **gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批激活前的特征矩阵<br/>\n",
    "       (struct Matrix **)gradeint  = 整批激活层的梯度<br/>\n",
    "功能：整批进行ReLU激活层梯度的计算<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_ReLU(lastTermGradient ,variable ,gradient)<br/>\n",
    "如上所示根据将整批上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_ReLU(lastTermGradient:list[Matrix] ,variable:list[Matrix] ,gradient:list[Matrix]):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Back_ReLU(lastTermGradient[num],variable[num],gradient[num])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_BatchNorm_Variable(struct Matrix *lastTermGradient , struct Matrix *variable,struct BN *coeff,struct Matrix *gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix *)variable  = 批标准化前的特征矩阵<br/>\n",
    "       (struct BN *)coeff         = 该层批标准化的参数\n",
    "       (struct Matrix *)gradeint  = 批标准化层的梯度<br/>\n",
    "功能：计算批标准化层的梯度，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： 2 * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Back_BatchNorm_Variable(lastTermGradient ,variable, coeff ,gradient)<br/>\n",
    "如上所示根据将上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_BatchNorm_Variable(lastTermGradient:Matrix ,variable:Matrix,coeff:BN ,gradient:Matrix):\n",
    "    EPSILON = 0.1 #建议宏定义\n",
    "    #size of gradient = size of last term gradient\n",
    "    #variable = the value before batchNorm\n",
    "    channel = lastTermGradient.channelSize\n",
    "    row = lastTermGradient.rowSize\n",
    "    column = lastTermGradient.columnSize\n",
    "    size = 1/(row*column) # 1/m\n",
    "    for i in range(channel):\n",
    "        gradient_Mean  = 0  #dl/dmuse =  -(summation(dl/dxhat))/para1 \n",
    "        gradient_Variance = 0  #dl/dsigma = summation((dl/dxhat)*(mean - variable)/para2)\n",
    "        gamma = coeff.gamma[i]\n",
    "        mean = coeff.mean[i]\n",
    "        para1 = 1/((coeff.variance[i] + EPSILON)**0.5) #1/(sigma^2+epsilon)^0.5\n",
    "        para2 = 1/(2*(para1**3)) #1/2(sigma^2+epsilon)^1.5\n",
    "        for j in range (row):\n",
    "            for k in range (column):\n",
    "                temp = lastTermGradient.feature[i][j][k]*gamma\n",
    "                gradient.feature[i][j][k] = temp \n",
    "                gradient_Mean -= temp\n",
    "                gradient_Variance += (temp*(mean - variable.feature[i][j][k]))\n",
    "        gradient_Mean = gradient_Mean/para1\n",
    "        gradient_Variance =gradient_Variance/para2\n",
    "        for j in range (row):\n",
    "            for k in range (column):\n",
    "                gradient.feature[i][j][k] = gradient_Mean * size + gradient_Variance * 2 * size * (variable.feature[i][j][k]) + gradient.feature[i][j][k] * para1 \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_BatchNorm_Variable(struct Matrix **lastTermGradient , struct Matrix **variable,struct BN *coeff,struct Matrix **gradient)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批批标准化前的特征矩阵<br/>\n",
    "       (struct BN *)coeff          = 该层批标准化的参数\n",
    "       (struct Matrix **)gradeint  = 整批批标准化层的梯度<br/>\n",
    "功能：整批计算批标准化层的梯度，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * 2 * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_BatchNorm_Variable(lastTermGradient ,variable, coeff ,gradient)<br/>\n",
    "如上所示根据将整批上一层的梯度进行计算并存储到gradient<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_BatchNorm_Variable(lastTermGradient:list[Matrix] ,variable:list[Matrix],coeff:BN ,gradient:list[Matrix]):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Back_BatchNorm_Variable(lastTermGradient[num],variable[num],coeff,gradient[num])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_BatchNorm_Weight(struct Matrix *lastTermGradient ,struct BN *coeff, struct Matrix *variable,float *gradientBeta,float *gradientGamma)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct BN *)coeff         = 该层批标准化的参数\n",
    "       (struct Matrix *)variable  = 批标准化后的特征矩阵<br/>\n",
    "       (float *)gradeintBeta  = 批标准化层Beta的梯度<br/>\n",
    "       (float *)gradeintGamma  = 批标准化层Gamma的梯度<br/>\n",
    "功能：计算批标准化层的权重Beta和Gamma的梯度，详情计算见梯度推导算法<br/>\n",
    "\n",
    "时间复杂度O(N) ： i * j * k <br/>\n",
    "\n",
    "调用例子 ： Back_BatchNorm_Weight(lastTermGradient , coeff , variable ,gradientBeta ,gradientGamma)<br/>\n",
    "如上所示根据将上一层的梯度进行计算权重的梯度并将结果累加gradientBeta和gradientGamma中<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_BatchNorm_Weight(lastTermGradient:Matrix , coeff:BN , variable:Matrix ,gradientBeta:list[float] ,gradientGamma:list[float]):\n",
    "    channel = lastTermGradient.channelSize\n",
    "    row = lastTermGradient.rowSize\n",
    "    column = lastTermGradient.columnSize\n",
    "    for i in range(channel):\n",
    "        beta = coeff.beta[i]\n",
    "        gamma = coeff.gamma[i]\n",
    "        for j in range (row):\n",
    "            for k in range (column):\n",
    "                gradientBeta[i] += lastTermGradient.feature[i][j][k]\n",
    "                #variable[i][j]][k] = Gamma[i] * variableHat[i][j][k] + Beta[i]\n",
    "                variable_Hat = (variable.feature[i][j][k] - beta)/gamma \n",
    "                gradientGamma[i] += (lastTermGradient.feature[i][j][k] * variable_Hat)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_BatchNorm_Weight(struct Matrix **lastTermGradient ,struct BN **coeff, struct Matrix **variable,float **gradientBeta,float **gradientGamma)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct BN *)coeff         = 该层批标准化的参数\n",
    "       (struct Matrix **)variable  = 整批批标准化后的特征矩阵<br/>\n",
    "       (float *)gradeintBeta  = 批标准化层Beta的梯度<br/>\n",
    "       (float *)gradeintGamma  = 批标准化层Gamma的梯度<br/>\n",
    "功能：累加批标准化层的权重Beta和Gamma的梯度，最后对beta和gamma进行梯度递减<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_BatchNorm_Weight(lastTermGradient , coeff , variable ,gradientBeta ,gradientGamma)<br/>\n",
    "如上所示根据将上一层的梯度进行计算权重的梯度并将结果累加gradientBeta和gradientGamma中，并进行梯度递减<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_BatchNorm_Weight(lastTermGradient:list[Matrix] , coeff:BN , variable:list[Matrix] ,gradientBeta:list[float] ,gradientGamma:list[float]):\n",
    "    size = coeff.channelSize\n",
    "    Back_ToZero(gradientBeta,size)\n",
    "    Back_ToZero(gradientGamma,size)\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Back_BatchNorm_Weight(lastTermGradient[num],coeff,variable[num],gradientBeta,gradientGamma)\n",
    "    Back_BatchNorm_Descent(coeff,gradientBeta,gradientGamma)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_BatchNorm(struct Matrix **lastTermGradient , struct Matrix **inputVariable,struct BN *coeff , struct Matrix **gradient ,struct Matrix **outputVariable,float *gradientBeta,float *gradientGamma )</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)inputVariable  = 整批批标准化前的特征矩阵<br/>\n",
    "       (struct BN *)coeff         = 该层批标准化的参数\n",
    "       (struct Matrix **) gradient  = 整批批标准化的梯度<br/>\n",
    "       (struct Matrix **) outputVariable  = 整批批标准化后的特征矩阵<br/>\n",
    "       (float *)gradeintBeta  = 批标准化层Beta的梯度<br/>\n",
    "       (float *)gradeintGamma  = 批标准化层Gamma的梯度<br/>\n",
    "功能：包含了Gradient_BatchNorm_Weight和Gradient_BatchNorm_Variable所有操作<br/>\n",
    "\n",
    "时间复杂度O(N) ： 3 * num * i * j * k <br/>\n",
    "\n",
    "调用例子 ： Gradient_BatchNorm(lastTermGradient , inputVariable ,  coeff , gradient, outputVariable ,gradientBeta ,gradientGamma)<br/>\n",
    "包含了Gradient_BatchNorm_Weight和Gradient_BatchNorm_Variable所有操作\n",
    "<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_BatchNorm(lastTermGradient:list[Matrix] , inputVariable:list[Matrix] ,  coeff:BN , gradient:list[Matrix], outputVariable:list[Matrix] ,gradientBeta:list[float] ,gradientGamma:list[float]):\n",
    "    Gradient_BatchNorm_Variable(lastTermGradient,inputVariable,coeff,gradient)\n",
    "    Gradient_BatchNorm_Weight(lastTermGradient,coeff,outputVariable,gradientBeta,gradientGamma)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_Convolution_Variable(struct Matrix *lastTermGradient ,struct Matrix **kernel ,struct Matrix *gradient,int stride, int paddingAmt)\n",
    "</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix **)kernel  = 该层卷积核<br/>\n",
    "       (struct Matrix *)gradient  = 该层的梯度<br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "功能：计算该特征向量卷积层的梯度，详细请见梯度算法分析文档<br/>\n",
    "\n",
    "时间复杂度O(N) ：  j * k * x * i * y * z<br/>\n",
    "\n",
    "调用例子 ： Back_Convolution_Variable(lastTermGradient ,kernel ,gradient ,paddingAmt, stride)<br/>\n",
    "利用上层梯度与卷积核计算该层的梯度<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_Convolution_Variable(lastTermGradient:Matrix ,kernel:list[Matrix] ,gradient:Matrix ,paddingAmt:int, stride:int):\n",
    "    channel = gradient.channelSize\n",
    "    size = lastTermGradient.rowSize\n",
    "    kernelSize = kernel[0].rowSize\n",
    "    kernelAmt = lastTermGradient.channelSize\n",
    "    for x in range (kernelAmt):\n",
    "        for j in range (size):\n",
    "            for k in range (size):\n",
    "                temp = 0\n",
    "                for i in range (channel):\n",
    "                    for y in range (kernelSize):\n",
    "                        for z in range (kernelSize):\n",
    "                            rowLoc = j * stride + y - paddingAmt #用于确定其在input内的行索引\n",
    "                            columnLoc = k * stride + z - paddingAmt #用于确定其在input内的例索引\n",
    "                            #用于判断是否为padding区域若为padding区域不需要计算\n",
    "                            if ((rowLoc < 0 or rowLoc >= input.rowSize ) or (columnLoc < 0 or columnLoc >= input.columnSize )):\n",
    "                                continue\n",
    "                            #若位于该位置进行点乘并叠加到暂时变量\n",
    "                            #dL/dX = dL/dy * dy/dx (only the kernel which sliding and fix the input will affect the gradient )\n",
    "                            gradient.feature[i][rowLoc][columnLoc] += (lastTermGradient.feature[x][j][k] * kernel[x].feature[i][y][z])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_Convolution_Variable(struct Matrix **lastTermGradient ,struct Matrix **kernel ,struct Matrix **gradient,int stride, int paddingAmt)\n",
    "</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批计算上一层的梯度<br/>\n",
    "       (struct Matrix **)kernel  = 整批计算该层卷积核<br/>\n",
    "       (struct Matrix **)gradient  = 整批计算该层的梯度<br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "功能：整批计算计算该特征向量卷积层的梯度，详细请见梯度算法分析文档<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * j * k * x * i * y * z<br/>\n",
    "\n",
    "调用例子 ： Gradient_Convolution_Variable(lastTermGradient ,kernel ,gradient ,paddingAmt, stride)<br/>\n",
    "利用整批上层梯度与卷积核计算该层的梯度<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_Convolution_Variable(lastTermGradient:list[Matrix] ,kernel:list[Matrix] ,gradient:list[Matrix] ,paddingAmt:int, stride:int):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Matrix_ToZero(gradient[num])\n",
    "        Back_Convolution_Variable(lastTermGradient[num] ,kernel ,gradient[num] ,paddingAmt, stride)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Back_Convolution_Kernel(struct Matrix *lastTermGradient ,struct Matrix **gradient,struct Matrix *variable ,int stride, int paddingAmt)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix *)lastTermGradeint  = 上一层的梯度<br/>\n",
    "       (struct Matrix **)gradient  = 该层卷积核的梯度<br/>\n",
    "       (struct Matrix *)variable  = 卷积前的特征矩阵<br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "功能：计算卷积层卷积核的梯度，详细请见梯度算法分析文档<br/>\n",
    "\n",
    "时间复杂度O(N) ：   j * k * x * i * y * z<br/>\n",
    "\n",
    "调用例子 ： Back_Convolution_Variable(lastTermGradient ,gradient ，variable,paddingAmt, stride)<br/>\n",
    "计算该上层梯度与该特征向量的卷积核梯度并累加到gradient中<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Back_Convolution_Kernel(lastTermGradient:Matrix ,gradient:list[Matrix],variable:Matrix ,paddingAmt:int, stride:int):\n",
    "    channel = variable.channelSize\n",
    "    size = lastTermGradient.rowSize\n",
    "    kernelSize = gradient[0].rowSize\n",
    "    kernelAmt = lastTermGradient.channelSize\n",
    "    for x in range (kernelAmt):\n",
    "        for j in range (size):\n",
    "            for k in range (size):\n",
    "                for i in range (channel):\n",
    "                    for y in range (kernelSize):\n",
    "                        for z in range (kernelSize):\n",
    "                            rowLoc = j * stride + y - paddingAmt #用于确定其在input内的行索引\n",
    "                            columnLoc = k * stride + z - paddingAmt #用于确定其在input内的例索引\n",
    "                            #用于判断是否为padding区域若为padding区域不需要计算\n",
    "                            if ((rowLoc < 0 or rowLoc >= input.rowSize ) or (columnLoc < 0 or columnLoc >= input.columnSize )):\n",
    "                                continue\n",
    "                            #若位于该位置进行点乘并叠加到暂时变量\n",
    "                            #dL/dTheta = dL/dy * dy/dTheta (only the kernel which sliding and fix the input will affect the gradient )\n",
    "                            gradient[x].feature[i][y][z] += (lastTermGradient.feature[x][j][k] * \n",
    "                                                                variable.feature[i][rowLoc][columnLoc])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_Convolution_Kernel(struct Matrix **lastTermGradient ,struct Matrix **gradient,struct Matrix **variable ,int stride, int paddingAmt)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)gradient  = 该层卷积核的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批卷积前的特征矩阵<br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "功能：计算卷积层卷积核的梯度，详细请见梯度算法分析文档<br/>\n",
    "\n",
    "时间复杂度O(N) ： num * j * k * x * i * y * z<br/>\n",
    "\n",
    "调用例子 ： Gradient_Convolution_kernel(lastTermGradient ,gradient ，variable,paddingAmt, stride)<br/>\n",
    "计算该上层梯度与该特征向量的卷积核梯度<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_Convolution_Kernel(lastTermGradient:list[Matrix] ,gradient:list[Matrix],variable:list[Matrix] ,paddingAmt:int, stride:int):\n",
    "    BATCH = 5  #批次大小建议宏定义\n",
    "    for num in range(BATCH):\n",
    "        Back_Convolution_Kernel(lastTermGradient[num],gradient,variable[num],stride,paddingAmt)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "void Gradient_Convolution(struct Matrix **lastTermGradient ,struct Matrix **kernel ,struct Matrix **gradient, struct Matrix **variable ,struct Matrix **gradientKernel, int stride, int paddingAmt)</br>\n",
    "<br/>\n",
    "变量： (struct Matrix **)lastTermGradeint  = 整批上一层的梯度<br/>\n",
    "       (struct Matrix **)kernel  = 整批计算该层卷积核<br/>\n",
    "       (struct Matrix **)gradient  = 该层卷积核的梯度<br/>\n",
    "       (struct Matrix **)variable  = 整批卷积前的特征矩阵<br/>\n",
    "       (struct Matrix **)gradientKernel  = 该层卷积核的梯度<br/>\n",
    "       (int) stride = 步长<br/>\n",
    "       (int) paddingAmt = 填充大小 (所谓填充即在input中添加全为0的特征)<br/>\n",
    "功能：计算卷积层的梯度和卷积核的梯度并对卷积核进行梯度递减<br/>\n",
    "\n",
    "时间复杂度O(N) ：2 * num * j * k * x * i * y * z<br/>\n",
    "\n",
    "调用例子 ： Gradient_Convolution(lastTermGradient ,kernel,gradient ,variable, gradientKernel, stride , paddingAmt)<br/>\n",
    "计算卷积层的梯度和卷积核的梯度并对卷积核进行梯度递减<br/>\n",
    "\n",
    "算法演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Gradient_Convolution(lastTermGradient:list[Matrix] ,kernel:list[Matrix],gradient:list[Matrix] ,variable:list[Matrix], gradientKernel:list[Matrix], stride:int, paddingAmt:int):\n",
    "    kernelAmt = lastTermGradient[0].channelSize\n",
    "    Gradient_Convolution_Variable(lastTermGradient,kernel,gradient,stride,paddingAmt)\n",
    "    for x in range(kernelAmt):\n",
    "        Matrix_ToZero(gradientKernel[x])\n",
    "    Gradient_Convolution_Kernel(lastTermGradient,gradientKernel,variable,stride,paddingAmt)\n",
    "    for x in range(kernelAmt):\n",
    "        Back_Descent(kernel[x],gradientKernel[x])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "df95319d8ce4e1d89f5365ae10992bc1f65da593082b1d264e8f529830ec2f02"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}