NBER Herfindahl indices of patent originality and generality (M. Wilder).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> National Bureau for Economic Research (NBER) inverted Herfindahl indices of patent 'originality' and 'generality'</h3>\n",
    "\n",
    "<p> Matt Wilder, University of Toronto <br>\n",
    "Please address questions and comments to <a href=\"mailto:matt.wilder@utoronto.ca\">matt.wilder@utoronto.ca</a>. </p>\n",
    "\n",
    "<p>Updated 14 April 2024</p>\n",
    "\n",
    "<p>These functions calculate inverted Herfindahl indices of patent 'originality' and 'generality' as introduced by Trajtenberg et al (1997) 'University versus corporate patents: A window on the basicness of invention' <i>Economics of Innovation and New Technology, 5</i>(1): 19-50. These variables are also included in the <a href=\"https://www.nber.org/system/files/working_papers/w8498/w8498.pdf\">NBER Patent Citation Data File</a> by Hall et al (2001).</p>\n",
    "\n",
    "<p>The originality index measures the diversity of a patent's backward citations—citations made by the patent to earlier patents—across different technological fields (i.e., the breadth of a patent's intellectual heritage):</p>\n",
    "\n",
    "$$\n",
    "\\text{Originality}_{i} = 1 - \\sum_{k=1}^{N_{i}} \\left( \\frac{n_{ik}}{N_{i}} \\right)^2\n",
    "$$\n",
    "\n",
    "<p>where \\( i \\) refers to a particular \"focal\" patent, \\( N_i \\) is the total number of backward citations contained in patent \\( i \\), \\( k \\) indexes each unique CPC classification cited by \\( i \\), and \\( n_{ik} \\) is the number of backward citations made by patent \\( i \\) to each CPC classification \\( k \\).</p>\n",
    "\n",
    "<p>The generality index measures the distribution of a patent's forward citations—citations made to the patent by later patents—across different technological fields (i.e., the breadth of a patent's influence):</p>\n",
    "\n",
    "$$\n",
    "\\text{Generality}_{i} = 1 - \\sum_{k=1}^{N_{i}} \\left( \\frac{n_{ik}}{N_{i}} \\right)^2\n",
    "$$\n",
    "\n",
    "<p>where \\( i \\) refers to a particular \"focal\" patent, \\( N_i \\) is the total number of forward citations received by patent \\( i \\), \\( k \\) indexes each unique CPC classification of patents that cite \\( i \\), and \\( n_{ik} \\) is the number of citations received by patent \\( i \\) from each CPC classification \\( k \\).</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first 20 patents in forward citations sample\n",
      "          to   patent patent_date        name to_cpc forward_citations\n",
      "0   10062352  9530381  2016-12-01     Bozarth   G09G               3.0\n",
      "1   10078977  9530381  2016-12-01     Bozarth   G09G               3.0\n",
      "2   10062352  9530381  2016-12-01     Bozarth   H01L               3.0\n",
      "3   10078977  9530381  2016-12-01     Bozarth   G02F               3.0\n",
      "4   10078977  9530381  2016-12-01     Bozarth   G06F               3.0\n",
      "5   10324237  9458989  2016-10-01  Hsu et al.   G02B               3.0\n",
      "6    9927616  9458989  2016-10-01  Hsu et al.   G02B               3.0\n",
      "7    9927616  9458989  2016-10-01  Hsu et al.   F21V               3.0\n",
      "8    9927616  9458989  2016-10-01  Hsu et al.   B82Y               3.0\n",
      "9    9927616  9458989  2016-10-01  Hsu et al.   G02F               3.0\n",
      "10   9927616  9458989  2016-10-01  Hsu et al.   G09F               3.0\n",
      "11   9927616  9458989  2016-10-01  Hsu et al.   Y10S               3.0\n",
      "12   9927616  9458989  2016-10-01  Hsu et al.   G03B               3.0\n",
      "13   9978299  9437131  2016-09-01      Nagara   G02B               2.0\n",
      "14   9978299  9437131  2016-09-01      Nagara   G09G               2.0\n",
      "15   9978299  9437131  2016-09-01      Nagara   G02F               2.0\n",
      "16   9978299  9437131  2016-09-01      Nagara   H01L               2.0\n",
      "17   9978299  9437131  2016-09-01      Nagara   H04N               2.0\n",
      "18  10180741  9373290  2016-06-01         Lee   G06F               3.0\n",
      "19  10180741  9373290  2016-06-01         Lee   G09G               3.0\n",
      "\n",
      "first 20 patents in backward citations sample\n",
      "       from   patent patent_date   name from_cpc backward_citations\n",
      "0   6906762  8941691  2015-01-01  Baron     G09G               11.0\n",
      "1   6369830  8941691  2015-01-01  Baron     G09G               11.0\n",
      "2   8436873  8941691  2015-01-01  Baron     G09G               11.0\n",
      "3   8432411  8941691  2015-01-01  Baron     G09G               11.0\n",
      "4   8416149  8941691  2015-01-01  Baron     G09G               11.0\n",
      "5   7746356  8941691  2015-01-01  Baron     G09G               11.0\n",
      "6   6906762  8941691  2015-01-01  Baron     H04N               11.0\n",
      "7   5745197  8941691  2015-01-01  Baron     H04N               11.0\n",
      "8   6195104  8941691  2015-01-01  Baron     H04N               11.0\n",
      "9   6906762  8941691  2015-01-01  Baron     G02F               11.0\n",
      "10  5745197  8941691  2015-01-01  Baron     G02F               11.0\n",
      "11  8436873  8941691  2015-01-01  Baron     G02F               11.0\n",
      "12  6906762  8941691  2015-01-01  Baron     G02B               11.0\n",
      "13  5745197  8941691  2015-01-01  Baron     G02B               11.0\n",
      "14  5022085  8941691  2015-01-01  Baron     G06T               11.0\n",
      "15  6369830  8941691  2015-01-01  Baron     G06T               11.0\n",
      "16  6195104  8941691  2015-01-01  Baron     A63F               11.0\n",
      "17  6195104  8941691  2015-01-01  Baron     G06F               11.0\n",
      "18  7746356  8941691  2015-01-01  Baron     G06F               11.0\n",
      "19  6507413  8941691  2015-01-01  Baron     B44C               11.0\n"
     ]
    }
   ],
   "source": [
    "'''load in sample data'''\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "fcit_sample_df = pd.read_csv('https://raw.githubusercontent.com/matt-wilder/patent-research/main/forward_citations_by_cpc_sample.csv',sep='\\t', dtype='str')\n",
    "bcit_sample_df = pd.read_csv('https://raw.githubusercontent.com/matt-wilder/patent-research/main/backward_citations_by_cpc_sample.csv',sep='\\t', dtype='str')\n",
    "\n",
    "print('first 20 patents in forward citations sample')\n",
    "print(fcit_sample_df.head(20))\n",
    "\n",
    "print('\\nfirst 20 patents in backward citations sample')\n",
    "print(bcit_sample_df.head(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>from</th>\n",
       "      <th>patent</th>\n",
       "      <th>patent_date</th>\n",
       "      <th>name</th>\n",
       "      <th>from_cpc</th>\n",
       "      <th>backward_citations</th>\n",
       "      <th>originality_hhi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6906762</td>\n",
       "      <td>8941691</td>\n",
       "      <td>2015-01-01</td>\n",
       "      <td>Baron</td>\n",
       "      <td>G09G</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.896552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6369830</td>\n",
       "      <td>8941691</td>\n",
       "      <td>2015-01-01</td>\n",
       "      <td>Baron</td>\n",
       "      <td>G09G</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.896552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8436873</td>\n",
       "      <td>8941691</td>\n",
       "      <td>2015-01-01</td>\n",
       "      <td>Baron</td>\n",
       "      <td>G09G</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.896552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8432411</td>\n",
       "      <td>8941691</td>\n",
       "      <td>2015-01-01</td>\n",
       "      <td>Baron</td>\n",
       "      <td>G09G</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.896552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8416149</td>\n",
       "      <td>8941691</td>\n",
       "      <td>2015-01-01</td>\n",
       "      <td>Baron</td>\n",
       "      <td>G09G</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.896552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>7619585</td>\n",
       "      <td>8890771</td>\n",
       "      <td>2014-11-01</td>\n",
       "      <td>Pance</td>\n",
       "      <td>H04N</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.847222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>7205959</td>\n",
       "      <td>8890771</td>\n",
       "      <td>2014-11-01</td>\n",
       "      <td>Pance</td>\n",
       "      <td>G02F</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.847222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>7205959</td>\n",
       "      <td>8890771</td>\n",
       "      <td>2014-11-01</td>\n",
       "      <td>Pance</td>\n",
       "      <td>G09F</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.847222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>7205959</td>\n",
       "      <td>8890771</td>\n",
       "      <td>2014-11-01</td>\n",
       "      <td>Pance</td>\n",
       "      <td>H04M</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.847222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>6760003</td>\n",
       "      <td>8890771</td>\n",
       "      <td>2014-11-01</td>\n",
       "      <td>Pance</td>\n",
       "      <td>G04G</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.847222</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       from   patent patent_date   name from_cpc backward_citations  \\\n",
       "0   6906762  8941691  2015-01-01  Baron     G09G               11.0   \n",
       "1   6369830  8941691  2015-01-01  Baron     G09G               11.0   \n",
       "2   8436873  8941691  2015-01-01  Baron     G09G               11.0   \n",
       "3   8432411  8941691  2015-01-01  Baron     G09G               11.0   \n",
       "4   8416149  8941691  2015-01-01  Baron     G09G               11.0   \n",
       "..      ...      ...         ...    ...      ...                ...   \n",
       "95  7619585  8890771  2014-11-01  Pance     H04N                4.0   \n",
       "96  7205959  8890771  2014-11-01  Pance     G02F                4.0   \n",
       "97  7205959  8890771  2014-11-01  Pance     G09F                4.0   \n",
       "98  7205959  8890771  2014-11-01  Pance     H04M                4.0   \n",
       "99  6760003  8890771  2014-11-01  Pance     G04G                4.0   \n",
       "\n",
       "    originality_hhi  \n",
       "0          0.896552  \n",
       "1          0.896552  \n",
       "2          0.896552  \n",
       "3          0.896552  \n",
       "4          0.896552  \n",
       "..              ...  \n",
       "95         0.847222  \n",
       "96         0.847222  \n",
       "97         0.847222  \n",
       "98         0.847222  \n",
       "99         0.847222  \n",
       "\n",
       "[100 rows x 7 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' calculate the Herfindahl index for backward citations (i.e., \"originality\")'''\n",
    "\n",
    "def hhi(series):\n",
    "    _, cnt = np.unique(series, return_counts=True)\n",
    "    return np.square(cnt/cnt.sum()).sum()    \n",
    "\n",
    "# the .agg method only accepts integers, so we must temporarily convert cpc classes to unique integers\n",
    "bcit_sample_df['cpc_code'], levels = pd.factorize(bcit_sample_df['from_cpc'])\n",
    "\n",
    "df3 = bcit_sample_df.groupby(['patent']).agg({'cpc_code': hhi}).reset_index()\n",
    "\n",
    "df3.columns = ['originality_hhi' if x=='cpc_code' else x for x in df3.columns] # rename the column 'originality_hhi'\n",
    "df3['originality_hhi'] = 1- df3['originality_hhi']  # originality is 1 - hhi\n",
    "\n",
    "bcit_sample_df = bcit_sample_df.merge(df3, on = \"patent\", how =\"outer\") # merge back into bcit dataframe\n",
    "\n",
    "# drop the temporary column\n",
    "bcit_sample_df = bcit_sample_df.drop(columns = 'cpc_code')\n",
    "\n",
    "bcit_sample_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>to</th>\n",
       "      <th>patent</th>\n",
       "      <th>patent_date</th>\n",
       "      <th>name</th>\n",
       "      <th>to_cpc</th>\n",
       "      <th>forward_citations</th>\n",
       "      <th>generality_hhi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10062352</td>\n",
       "      <td>9530381</td>\n",
       "      <td>2016-12-01</td>\n",
       "      <td>Bozarth</td>\n",
       "      <td>G09G</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.720000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10078977</td>\n",
       "      <td>9530381</td>\n",
       "      <td>2016-12-01</td>\n",
       "      <td>Bozarth</td>\n",
       "      <td>G09G</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.720000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10062352</td>\n",
       "      <td>9530381</td>\n",
       "      <td>2016-12-01</td>\n",
       "      <td>Bozarth</td>\n",
       "      <td>H01L</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.720000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10078977</td>\n",
       "      <td>9530381</td>\n",
       "      <td>2016-12-01</td>\n",
       "      <td>Bozarth</td>\n",
       "      <td>G02F</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.720000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10078977</td>\n",
       "      <td>9530381</td>\n",
       "      <td>2016-12-01</td>\n",
       "      <td>Bozarth</td>\n",
       "      <td>G06F</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.720000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>9734658</td>\n",
       "      <td>6906762</td>\n",
       "      <td>2005-06-01</td>\n",
       "      <td>Witehira et al.</td>\n",
       "      <td>A63F</td>\n",
       "      <td>154.0</td>\n",
       "      <td>0.249851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>9566500</td>\n",
       "      <td>6906762</td>\n",
       "      <td>2005-06-01</td>\n",
       "      <td>Witehira et al.</td>\n",
       "      <td>A63F</td>\n",
       "      <td>154.0</td>\n",
       "      <td>0.249851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>9595159</td>\n",
       "      <td>6906762</td>\n",
       "      <td>2005-06-01</td>\n",
       "      <td>Witehira et al.</td>\n",
       "      <td>A63F</td>\n",
       "      <td>154.0</td>\n",
       "      <td>0.249851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>9881453</td>\n",
       "      <td>6906762</td>\n",
       "      <td>2005-06-01</td>\n",
       "      <td>Witehira et al.</td>\n",
       "      <td>A63F</td>\n",
       "      <td>154.0</td>\n",
       "      <td>0.249851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>8749582</td>\n",
       "      <td>6906762</td>\n",
       "      <td>2005-06-01</td>\n",
       "      <td>Witehira et al.</td>\n",
       "      <td>A63F</td>\n",
       "      <td>154.0</td>\n",
       "      <td>0.249851</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           to   patent patent_date             name to_cpc forward_citations  \\\n",
       "0    10062352  9530381  2016-12-01          Bozarth   G09G               3.0   \n",
       "1    10078977  9530381  2016-12-01          Bozarth   G09G               3.0   \n",
       "2    10062352  9530381  2016-12-01          Bozarth   H01L               3.0   \n",
       "3    10078977  9530381  2016-12-01          Bozarth   G02F               3.0   \n",
       "4    10078977  9530381  2016-12-01          Bozarth   G06F               3.0   \n",
       "..        ...      ...         ...              ...    ...               ...   \n",
       "145   9734658  6906762  2005-06-01  Witehira et al.   A63F             154.0   \n",
       "146   9566500  6906762  2005-06-01  Witehira et al.   A63F             154.0   \n",
       "147   9595159  6906762  2005-06-01  Witehira et al.   A63F             154.0   \n",
       "148   9881453  6906762  2005-06-01  Witehira et al.   A63F             154.0   \n",
       "149   8749582  6906762  2005-06-01  Witehira et al.   A63F             154.0   \n",
       "\n",
       "     generality_hhi  \n",
       "0          0.720000  \n",
       "1          0.720000  \n",
       "2          0.720000  \n",
       "3          0.720000  \n",
       "4          0.720000  \n",
       "..              ...  \n",
       "145        0.249851  \n",
       "146        0.249851  \n",
       "147        0.249851  \n",
       "148        0.249851  \n",
       "149        0.249851  \n",
       "\n",
       "[150 rows x 7 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' calculate the Herfindahl index for forward citations (i.e., \"generality\")'''\n",
    "\n",
    "def hhi(series):\n",
    "    _, cnt = np.unique(series, return_counts=True)\n",
    "    return np.square(cnt/cnt.sum()).sum()    \n",
    "\n",
    "# the .agg method only accepts integers, so we must temporarily convert cpc classes to unique integers\n",
    "fcit_sample_df['cpc_code'], levels = pd.factorize(fcit_sample_df['to_cpc']) \n",
    "\n",
    "df2 = fcit_sample_df.groupby(['patent']).agg({'cpc_code': hhi}).reset_index()\n",
    "\n",
    "df2.columns = ['generality_hhi' if x=='cpc_code' else x for x in df2.columns] # rename the column 'generality_hhi'\n",
    "df2['generality_hhi'] = 1- df2['generality_hhi']  # generality is 1 - hhi\n",
    "\n",
    "fcit_sample_df = fcit_sample_df.merge(df2, on = \"patent\", how =\"outer\") # merge back into sample_df\n",
    "\n",
    "# drop the temporary column\n",
    "fcit_sample_df = fcit_sample_df.drop(columns = 'cpc_code')\n",
    "\n",
    "fcit_sample_df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}