11 - Vertex AI > Features - Feature Store.ipynb

{"cells": [{"cell_type": "markdown", "id": "60e9fd90", "metadata": {}, "source": ["![ga4](https://www.google-analytics.com/collect?v=2&tid=G-6VDTYWLKX6&cid=1&en=page_view&sid=1&dl=statmike%2Fvertex-ai-mlops&dt=11+-+Vertex+AI+%3E+Features+-+Feature+Store.ipynb)\n", "\n", "# 11 - Vertex AI > Features - Feature Store\n", "\n", "This is a demonstration of [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/overview). A feature store is a central repository for organizing, storing, and retrieving features.  This is a fully managed service that scales the underlying compute and storage resources.  The feature store becomes a central location for serving features for training and prediction with low-latency. It stores feature values at points-in-time:\n", "\n", "-  Point-in-time lookups for retrieving features for model training. Retrieve feature values prior to a prediction to prevent data leakage.\n", "-  Manage training-serving skew\n", "\n", "### Prerequisites:\n", "-  01 - BigQuery - Table Data Source\n", "-  Any of 02-05 That Deploy A Model To An Endpoint\n", "   -  Used to demonstrate online predictions with feature store serving features\n", "\n", "### Resources:\n", "-  [Python Client for Vertex AI](https://googleapis.dev/python/aiplatform/latest/aiplatform.html)\n", "   -  Currently using the [v1beta1 services](https://googleapis.dev/python/aiplatform/latest/aiplatform_v1beta1/services.html)\n", "-  [Feature Store Overview](https://cloud.google.com/vertex-ai/docs/featurestore/overview)\n", "-  [Data Model and Concepts](https://cloud.google.com/vertex-ai/docs/featurestore/concepts)\n", "-  [Best Practices](https://cloud.google.com/vertex-ai/docs/featurestore/best-practices) including info on composite entity types\n", "\n", "### Conceptual Flow & Workflow\n", "<p align=\"center\">\n", "  <img alt=\"Conceptual Flow\" src=\"./architectures/slides/11_arch.png\" width=\"45%\">\n", "&nbsp; &nbsp; &nbsp; &nbsp;\n", "  <img alt=\"Workflow\" src=\"./architectures/slides/11_console.png\" width=\"45%\">\n", "</p>"]}, {"cell_type": "markdown", "id": "5ea11901", "metadata": {}, "source": ["---\n", "## Setup"]}, {"cell_type": "markdown", "id": "ce66014a", "metadata": {}, "source": ["inputs:"]}, {"cell_type": "code", "execution_count": 12, "id": "40159729", "metadata": {}, "outputs": [], "source": ["REGION = 'us-central1'\n", "PROJECT_ID='statmike-demo2'\n", "DATANAME = 'fraud'\n", "NOTEBOOK = '11'\n", "\n", "ENTITYTYPE_ID = 'transaction'\n", "\n", "# Model Training\n", "VAR_TARGET = 'Class'\n", "VAR_OMIT = 'transaction_id' # add more variables to the string with space delimiters"]}, {"cell_type": "markdown", "id": "753cceef", "metadata": {}, "source": ["packages:"]}, {"cell_type": "code", "execution_count": 13, "id": "8e3a4d0e", "metadata": {}, "outputs": [], "source": ["#from google.cloud.aiplatform_v1beta1 import (FeaturestoreOnlineServingServiceClient, FeaturestoreServiceClient, types)\n", "from google.cloud import aiplatform\n", "\n", "from google.protobuf.duration_pb2 import Duration\n", "from google.protobuf.timestamp_pb2 import Timestamp\n", "from google.protobuf.field_mask_pb2 import FieldMask\n", "\n", "from google.cloud import bigquery\n", "#from google.cloud.aiplatform_v1beta1 import (PredictionServiceClient, EndpointServiceClient)\n", "from google.protobuf import json_format\n", "from google.protobuf.struct_pb2 import Value\n", "import json\n", "import numpy as np"]}, {"cell_type": "markdown", "id": "7d2871b3", "metadata": {}, "source": ["clients:"]}, {"cell_type": "code", "execution_count": 14, "id": "6e393739", "metadata": {}, "outputs": [], "source": ["client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n", "\n", "clients = {}\n", "clients['fs'] = aiplatform.gapic.FeaturestoreServiceClient(client_options = client_options)\n", "clients['fs_olserve'] = aiplatform.gapic.FeaturestoreOnlineServingServiceClient(client_options = client_options)\n", "\n", "clients['bq'] = bigquery.Client()\n", "\n", "aiplatform.init(project=PROJECT_ID, location=REGION)"]}, {"cell_type": "markdown", "id": "2210d4ce", "metadata": {}, "source": ["parameters:"]}, {"cell_type": "code", "execution_count": 15, "id": "5a1b80a0", "metadata": {}, "outputs": [], "source": ["PARENT = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n", "DIR = f\"temp/{NOTEBOOK}\""]}, {"cell_type": "markdown", "id": "7a0eaefd", "metadata": {}, "source": ["environment:"]}, {"cell_type": "code", "execution_count": 16, "id": "e66c4142", "metadata": {}, "outputs": [], "source": ["!rm -rf {DIR}\n", "!mkdir -p {DIR}"]}, {"cell_type": "markdown", "id": "3829cad3", "metadata": {}, "source": ["---\n", "## Feature Store Data model\n", "Feature Store organizes data with the following 3 important hierarchical concepts:\n", "\n", "Featurestore -> EntityType -> Feature\n", "\n", "- **Featurestore**: the place to store your features\n", "    - **EntityType**: under a Featurestore, an EntityType describes an object to be modeled, real one or virtual one.\n", "        - **Feature**: under an EntityType, a feature describes an attribute of the EntityType\n", "\n", "For the digits data used in these examples, the feature store will be called digits_featurestore.  The store has 1 entity type: images.  The features will be the pixels and the target values."]}, {"cell_type": "markdown", "id": "f23fa820", "metadata": {}, "source": ["---\n", "## Create Feature Store"]}, {"cell_type": "code", "execution_count": 17, "id": "4e54b9c5", "metadata": {}, "outputs": [], "source": ["FEATURESTORE_ID = DATANAME"]}, {"cell_type": "code", "execution_count": 19, "id": "c56ecb54", "metadata": {}, "outputs": [], "source": ["featurestore_lro = clients['fs'].create_featurestore(\n", "    request = aiplatform.gapic.CreateFeaturestoreRequest(\n", "        parent = PARENT,\n", "        featurestore_id = FEATURESTORE_ID,\n", "        featurestore = aiplatform.gapic.Featurestore(\n", "            #description/display_name = f\"Notebook {NOTEBOOK} demonstration of Vertex AI Features (feature store) using {DATANAME} data\",\n", "            labels = {'notebook':f'{NOTEBOOK}'},\n", "            online_serving_config = aiplatform.gapic.Featurestore.OnlineServingConfig(\n", "                fixed_node_count = 2\n", "            ),\n", "        ),\n", "    )\n", ")"]}, {"cell_type": "code", "execution_count": 20, "id": "ee4ee6ff", "metadata": {}, "outputs": [{"data": {"text/plain": ["name: \"projects/764015827198/locations/us-central1/featurestores/fraud\""]}, "execution_count": 20, "metadata": {}, "output_type": "execute_result"}], "source": ["featurestore_lro.result()"]}, {"cell_type": "markdown", "id": "a81a2ddf", "metadata": {}, "source": ["Use `get_featurestore` to see details of specified feature store:"]}, {"cell_type": "code", "execution_count": 21, "id": "b4c911b1", "metadata": {}, "outputs": [{"data": {"text/plain": ["name: \"projects/764015827198/locations/us-central1/featurestores/fraud\"\n", "create_time {\n", "  seconds: 1638982792\n", "  nanos: 491139000\n", "}\n", "update_time {\n", "  seconds: 1638982792\n", "  nanos: 585025000\n", "}\n", "etag: \"AMEw9yNxcAN7A5S1IctiEi4ldC9nDVUCM-STwZ8Yw8ZhzGohPCV0ITJdTcV7copy_NJ5\"\n", "labels {\n", "  key: \"notebook\"\n", "  value: \"07\"\n", "}\n", "online_serving_config {\n", "  fixed_node_count: 2\n", "}\n", "state: STABLE"]}, "execution_count": 21, "metadata": {}, "output_type": "execute_result"}], "source": ["clients['fs'].get_featurestore(name=clients['fs'].featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID))"]}, {"cell_type": "markdown", "id": "99f492d2", "metadata": {}, "source": ["Use `list_featurestores` to see details of all feature stores:"]}, {"cell_type": "code", "execution_count": 22, "id": "ace5cd81", "metadata": {}, "outputs": [{"data": {"text/plain": ["ListFeaturestoresPager<featurestores {\n", "  name: \"projects/764015827198/locations/us-central1/featurestores/fraud\"\n", "  create_time {\n", "    seconds: 1638982792\n", "    nanos: 491139000\n", "  }\n", "  update_time {\n", "    seconds: 1638982792\n", "    nanos: 585025000\n", "  }\n", "  etag: \"AMEw9yNpYLNnqe_d4QsDtt-CjUyVQGPXI8FdzC5iIlBHYu3Yb29KZJ2iePvEJlC0sjpx\"\n", "  labels {\n", "    key: \"notebook\"\n", "    value: \"07\"\n", "  }\n", "  online_serving_config {\n", "    fixed_node_count: 2\n", "  }\n", "  state: STABLE\n", "}\n", ">"]}, "execution_count": 22, "metadata": {}, "output_type": "execute_result"}], "source": ["clients['fs'].list_featurestores(parent=PARENT)"]}, {"cell_type": "markdown", "id": "34eaf52c", "metadata": {}, "source": ["---\n", "## Create Entity Type"]}, {"cell_type": "code", "execution_count": 26, "id": "f5ac9805", "metadata": {}, "outputs": [], "source": ["entitytype_lro = clients['fs'].create_entity_type(\n", "    request = aiplatform.gapic.CreateEntityTypeRequest(\n", "        parent = clients['fs'].featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n", "        entity_type_id = ENTITYTYPE_ID,\n", "        entity_type = aiplatform.gapic.EntityType(\n", "            description = f\"Entity: {ENTITYTYPE_ID}, for data: {DATANAME}\"\n", "        ),\n", "    )\n", ")"]}, {"cell_type": "code", "execution_count": 27, "id": "535ff4de", "metadata": {}, "outputs": [{"data": {"text/plain": ["name: \"projects/764015827198/locations/us-central1/featurestores/fraud/entityTypes/transaction\""]}, "execution_count": 27, "metadata": {}, "output_type": "execute_result"}], "source": ["entitytype_lro.result()"]}, {"cell_type": "markdown", "id": "2f04c291", "metadata": {}, "source": ["Use `list_entity_types` to see details of all entity types:"]}, {"cell_type": "code", "execution_count": 28, "id": "925e906a", "metadata": {}, "outputs": [{"data": {"text/plain": ["ListEntityTypesPager<entity_types {\n", "  name: \"projects/764015827198/locations/us-central1/featurestores/fraud/entityTypes/transaction\"\n", "  description: \"Entity: transaction, for data: fraud\"\n", "  create_time {\n", "    seconds: 1638988806\n", "    nanos: 949907000\n", "  }\n", "  update_time {\n", "    seconds: 1638988806\n", "    nanos: 949907000\n", "  }\n", "  etag: \"AMEw9yN5AepXX82rfwPCQKeYhfvAKa7nNCGwOnDMFILFTJ1JuUVj5KcaiHocYcFJn-Tv\"\n", "}\n", ">"]}, "execution_count": 28, "metadata": {}, "output_type": "execute_result"}], "source": ["clients['fs'].list_entity_types(parent = f\"{PARENT}/featurestores/{FEATURESTORE_ID}\")"]}, {"cell_type": "markdown", "id": "c92cdcf0", "metadata": {}, "source": ["---\n", "## Create Features"]}, {"cell_type": "markdown", "id": "e7c442eb", "metadata": {}, "source": ["Get the schema of the data source for new features:"]}, {"cell_type": "code", "execution_count": 29, "id": "cbf77a54", "metadata": {}, "outputs": [], "source": ["schema = clients['bq'].query(query = f\"SELECT * FROM {DATANAME}.INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '{DATANAME}_prepped'\").to_dataframe()"]}, {"cell_type": "code", "execution_count": 30, "id": "a347c398", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>table_catalog</th>\n", "      <th>table_schema</th>\n", "      <th>table_name</th>\n", "      <th>column_name</th>\n", "      <th>ordinal_position</th>\n", "      <th>is_nullable</th>\n", "      <th>data_type</th>\n", "      <th>is_generated</th>\n", "      <th>generation_expression</th>\n", "      <th>is_stored</th>\n", "      <th>is_hidden</th>\n", "      <th>is_updatable</th>\n", "      <th>is_system_defined</th>\n", "      <th>is_partitioning_column</th>\n", "      <th>clustering_ordinal_position</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>0</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>Time</td>\n", "      <td>1</td>\n", "      <td>YES</td>\n", "      <td>INT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>1</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V1</td>\n", "      <td>2</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>2</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V2</td>\n", "      <td>3</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>3</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V3</td>\n", "      <td>4</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>4</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V4</td>\n", "      <td>5</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>5</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V5</td>\n", "      <td>6</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>6</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V6</td>\n", "      <td>7</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>7</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V7</td>\n", "      <td>8</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>8</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V8</td>\n", "      <td>9</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>9</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V9</td>\n", "      <td>10</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>10</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V10</td>\n", "      <td>11</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>11</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V11</td>\n", "      <td>12</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>12</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V12</td>\n", "      <td>13</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>13</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V13</td>\n", "      <td>14</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>14</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V14</td>\n", "      <td>15</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>15</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V15</td>\n", "      <td>16</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>16</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V16</td>\n", "      <td>17</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>17</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V17</td>\n", "      <td>18</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>18</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V18</td>\n", "      <td>19</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>19</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V19</td>\n", "      <td>20</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>20</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V20</td>\n", "      <td>21</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>21</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V21</td>\n", "      <td>22</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>22</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V22</td>\n", "      <td>23</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>23</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V23</td>\n", "      <td>24</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>24</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V24</td>\n", "      <td>25</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>25</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V25</td>\n", "      <td>26</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>26</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V26</td>\n", "      <td>27</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>27</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V27</td>\n", "      <td>28</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>28</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>V28</td>\n", "      <td>29</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>29</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>Amount</td>\n", "      <td>30</td>\n", "      <td>YES</td>\n", "      <td>FLOAT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>30</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>Class</td>\n", "      <td>31</td>\n", "      <td>YES</td>\n", "      <td>INT64</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>31</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>transaction_id</td>\n", "      <td>32</td>\n", "      <td>YES</td>\n", "      <td>STRING</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>32</th>\n", "      <td>statmike-demo2</td>\n", "      <td>fraud</td>\n", "      <td>fraud_prepped</td>\n", "      <td>splits</td>\n", "      <td>33</td>\n", "      <td>YES</td>\n", "      <td>STRING</td>\n", "      <td>NEVER</td>\n", "      <td>None</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>None</td>\n", "      <td>NO</td>\n", "      <td>NO</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["     table_catalog table_schema     table_name     column_name  \\\n", "0   statmike-demo2        fraud  fraud_prepped            Time   \n", "1   statmike-demo2        fraud  fraud_prepped              V1   \n", "2   statmike-demo2        fraud  fraud_prepped              V2   \n", "3   statmike-demo2        fraud  fraud_prepped              V3   \n", "4   statmike-demo2        fraud  fraud_prepped              V4   \n", "5   statmike-demo2        fraud  fraud_prepped              V5   \n", "6   statmike-demo2        fraud  fraud_prepped              V6   \n", "7   statmike-demo2        fraud  fraud_prepped              V7   \n", "8   statmike-demo2        fraud  fraud_prepped              V8   \n", "9   statmike-demo2        fraud  fraud_prepped              V9   \n", "10  statmike-demo2        fraud  fraud_prepped             V10   \n", "11  statmike-demo2        fraud  fraud_prepped             V11   \n", "12  statmike-demo2        fraud  fraud_prepped             V12   \n", "13  statmike-demo2        fraud  fraud_prepped             V13   \n", "14  statmike-demo2        fraud  fraud_prepped             V14   \n", "15  statmike-demo2        fraud  fraud_prepped             V15   \n", "16  statmike-demo2        fraud  fraud_prepped             V16   \n", "17  statmike-demo2        fraud  fraud_prepped             V17   \n", "18  statmike-demo2        fraud  fraud_prepped             V18   \n", "19  statmike-demo2        fraud  fraud_prepped             V19   \n", "20  statmike-demo2        fraud  fraud_prepped             V20   \n", "21  statmike-demo2        fraud  fraud_prepped             V21   \n", "22  statmike-demo2        fraud  fraud_prepped             V22   \n", "23  statmike-demo2        fraud  fraud_prepped             V23   \n", "24  statmike-demo2        fraud  fraud_prepped             V24   \n", "25  statmike-demo2        fraud  fraud_prepped             V25   \n", "26  statmike-demo2        fraud  fraud_prepped             V26   \n", "27  statmike-demo2        fraud  fraud_prepped             V27   \n", "28  statmike-demo2        fraud  fraud_prepped             V28   \n", "29  statmike-demo2        fraud  fraud_prepped          Amount   \n", "30  statmike-demo2        fraud  fraud_prepped           Class   \n", "31  statmike-demo2        fraud  fraud_prepped  transaction_id   \n", "32  statmike-demo2        fraud  fraud_prepped          splits   \n", "\n", "    ordinal_position is_nullable data_type is_generated generation_expression  \\\n", "0                  1         YES     INT64        NEVER                  None   \n", "1                  2         YES   FLOAT64        NEVER                  None   \n", "2                  3         YES   FLOAT64        NEVER                  None   \n", "3                  4         YES   FLOAT64        NEVER                  None   \n", "4                  5         YES   FLOAT64        NEVER                  None   \n", "5                  6         YES   FLOAT64        NEVER                  None   \n", "6                  7         YES   FLOAT64        NEVER                  None   \n", "7                  8         YES   FLOAT64        NEVER                  None   \n", "8                  9         YES   FLOAT64        NEVER                  None   \n", "9                 10         YES   FLOAT64        NEVER                  None   \n", "10                11         YES   FLOAT64        NEVER                  None   \n", "11                12         YES   FLOAT64        NEVER                  None   \n", "12                13         YES   FLOAT64        NEVER                  None   \n", "13                14         YES   FLOAT64        NEVER                  None   \n", "14                15         YES   FLOAT64        NEVER                  None   \n", "15                16         YES   FLOAT64        NEVER                  None   \n", "16                17         YES   FLOAT64        NEVER                  None   \n", "17                18         YES   FLOAT64        NEVER                  None   \n", "18                19         YES   FLOAT64        NEVER                  None   \n", "19                20         YES   FLOAT64        NEVER                  None   \n", "20                21         YES   FLOAT64        NEVER                  None   \n", "21                22         YES   FLOAT64        NEVER                  None   \n", "22                23         YES   FLOAT64        NEVER                  None   \n", "23                24         YES   FLOAT64        NEVER                  None   \n", "24                25         YES   FLOAT64        NEVER                  None   \n", "25                26         YES   FLOAT64        NEVER                  None   \n", "26                27         YES   FLOAT64        NEVER                  None   \n", "27                28         YES   FLOAT64        NEVER                  None   \n", "28                29         YES   FLOAT64        NEVER                  None   \n", "29                30         YES   FLOAT64        NEVER                  None   \n", "30                31         YES     INT64        NEVER                  None   \n", "31                32         YES    STRING        NEVER                  None   \n", "32                33         YES    STRING        NEVER                  None   \n", "\n", "   is_stored is_hidden is_updatable is_system_defined is_partitioning_column  \\\n", "0       None        NO         None                NO                     NO   \n", "1       None        NO         None                NO                     NO   \n", "2       None        NO         None                NO                     NO   \n", "3       None        NO         None                NO                     NO   \n", "4       None        NO         None                NO                     NO   \n", "5       None        NO         None                NO                     NO   \n", "6       None        NO         None                NO                     NO   \n", "7       None        NO         None                NO                     NO   \n", "8       None        NO         None                NO                     NO   \n", "9       None        NO         None                NO                     NO   \n", "10      None        NO         None                NO                     NO   \n", "11      None        NO         None                NO                     NO   \n", "12      None        NO         None                NO                     NO   \n", "13      None        NO         None                NO                     NO   \n", "14      None        NO         None                NO                     NO   \n", "15      None        NO         None                NO                     NO   \n", "16      None        NO         None                NO                     NO   \n", "17      None        NO         None                NO                     NO   \n", "18      None        NO         None                NO                     NO   \n", "19      None        NO         None                NO                     NO   \n", "20      None        NO         None                NO                     NO   \n", "21      None        NO         None                NO                     NO   \n", "22      None        NO         None                NO                     NO   \n", "23      None        NO         None                NO                     NO   \n", "24      None        NO         None                NO                     NO   \n", "25      None        NO         None                NO                     NO   \n", "26      None        NO         None                NO                     NO   \n", "27      None        NO         None                NO                     NO   \n", "28      None        NO         None                NO                     NO   \n", "29      None        NO         None                NO                     NO   \n", "30      None        NO         None                NO                     NO   \n", "31      None        NO         None                NO                     NO   \n", "32      None        NO         None                NO                     NO   \n", "\n", "    clustering_ordinal_position  \n", "0                           NaN  \n", "1                           NaN  \n", "2                           NaN  \n", "3                           NaN  \n", "4                           NaN  \n", "5                           NaN  \n", "6                           NaN  \n", "7                           NaN  \n", "8                           NaN  \n", "9                           NaN  \n", "10                          NaN  \n", "11                          NaN  \n", "12                          NaN  \n", "13                          NaN  \n", "14                          NaN  \n", "15                          NaN  \n", "16                          NaN  \n", "17                          NaN  \n", "18                          NaN  \n", "19                          NaN  \n", "20                          NaN  \n", "21                          NaN  \n", "22                          NaN  \n", "23                          NaN  \n", "24                          NaN  \n", "25                          NaN  \n", "26                          NaN  \n", "27                          NaN  \n", "28                          NaN  \n", "29                          NaN  \n", "30                          NaN  \n", "31                          NaN  \n", "32                          NaN  "]}, "execution_count": 30, "metadata": {}, "output_type": "execute_result"}], "source": ["schema"]}, {"cell_type": "markdown", "id": "8cea6abc", "metadata": {}, "source": ["Prepare a request for `batch_create_features`:\n", "- specification for the features, data type and descriptions ...."]}, {"cell_type": "code", "execution_count": 31, "id": "a73a86cf", "metadata": {}, "outputs": [], "source": ["REQUESTS = []\n", "for i in range(schema.shape[0]):\n", "    \n", "    if schema['column_name'][i] in [VAR_TARGET, 'splits'] + VAR_OMIT.split():\n", "        continue\n", "    \n", "    if schema['data_type'][i] == 'STRING': value_type = aiplatform.gapic.Feature.ValueType.STRING\n", "    elif schema['data_type'][i] == 'INT64': value_type = aiplatform.gapic.Feature.ValueType.INT64\n", "    elif schema['data_type'][i] == 'FLOAT64': value_type = aiplatform.gapic.Feature.ValueType.DOUBLE\n", "    \n", "    description = f\"Column named {schema['column_name'][i]} from BQ Table {PROJECT_ID}.{DATANAME}.{DATANAME}_prepped\"\n", "    \n", "    REQUESTS.append(\n", "        aiplatform.gapic.CreateFeatureRequest(\n", "            feature = aiplatform.gapic.Feature(\n", "                value_type = value_type,\n", "                description = description,\n", "                # optional, monitoring_config here as override, otherwise it inherits from entity_type\n", "            ),\n", "            feature_id = schema['column_name'][i].lower(),\n", "        )    \n", "    )"]}, {"cell_type": "code", "execution_count": 32, "id": "a1165a0e", "metadata": {}, "outputs": [], "source": ["batchfeatures = clients['fs'].batch_create_features(\n", "    parent = clients['fs'].entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, ENTITYTYPE_ID),\n", "    requests = REQUESTS,\n", ")"]}, {"cell_type": "code", "execution_count": 62, "id": "eeabe921", "metadata": {}, "outputs": [], "source": ["#list(item.name for item in batchfeatures.result().features)"]}, {"cell_type": "markdown", "id": "e2fe9995", "metadata": {}, "source": ["---\n", "## Search Features\n", "Search goes across all Feature Stores and Entity Types.\n", "\n", "Also, use the list_features function to list all."]}, {"cell_type": "code", "execution_count": 33, "id": "a3acd3be", "metadata": {}, "outputs": [{"data": {"text/plain": ["name: \"projects/764015827198/locations/us-central1/featurestores/fraud/entityTypes/transaction/features/amount\"\n", "description: \"Column named Amount from BQ Table statmike-demo2.fraud.fraud_prepped\"\n", "value_type: DOUBLE\n", "create_time {\n", "  seconds: 1638990157\n", "  nanos: 457386000\n", "}\n", "update_time {\n", "  seconds: 1638990157\n", "  nanos: 457386000\n", "}"]}, "execution_count": 33, "metadata": {}, "output_type": "execute_result"}], "source": ["# return the first feature:\n", "list(clients['fs'].search_features(location = PARENT))[0]"]}, {"cell_type": "code", "execution_count": 35, "id": "5dd7dc03", "metadata": {}, "outputs": [{"data": {"text/plain": ["[name: \"projects/764015827198/locations/us-central1/featurestores/fraud/entityTypes/transaction/features/time\"\n", " description: \"Column named Time from BQ Table statmike-demo2.fraud.fraud_prepped\"\n", " value_type: INT64\n", " create_time {\n", "   seconds: 1638990157\n", "   nanos: 337811000\n", " }\n", " update_time {\n", "   seconds: 1638990157\n", "   nanos: 337811000\n", " }]"]}, "execution_count": 35, "metadata": {}, "output_type": "execute_result"}], "source": ["# find all features with INT64 value type\n", "list(clients['fs'].search_features(aiplatform.gapic.SearchFeaturesRequest(location = PARENT, query = \"value_type=INT64\")))"]}, {"cell_type": "code", "execution_count": 36, "id": "dab3603a", "metadata": {}, "outputs": [{"data": {"text/plain": ["[name: \"projects/764015827198/locations/us-central1/featurestores/fraud/entityTypes/transaction/features/v19\"\n", " description: \"Column named V19 from BQ Table statmike-demo2.fraud.fraud_prepped\"\n", " value_type: DOUBLE\n", " create_time {\n", "   seconds: 1638990157\n", "   nanos: 381741000\n", " }\n", " update_time {\n", "   seconds: 1638990157\n", "   nanos: 381741000\n", " },\n", " name: \"projects/764015827198/locations/us-central1/featurestores/fraud/entityTypes/transaction/features/v9\"\n", " description: \"Column named V9 from BQ Table statmike-demo2.fraud.fraud_prepped\"\n", " value_type: DOUBLE\n", " create_time {\n", "   seconds: 1638990157\n", "   nanos: 355483000\n", " }\n", " update_time {\n", "   seconds: 1638990157\n", "   nanos: 355483000\n", " }]"]}, "execution_count": 36, "metadata": {}, "output_type": "execute_result"}], "source": ["# find all features of the form V*9 with DOUBLE value type\n", "list(clients['fs'].search_features(aiplatform.gapic.SearchFeaturesRequest(location = PARENT, query = \"feature_id:V*9 AND value_type=DOUBLE\")))"]}, {"cell_type": "markdown", "id": "71e5412f", "metadata": {}, "source": ["---\n", "## Import Feature Values\n", "- BigQuery (THIS DEMO)\n", "- Avro\n", "- CSV"]}, {"cell_type": "markdown", "id": "43a9fcf4", "metadata": {}, "source": ["Prepare a source table with timestamp (update_time) and unique id's for each entity"]}, {"cell_type": "code", "execution_count": 37, "id": "3e9e56b2", "metadata": {}, "outputs": [], "source": ["query = f\"\"\"\n", "CREATE OR REPLACE TABLE {PROJECT_ID}.{DATANAME}.{DATANAME}_featurestore_import AS\n", "WITH A AS \n", "    (SELECT *, CAST(FLOOR(10*RAND()) AS INT64) day_offset\n", "    FROM {PROJECT_ID}.{DATANAME}.{DATANAME}_prepped)\n", "SELECT * EXCEPT(day_offset),\n", "        DATE_SUB(CURRENT_TIMESTAMP, INTERVAL day_offset DAY) AS update_time\n", "FROM A\n", "\"\"\"\n", "bqjob = clients['bq'].query(query = query)"]}, {"cell_type": "code", "execution_count": 38, "id": "11bbcf9f", "metadata": {}, "outputs": [{"data": {"text/plain": ["<google.cloud.bigquery.table._EmptyRowIterator at 0x7f11156b0110>"]}, "execution_count": 38, "metadata": {}, "output_type": "execute_result"}], "source": ["bqjob.result()"]}, {"cell_type": "markdown", "id": "0d99edf8", "metadata": {}, "source": ["Create Feature specification for each feature in the input source:"]}, {"cell_type": "code", "execution_count": 39, "id": "bc45bed6", "metadata": {}, "outputs": [], "source": ["FEATURE_SPECS = []\n", "for i in range(schema.shape[0]):\n", "    if schema['column_name'][i] in [VAR_TARGET, 'splits'] + VAR_OMIT.split():\n", "        continue\n", "    \n", "    FEATURE_SPECS.append(\n", "        aiplatform.gapic.ImportFeatureValuesRequest.FeatureSpec(\n", "            id = schema['column_name'][i].lower(),\n", "            source_field = schema['column_name'][i]\n", "        )\n", "    )"]}, {"cell_type": "code", "execution_count": 40, "id": "fa7ba054", "metadata": {}, "outputs": [], "source": ["import_request = aiplatform.gapic.ImportFeatureValuesRequest(\n", "    entity_type = clients['fs'].entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, ENTITYTYPE_ID),\n", "    bigquery_source = aiplatform.gapic.BigQuerySource(input_uri = f'bq://{PROJECT_ID}.{DATANAME}.{DATANAME}_featurestore_import'),\n", "    feature_time_field = \"update_time\",\n", "    feature_time = Timestamp().GetCurrentTime(),\n", "    entity_id_field = \"transaction_id\",\n", "    feature_specs = FEATURE_SPECS,\n", "    worker_count = 4,\n", ")"]}, {"cell_type": "code", "execution_count": 41, "id": "0f8d4c24", "metadata": {}, "outputs": [], "source": ["importjob = clients['fs'].import_feature_values(request = import_request)"]}, {"cell_type": "code", "execution_count": 42, "id": "a93ef0a9", "metadata": {}, "outputs": [{"data": {"text/plain": ["imported_entity_count: 284807\n", "imported_feature_value_count: 8544210"]}, "execution_count": 42, "metadata": {}, "output_type": "execute_result"}], "source": ["importjob.result()"]}, {"cell_type": "markdown", "id": "7137ad98", "metadata": {}, "source": ["---\n", "## Prediction with Feature Store for Serving Features"]}, {"cell_type": "markdown", "id": "1803b00d", "metadata": {}, "source": ["### Entity Id's\n", "Retrieve a list of entity id's from the source BigQuery table.  These are in the column `transaction_id`."]}, {"cell_type": "code", "execution_count": 43, "id": "5ecdb9e4", "metadata": {}, "outputs": [], "source": ["unique_id = clients['bq'].query(query = f\"SELECT * FROM {DATANAME}.{DATANAME}_prepped WHERE splits='TEST' LIMIT 10\").to_dataframe()"]}, {"cell_type": "code", "execution_count": 44, "id": "a4424240", "metadata": {}, "outputs": [{"data": {"text/plain": ["'07fdced0-3837-47a1-9526-64d74ad9b113'"]}, "execution_count": 44, "metadata": {}, "output_type": "execute_result"}], "source": ["unique_id['transaction_id'][0]"]}, {"cell_type": "markdown", "id": "8fc3a6c1", "metadata": {}, "source": ["### Data For Prediction: Single Entity Served by Vertex AI > Features (Feature Store)"]}, {"cell_type": "code", "execution_count": 47, "id": "895c87bc", "metadata": {}, "outputs": [], "source": ["feature_values = clients['fs_olserve'].read_feature_values(\n", "    request = aiplatform.gapic.ReadFeatureValuesRequest(\n", "        entity_type = clients['fs'].entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, ENTITYTYPE_ID),\n", "        entity_id = unique_id['transaction_id'][0],\n", "        feature_selector = aiplatform.gapic.FeatureSelector(id_matcher=aiplatform.gapic.IdMatcher(ids=['*'])),\n", "    )\n", ")"]}, {"cell_type": "code", "execution_count": 48, "id": "0e290e7d", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["['v18', 'time', 'v26', 'v9', 'v3', 'v24', 'v8', 'v1', 'v4', 'v5', 'v14', 'v28', 'v27', 'v6', 'v16', 'v21', 'v7', 'v25', 'v22', 'v2', 'v15', 'v12', 'amount', 'v23', 'v13', 'v17', 'v19', 'v11', 'v10', 'v20']\n"]}], "source": ["print(list(item.id for item in feature_values.header.feature_descriptors))"]}, {"cell_type": "code", "execution_count": 49, "id": "bb463851", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["[-1.01262330333815, 0.0, 1.0288470541571502, -0.13296614676554902, 0.79650222865, -0.25821805726084, 0.46149546559133703, 1.07306279044003, 1.75438907383858, -0.219909538224871, 0.223515261314392, -0.0131252979106899, -0.0376919034755461, 0.930348759049177, -0.16033206009858803, -0.0452503674451511, -0.515894450159377, 0.171860617271716, 0.035042952311969, -0.026793043261075, -0.864056157762422, 0.404511129977238, 0.0, 0.0485385294222296, -1.3826303643001698, 0.336310126389912, -0.7461822853235991, 1.03689341495792, 0.5056034354709891, -0.26507148271390196]\n"]}], "source": ["print(list(item.value.double_value for item in feature_values.entity_view.data))"]}, {"cell_type": "markdown", "id": "d2329aca", "metadata": {}, "source": ["### Prepare a record for prediction: instance and parameters lists"]}, {"cell_type": "code", "execution_count": 50, "id": "2f9319bc", "metadata": {}, "outputs": [], "source": ["newob = {}\n", "features = list(item.id for item in feature_values.header.feature_descriptors)\n", "for e, f in enumerate(features):\n", "    newob[f.capitalize()] = feature_values.entity_view.data[e].value.double_value"]}, {"cell_type": "code", "execution_count": 51, "id": "bf8fee66", "metadata": {}, "outputs": [], "source": ["instances = [json_format.ParseDict(newob, Value())]\n", "parameters = json_format.ParseDict({}, Value())"]}, {"cell_type": "markdown", "id": "c348a58e", "metadata": {}, "source": ["### Pick An Endpoint\n", "A list index of [0] here retrieves the first endpoint in this project:"]}, {"cell_type": "code", "execution_count": 52, "id": "f095ed8d", "metadata": {}, "outputs": [{"data": {"text/plain": ["'05f_fraud_20211203120602'"]}, "execution_count": 52, "metadata": {}, "output_type": "execute_result"}], "source": ["aiplatform.Endpoint.list()[0].display_name"]}, {"cell_type": "code", "execution_count": 53, "id": "ad0ef300", "metadata": {}, "outputs": [], "source": ["endpoint = aiplatform.Endpoint(endpoint_name = aiplatform.Endpoint.list()[0].name)"]}, {"cell_type": "markdown", "id": "11bbfd5a", "metadata": {}, "source": ["### Get Predictions: Python Client"]}, {"cell_type": "code", "execution_count": 54, "id": "2dc9ff1a", "metadata": {}, "outputs": [], "source": ["prediction = endpoint.predict(instances = instances, parameters = parameters)"]}, {"cell_type": "code", "execution_count": 55, "id": "ff42dacd", "metadata": {}, "outputs": [{"data": {"text/plain": ["Prediction(predictions=[[0.999251425, 0.000748594233]], deployed_model_id='2291109553401495552', explanations=None)"]}, "execution_count": 55, "metadata": {}, "output_type": "execute_result"}], "source": ["prediction"]}, {"cell_type": "code", "execution_count": 56, "id": "904c9f80", "metadata": {}, "outputs": [{"data": {"text/plain": ["[0.999251425, 0.000748594233]"]}, "execution_count": 56, "metadata": {}, "output_type": "execute_result"}], "source": ["prediction.predictions[0]"]}, {"cell_type": "code", "execution_count": 57, "id": "ae41e8de", "metadata": {}, "outputs": [{"data": {"text/plain": ["0"]}, "execution_count": 57, "metadata": {}, "output_type": "execute_result"}], "source": ["np.argmax(prediction.predictions[0])"]}, {"cell_type": "markdown", "id": "764b7378", "metadata": {}, "source": ["### Get Predictions: REST"]}, {"cell_type": "code", "execution_count": 58, "id": "98b5f76a", "metadata": {}, "outputs": [], "source": ["with open(f'{DIR}/request.json','w') as file:\n", "    file.write(json.dumps({\"instances\": [newob]}))"]}, {"cell_type": "code", "execution_count": 59, "id": "b208b88b", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["{\n", "  \"predictions\": [\n", "    [\n", "      0.999251425,\n", "      0.000748594233\n", "    ]\n", "  ],\n", "  \"deployedModelId\": \"2291109553401495552\",\n", "  \"model\": \"projects/764015827198/locations/us-central1/models/4415994938915815424\",\n", "  \"modelDisplayName\": \"05f_fraud_20211203120602\"\n", "}\n"]}], "source": ["!curl -X POST \\\n", "-H \"Authorization: Bearer \"$(gcloud auth application-default print-access-token) \\\n", "-H \"Content-Type: application/json; charset=utf-8\" \\\n", "-d @{DIR}/request.json \\\n", "https://{REGION}-aiplatform.googleapis.com/v1/{endpoint.resource_name}:predict"]}, {"cell_type": "markdown", "id": "1ec58e66", "metadata": {}, "source": ["### Get Predictions: gcloud (CLI)"]}, {"cell_type": "code", "execution_count": 61, "id": "714b7eeb", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Using endpoint [https://us-central1-prediction-aiplatform.googleapis.com/]\n", "[[0.999251425, 0.000748594233]]\n"]}], "source": ["!gcloud beta ai endpoints predict {endpoint.name.rsplit('/',1)[-1]} --region={REGION} --json-request={DIR}/request.json"]}, {"cell_type": "markdown", "id": "a3521490", "metadata": {}, "source": ["### Data For Prediction: Multiple Entities Served by Vertex AI > Features (Feature Store)"]}, {"cell_type": "code", "execution_count": 62, "id": "3c29d220", "metadata": {}, "outputs": [{"data": {"text/plain": ["0    07fdced0-3837-47a1-9526-64d74ad9b113\n", "1    7c1f61ba-7586-414e-ba8a-1c4385d59933\n", "2    a3046c87-cee5-40fd-9302-4d230b823246\n", "3    bc3f2800-a4bb-4077-b017-f55f03c4f00c\n", "4    181ee7ba-e3b0-4a34-9ebd-50ce98c4a350\n", "5    b8c61520-7288-444b-a9de-b270d8d490c0\n", "6    908af6f9-d158-4d14-8335-7a6136223842\n", "7    6830e0dd-1bcb-4d6f-a98b-32de7256922e\n", "8    52225b6b-bd48-42ce-8e33-de5260735d3e\n", "9    fe1e9c5f-ff54-4d0d-ba28-120b31739944\n", "Name: transaction_id, dtype: object"]}, "execution_count": 62, "metadata": {}, "output_type": "execute_result"}], "source": ["unique_id['transaction_id']"]}, {"cell_type": "code", "execution_count": 63, "id": "fd6f0210", "metadata": {}, "outputs": [], "source": ["multi_feature_values = clients['fs_olserve'].streaming_read_feature_values(\n", "    request = aiplatform.gapic.StreamingReadFeatureValuesRequest(\n", "        entity_type = clients['fs'].entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, ENTITYTYPE_ID),\n", "        entity_ids = unique_id['transaction_id'],\n", "        feature_selector = aiplatform.gapic.FeatureSelector(id_matcher=aiplatform.gapic.IdMatcher(ids=['*'])),\n", "    )\n", ")"]}, {"cell_type": "code", "execution_count": 64, "id": "7f290928", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\n", "[]\n", "07fdced0-3837-47a1-9526-64d74ad9b113\n", "[-0.16033206009858803, 1.07306279044003, 0.035042952311969, -1.01262330333815, -0.026793043261075, 0.46149546559133703, -0.26507148271390196, 0.930348759049177, 0.171860617271716, -0.0452503674451511, -1.3826303643001698, -0.219909538224871, 0.0, 0.79650222865, -0.13296614676554902, 1.03689341495792, 1.0288470541571502, 0.404511129977238, 1.75438907383858, 0.336310126389912, 0.223515261314392, -0.0131252979106899, -0.25821805726084, -0.515894450159377, -0.7461822853235991, 0.5056034354709891, 0.0, 0.0485385294222296, -0.0376919034755461, -0.864056157762422]\n", "181ee7ba-e3b0-4a34-9ebd-50ce98c4a350\n", "[-0.39688716246813394, -0.394433988274438, 0.840231685849197, -0.22200848597894002, 0.665004897142369, -0.32351742819282503, -0.215222472550955, -1.1383662635890401, 0.5173173344698, 0.33057526374587104, -1.70199813974206, 2.2992821926639704, 0.0, -1.07329808435329, -0.680326782136273, -1.9495022115490201, 0.580970913097478, -1.25422907791396, -1.05806617698414, -0.538856016137508, 0.987074949284179, 0.132588989433156, 0.0800428926455472, 1.62791798267705, -0.4493931202105229, -0.614217827746417, 0.0, -0.514307098835192, 0.0065389389428550295, -0.73587254715049]\n", "52225b6b-bd48-42ce-8e33-de5260735d3e\n", "[0.156713333633212, -0.30255064074533505, 0.134118706330542, 0.310724851475457, 0.185773627357189, 0.100368421241235, -0.0107537981401818, -0.150545038623116, -0.474369832844814, -0.0410618744424278, -1.21434292162426, -0.7366786717580849, 0.0, 2.5506185352868904, 0.534444943982974, 0.982331081565978, 0.894521558966853, 0.242358406683679, -0.54698141727112, -0.272969581815174, -0.342386214675367, -0.17742684072069, 0.577581061940322, -0.0287790651569902, 0.622559950704842, -0.42013215470204496, 0.0, -0.119993258401215, -0.153850452687124, -0.48871848161014103]\n", "6830e0dd-1bcb-4d6f-a98b-32de7256922e\n", "[1.1407253936306498, 1.8776310265378702, -0.286038215429858, 0.20515412824383603, 1.2338477284390101, -0.23328125142790002, -0.193509038180537, -0.829882783956245, 0.0330868308568062, -0.18146945078446303, 0.6645482293270161, 0.897299673442182, 0.0, -1.5561571462548602, -1.08956046295809, 0.000470363775904592, -0.0627383090152664, 0.0352038942118656, 4.376486579701259, 1.85126093600225, -2.9478697002376606, 0.0165136506440117, 0.104445951640915, 0.5105714457311821, -1.91109912141651, 0.187423368255943, 0.0, 0.158538183274642, 0.0101411456085235, -1.2904661570695801]\n", "7c1f61ba-7586-414e-ba8a-1c4385d59933\n", "[0.30365194692519204, 1.8785625824583, 0.404327302655266, -0.5927119626227779, 0.0207846148454918, 1.1361537148886698, -0.252052834604407, 4.7094420694559105, 0.140099630383241, 0.0799983909242375, -0.34191364978890193, 2.50784747862852, 0.0, -1.62111326086483, -0.39575460848474897, -0.9456849350021259, 0.155684269122107, -0.00791141657117167, 2.90881274895554, -0.47143351886327, -0.121323711423773, -0.0538916954122686, 0.7075376605487209, -0.830625971166057, -1.38868063809564, 1.31141279262982, 0.0, 0.121098219316919, 0.0163749473296811, -1.53138788137664]\n", "908af6f9-d158-4d14-8335-7a6136223842\n", "[-0.536997697085724, 0.16221000911199, -0.0568732450417888, -0.6675023503444639, 0.761508773520782, -0.315149615406484, -0.211784733569759, -1.24065758033991, -0.240304959800418, -0.060938406030107, -0.130827566172047, 0.6422733119076961, 0.0, 0.0227393110918034, -0.0361434738592031, -0.896240834538622, -0.150329423162144, 0.17639523027959, -0.812837175669926, -0.260741813414885, 0.31285410894734, 0.0321052815325069, -0.00498628379144568, 1.17533737412722, 0.00823754758968418, -0.796593271880895, 0.0, -0.0848278339833236, 0.0282714266447215, -0.309104206384283]\n", "a3046c87-cee5-40fd-9302-4d230b823246\n", "[0.45441738029237705, -1.1697438529655901, -0.546607316105327, 0.146580100118293, 0.462877516201437, -0.5228039864503761, 0.280455162260437, -0.9184125358364649, -0.283979160054203, -0.418851207093186, 2.01864097036614, 1.01281739344023, 0.0, 1.5875791232791698, 1.38824716825758, 2.19601645201901, 0.556660650458781, -1.8245379583769097, -1.2525598943701, -0.46823157631313006, 1.0829956899864, -0.0717727232169664, -0.0212689058377677, 0.817632446548305, -0.34289260829876, -0.022449424083296, 0.0, -0.0421438220210644, -0.191722193195158, -1.0514387886508298]\n", "b8c61520-7288-444b-a9de-b270d8d490c0\n", "[0.78423849536753, 1.94932712656623, 0.8010802674239279, 1.03669675665933, -0.21870411222908298, 0.145507258307357, -0.29085780954907103, -0.078090124045179, -0.100162581552513, 0.24275963392967, -0.679084276463019, -0.454008735258874, 0.0, -0.31695004886246303, 0.785384911860287, 0.353160750658373, -0.558117334023412, 0.42230632029644705, 1.1483425625158998, -1.11245428244518, 0.330855230394838, -0.0458835223245063, -0.372846620594346, -0.640465480756055, -0.345400671972679, 0.42008765030006295, 0.0, 0.10082688931491, 0.0489442950816396, 0.18185977219790803]\n", "bc3f2800-a4bb-4077-b017-f55f03c4f00c\n", "[-0.23838425367294, -2.72840326782693, 0.645385804966846, -0.0750034302197183, -2.3403458240324397, 0.7575607985377439, 0.9931286417631279, -0.807293618157396, -0.10495621368074, 0.534355612834265, 0.279370547429626, 3.0147275333541, 0.0, 2.55103925646015, -1.27929660307995, 1.6508061965178302, 1.35578103042191, 0.5679201977414929, 2.6230919017667595, 0.387930742700835, 0.391247152952605, 0.125386264532128, -0.21278387747682698, -2.28662122963225, 0.522880333043809, 0.812136013094236, 0.0, 0.37013947079620396, -0.0530626045501945, 1.5466051558266398]\n", "fe1e9c5f-ff54-4d0d-ba28-120b31739944\n", "[-1.6631251702336698, -2.11737070828966, -0.0374106817378044, -1.08276742737981, 1.4664699091607298, -2.36417696234317, 1.58795335104536, -1.5025097941923002, 0.0285701369550623, -0.974642340800635, -1.8681030673985701, 1.61548617126212, 0.0, 0.928210116822821, 3.59536713983271, 0.38347914388931603, -0.383285189532696, -1.27872585158, -1.6669072333352, -1.1633461849812, -2.2337159459971203, -1.17896793619143, 0.351925322379141, 2.97811163413337, -0.331580275959789, 4.6882701193857095, 0.0, -0.45373115404748604, -0.383632360663778, 0.313742953967041]\n"]}], "source": ["for i in multi_feature_values:\n", "    print(i.entity_view.entity_id)\n", "    print(list(item.value.double_value for item in i.entity_view.data))"]}, {"cell_type": "markdown", "id": "fd35d198", "metadata": {}, "source": ["### Data For Training: Batch (For training or large scale prediction)"]}, {"cell_type": "code", "execution_count": 65, "id": "26631faf", "metadata": {}, "outputs": [], "source": ["# get current timestamp (protobuf3 is seconds since ephoch (1970))\n", "timestamp = Timestamp()\n", "timestamp.GetCurrentTime()\n", "\n", "# adjust timestamp to 2 days ago: 60*60*24*4\n", "newtimestamp = Timestamp(seconds = timestamp.seconds - 60*60*24*2, nanos = timestamp.nanos)\n", "\n", "batch_request = aiplatform.gapic.ExportFeatureValuesRequest(\n", "    entity_type = clients['fs'].entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, ENTITYTYPE_ID),\n", "    snapshot_export = aiplatform.gapic.ExportFeatureValuesRequest.SnapshotExport(snapshot_time = Timestamp(seconds=newtimestamp.seconds)),\n", "    destination = aiplatform.gapic.FeatureValueDestination(bigquery_destination = aiplatform.gapic.BigQueryDestination(output_uri = f'bq://{PROJECT_ID}.{DATANAME}.{DATANAME}_fs_training')),\n", "    feature_selector = aiplatform.gapic.FeatureSelector(id_matcher = aiplatform.gapic.IdMatcher(ids = ['*']))\n", ")"]}, {"cell_type": "code", "execution_count": 66, "id": "80b68bfe", "metadata": {}, "outputs": [], "source": ["batchjob = clients['fs'].export_feature_values(batch_request)"]}, {"cell_type": "code", "execution_count": 67, "id": "be026f46", "metadata": {}, "outputs": [{"data": {"text/plain": []}, "execution_count": 67, "metadata": {}, "output_type": "execute_result"}], "source": ["batchjob.result()"]}, {"cell_type": "markdown", "id": "3bfaadd0", "metadata": {}, "source": ["By Adjusting the `snapshot_time` to 2 days ago, the batch_request creates a BigQuery table that has all the orginal rows, 1 per entity, but the features are null for 20% of the rows.  This is because the features were loaded with `feature_time_field = \"update_time\"` and `update_time` was set to a random day between today and 10 days ago."]}, {"cell_type": "code", "execution_count": 69, "id": "9a8345bc", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>Null_Rows</th>\n", "      <th>counts</th>\n", "      <th>Percentage</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>0</th>\n", "      <td>True</td>\n", "      <td>56980</td>\n", "      <td>20.006531</td>\n", "    </tr>\n", "    <tr>\n", "      <th>1</th>\n", "      <td>False</td>\n", "      <td>227827</td>\n", "      <td>79.993469</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["   Null_Rows  counts  Percentage\n", "0       True   56980   20.006531\n", "1      False  227827   79.993469"]}, "execution_count": 69, "metadata": {}, "output_type": "execute_result"}], "source": ["query = f\"\"\"\n", "SELECT \n", "    CASE WHEN {list(newob.keys())[0]} is not null then False ELSE True END as Null_Rows, \n", "    count(*) as counts,\n", "    100*count(*) / (sum(count(*)) OVER()) as Percentage\n", "FROM {PROJECT_ID}.{DATANAME}.{DATANAME}_fs_training\n", "GROUP BY Null_Rows\n", "\"\"\"\n", "clients['bq'].query(query = query).to_dataframe()"]}, {"cell_type": "markdown", "id": "13fef91a", "metadata": {}, "source": ["---\n", "## Remove Resources\n", "see notebook \"99 - Cleanup\""]}], "metadata": {"environment": {"kernel": "python3", "name": "tf2-gpu.2-3.m94", "type": "gcloud", "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m94"}, "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.12"}}, "nbformat": 4, "nbformat_minor": 5}