Origin/feature generate examples (#1)

* Added generate examples script and trusted dashboard table

Author: colin-roy-ehri

Diff for: .gitignore

@@ -8,6 +8,7 @@ terraform.tfstate*
 *.tfstate
 .venv
 node_modules
+looker.ini
 
 .vertex_cf_auth_token
 dist

Diff for: explore-assistant-backend/terraform/bigquery_examples.tf

@@ -42,6 +42,31 @@ resource "google_bigquery_job" "create_explore_assistant_examples_table" {
   }
 }
 
+resource "google_bigquery_job" "create_explore_assistant_examples_table" {
+  job_id = "create_explore_assistant_examples_table-${formatdate("YYYYMMDDhhmmss", timestamp())}"
+  query {
+    query              = <<EOF
+    CREATE OR REPLACE TABLE `${google_bigquery_dataset.dataset.dataset_id}.trusted_dashboards` (
+        explore_id STRING OPTIONS (description = 'Explore id of the explore to pull examples for in a format of -> lookml_model:lookml_explore'),
+        lookml STRING OPTIONS (description = 'LookML dashboard copy for authoritative dashboard(s) based on the given explore_id.')
+    )
+  EOF  
+    create_disposition = ""
+    write_disposition  = ""
+    allow_large_results = false
+    flatten_results = false
+    maximum_billing_tier = 0
+    schema_update_options = [ ]
+    use_legacy_sql = false
+  }
+
+  location = var.deployment_region
+  depends_on = [ time_sleep.wait_after_apis_activate]
+
+  lifecycle {
+    ignore_changes  = [query, job_id]
+  }
+}
 
 resource "google_bigquery_job" "create_explore_assistant_refinement_examples_table" {
   job_id = "create_explore_assistant_refinement_examples_table-${formatdate("YYYYMMDDhhmmss", timestamp())}"

Diff for: explore-assistant-examples/README.md

@@ -1,6 +1,9 @@
 # BigQuery Data Loader
 
-This script facilitates the loading of JSON data into Google BigQuery while managing data freshness by ensuring existing rows related to an `explore_id` are deleted before new data is inserted. The script employs a temporary table mechanism to circumvent limitations related to immediate updates or deletions in BigQuery's streaming buffer.
+This folder includes two scripts. 
+The first script (generate_examples.py) will create input/output example pairs for training or one-shot use. These are based on the top queries for a chosen model and explore. The script will also create measure and dimension lists for later use.
+
+The loading script (load_examples.py) facilitates the loading of JSON data into Google BigQuery while managing data freshness by ensuring existing rows related to an `explore_id` are deleted before new data is inserted. The script employs a temporary table mechanism to circumvent limitations related to immediate updates or deletions in BigQuery's streaming buffer.
 
 ## Prerequisites
 
@@ -10,6 +13,11 @@ Before you run this script, you need to ensure that your environment is set up w
 2. **Google Cloud SDK** - Install and configure the Google Cloud SDK (gcloud).
 3. **BigQuery API Access** - Ensure that the BigQuery API is enabled in your Google Cloud project.
 4. **Google Cloud Authentication** - Set up authentication by downloading a service account key and setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to that key file.
+5. **Looker SDK Initialization** - Set up authentication for the Looker SDK by specifying these variables:
+`LOOKERSDK_BASE_URL`	A URL like https://my.looker.com:19999. No default value.
+`LOOKERSDK_CLIENT_ID` API credentials client_id. This and client_secret must be provided in some fashion to the Node SDK, or no calls to the API will be authorized. No default value.
+`LOOKERSDK_CLIENT_SECRET` API credentials client_secret. No default value.
+
 
 ## Setup
 
@@ -23,7 +31,7 @@ pip install -r requirements.txt
 ```
 ## Usage
 
-### Script Parameters
+### Loading Script Parameters
 
 The script accepts several command line arguments to specify the details required for loading data into BigQuery:
 
@@ -33,7 +41,7 @@ The script accepts several command line arguments to specify the details require
 - `--explore_id`: **Required.** A unique identifier for the dataset rows related to a specific use case or query (used in deletion and insertion).
 - `--json_file`: The path to the JSON file containing the data to be loaded. Defaults to `examples.json`.
 
-### Running the Script
+### Running the Loading Script
 
  **Before Running:** make sure the .env file in this directory is updated to reference your project_id, dataset_id and explore_id
 
@@ -79,9 +87,19 @@ chmod +x update_examples.sh
 ./update_examples.sh
 ```
 
+
+Load the trusted dashboard lookml
+
+```bash
+ python load_examples.py --project_id YOUR_PROJECT_ID --explore_id YOUR_EXPLORE_ID --table_id trusted_dashboards --json_file trusted_dashboards.lkml --format text --column_name lookml
+=======
+chmod +x update_examples.sh
+```
+
+
 ### Description
 
-This Python script is designed to manage data uploads from a JSON file into a Google BigQuery table, particularly focusing on scenarios where specific entries identified by an `explore_id` need to be refreshed or updated in the dataset.
+The load_examples Python script is designed to manage data uploads from a JSON file into a Google BigQuery table, particularly focusing on scenarios where specific entries identified by an `explore_id` need to be refreshed or updated in the dataset.
 
 1. **Command Line Interface (CLI)**:
    - The script uses `argparse` to define and handle command line inputs that specify the Google Cloud project, dataset, and table details, as well as the path to the JSON data file.
@@ -100,3 +118,23 @@ This Python script is designed to manage data uploads from a JSON file into a Go
 
 6. **Error Handling**:
    - Throughout the data deletion and insertion processes, the script checks for and reports any errors that occur. This is vital for debugging and ensuring data integrity.
+
+### Generation Script Parameters
+The generate_examples.py script accepts several command line arguments to specify the details required for generating example files:
+
+- `--model`: Required. Looker model name.
+- `--explore`: Required. Looker explore name.
+- `--project_id`: Required. Google Cloud project ID.
+- `--location`: Required. Google Cloud location.
+
+# Running the Generation Script
+The generate_examples.py script fetches information about an explores' fields and top queries. It calls Gemini to generate sample questions that could be answered by the top queries. These can be tuned or used directly as examples to upload to the Explore Assistant.
+
+```bash
+python generate_examples.py --model YOUR_MODEL_NAME --explore YOUR_EXPLORE_NAME --project_id YOUR_GCP_PROJECT_ID --location YOUR_GCP_LOCATION
+```
+   
+If desired, you can directly upload the files after generation by using the --chain_load argument.
+```bash
+python generate_examples.py --model YOUR_MODEL_NAME --explore YOUR_EXPLORE_NAME --project_id YOUR_GCP_PROJECT_ID --location YOUR_GCP_LOCATION --chain_load
+```