Enhancement/support deployment as a function (#63)

* Bump rsa from 4.6 to 4.7 Bumps [rsa](https://github.com/sybrenstuvel/python-rsa) from 4.6 to 4.7. - [Release notes](https://github.com/sybrenstuvel/python-rsa/releases) - [Changelog](https://github.com/sybrenstuvel/python-rsa/blob/main/CHANGELOG.md) - [Commits](sybrenstuvel/python-rsa@version-4.6...version-4.7) Signed-off-by: dependabot[bot] <[email protected]> * Support deployment as a Google Cloud Function * Update on readme to add Google Cloud Function section * refactoring in handle function * update the readme with command to test the function * review (PR#63): use actual return message and status code as response * review (PR#63): update pipfile and pipfile.lock * review (PR#63): improve readability and change the structure of README.md * review (PR#63): add a message submission at the end of execution * review (PR#63): remove a line with debug statement from README.md * review (PR#63): apply requested changes on main.py * review (PR#63): apply requested changes on chat.py and slack.py * review (PR#63): apply requested changes on chat.py and readme.md * review (PR#63): fix minor typos Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
kieras · Feb 15, 2022 · 9ab62a6 · 9ab62a6
1 parent 63f359e
commit 9ab62a6
Show file tree

Hide file tree

Showing 8 changed files with 775 additions and 411 deletions.
diff --git a/.gcloudignore b/.gcloudignore
@@ -0,0 +1,17 @@
+# This file specifies files that are *not* uploaded to Google Cloud
+# using gcloud. It follows the same syntax as .gitignore, with the addition of
+# "#!include" directives (which insert the entries of the given .gitignore-style
+# file at that point).
+#
+# For more information, run:
+#   $ gcloud topic gcloudignore
+#
+.gcloudignore
+# If you would like to upload your .git directory, .gitignore file or files
+# from your .gitignore file, remove the corresponding line
+# below:
+.git
+.gitignore
+
+node_modules
+#!include:.gitignore
diff --git a/Pipfile b/Pipfile
@@ -4,20 +4,21 @@ url = "https://pypi.org/simple"
 name = "pypi"
 
 [packages]
-google-api-python-client = "==1.7.11"
-google-cloud-bigquery = "==1.11.1"
-grpcio = "==1.19.0"
+google-api-python-client = "==2.37.0"
+google-cloud-bigquery = "==2.32.0"
+grpcio = "==1.43.0"
 oauth2client = "==3.0.0"
-confuse = "==0.5.0"
+confuse = "==1.7.0"
 slackclient = "==1.3.1"
-requests = "==2.22.0"
+requests = "==2.27.1"
+functions-framework= "==3.*"
 
 [requires]
 python_version = "3.8"
 
 [dev-packages]
-pytest = "==3.9.3"
-pytest-mock = "==1.10.0"
+pytest = "==7.0.0"
+pytest-mock = "==3.7.0"
 coverage = "*"
 pylint = "*"
 rope = "*"
diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/README.md b/README.md
@@ -2,8 +2,18 @@
 
 Helps your team control infra costs by pointing potential unused 'zombie' projects.
 
+If you use the Slack integration, the Owners of the Projects receive messages like this one:
+
+![Example Slack message](example-slack-message.png?raw=true "Example Slack message")
+
+If you use the Google Chat integration, messages similar to this one are sent to your Chat Space:
+
+![Example Chat message](example-chat-message.png?raw=true "Example Chat message")
+
 ## Installation
 
+### Dependencies and Config
+
 Clone this repository.
 
 Install Python dependencies and prepare the config file:
@@ -13,9 +23,24 @@ pipenv install --ignore-pipfile --dev
 cp example-config.yaml config.yaml
 ```
 
-## Usage
+Change `config.yaml` to fit your needs.
+
+### Usage as a CLI command
+
+
+If you want to execute the program using your GCP user credentials, use the commands below:
+```bash
+gcloud auth application-default login
+gcloud config set project PROJECT_ID
+```
 
-Change `config.yaml` to you needs and run the following command:
+Instead of using your GCP user credentials, you can use a Service Account Key. In this case, use the following command:
+
+```bash
+export GOOGLE_APPLICATION_CREDENTIALS='service-account-key.json'
+```
+
+Run the following command:
 
 ```bash
 pipenv run python main.py
@@ -30,14 +55,19 @@ pipenv run python main.py
 
 See the `example-bigquery-billing-costs-view.sql` file of an example query to use for adding Project cost information from your [Billing Export](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) data in BigQuery.
 
-## Example message on Slack
-
-Owners of the Projects receive messages like this one:
+### Usage as a Google Cloud Function
 
-![Example Slack message](example-slack-message.png?raw=true "Example Slack message")
+If you want to deploy the code as a Cloud Function, run the following command:
 
-## Example message on Chat
+```bash
+gcloud functions deploy zombie-project-watcher \
+    --entry-point=http_request \
+    --runtime python38 \
+    --trigger-http
+```
 
-Messages like this one are sent to your Chat room:
+If you need to run or debug the  Google Cloud Function locally, run the command:
 
-![Example Chat message](example-chat-message.png?raw=true "Example Chat message")
+```bash
+functions-framework --target http_request --debug
+```
diff --git a/chat.py b/chat.py
@@ -4,6 +4,7 @@
 import requests
 from pprint import pformat
 from config import CONFIG
+from datetime import datetime as dt
 
 logger = logging.getLogger(__name__)
 
@@ -18,7 +19,23 @@
 COST_MIN_TO_NOTIFY = CONFIG['chat']['cost_min_to_notify'].get(float)
 
 
+
+def send_message(message):
+    logger.info('Sending Chat message to webhook %s:\n%s', WEBHOOK_URL, message)
+    if PRINT_ONLY:
+        return
+    message_headers = {'Content-Type': 'application/json; charset=UTF-8'}
+    message_data = {
+        'text': message
+    }
+    message_data_json = json.dumps(message_data, indent=2)
+    response = requests.post(
+        WEBHOOK_URL, data=message_data_json, headers=message_headers)
+    if response.status_code != 200:
+        logger.error('Error sending message to Chat. Error: %s, Response: %s, Webhook: %s', response.status_code, pformat(response.text), WEBHOOK_URL)
+
 def send_messages_to_chat(projects_by_owner):
+    number_of_notified_projects = 0
     if not CHAT_ACTIVATED:
         logger.info('Chat integration is not active.')
         return
@@ -37,24 +54,21 @@ def send_messages_to_chat(projects_by_owner):
             emoji_codepoint = chr(int(COST_ALERT_EMOJI, base = 16))
             cost_alert_emoji = "{} ".format(emoji_codepoint)
             if cost <= COST_MIN_TO_NOTIFY:
-                logger.debug('- `{}/{}`, will not be in the message, due to its cost being lower than the minimum warning value'.format(owner, project_id))
+                logger.debug('- `{}/{}`, will not be in the message, due to its cost being lower than the minimum warning value'\
+                    .format(owner, project_id))
             else:
                 if cost > COST_ALERT_THRESHOLD:
                     emoji = ' ' + cost_alert_emoji
                 send_message_to_this_owner = True
-                message += "- `{}/{}` created `{} days ago`, costing *`{}`* {}.{}\n".format(org, project_id, created_days_ago, cost, currency, emoji)
+                message += "`{}/{}` created `{} days ago`, costing *`{}`* {}.{}\n"\
+                    .format(org, project_id, created_days_ago, cost, currency, emoji)
+                number_of_notified_projects = number_of_notified_projects + 1
         message += "\nIf these projects are not being used anymore, please consider `deleting them to reduce infra costs` and clutter."
 
         if send_message_to_this_owner:
-            logger.info('Sending Chat message to webhook %s:\n%s', WEBHOOK_URL, message)
-            if PRINT_ONLY:
-                return
-            message_headers = {'Content-Type': 'application/json; charset=UTF-8'}
-            message_data = {
-                'text': message
-            }
-            message_data_json = json.dumps(message_data, indent=2)
-            response = requests.post(
-                WEBHOOK_URL, data=message_data_json, headers=message_headers)
-            if response.status_code != 200:
-                logger.error('Error sending message to Chat. Error: %s, Response: %s, Webhook: %s', response.status_code, pformat(response.text), WEBHOOK_URL)
+            send_message(message)
+
+    today_weekday=dt.today().strftime('%A')
+    final_of_execution_message = f'Happy {today_weekday}!!! \nZombie Projects Watcher ran successfully \
+        and found {number_of_notified_projects} projects with costs higher than the defined notification threshold of ${COST_MIN_TO_NOTIFY}.'
+    send_message(final_of_execution_message)
diff --git a/main.py b/main.py
@@ -3,9 +3,11 @@
 import json
 from pprint import pformat
 from datetime import datetime as dt
+import traceback as tb
 from googleapiclient import discovery
 
 from logging_config import setup_logging
+import functions_framework
 
 setup_logging()
 logger = logging.getLogger(__name__)
@@ -24,7 +26,7 @@
     filter_whitelisted_users
 )
 from billing import query_billing_info
-from slack import send_messages
+from slack import send_messages_to_slack
 from chat import send_messages_to_chat
 
 
@@ -45,6 +47,30 @@
 DEBUG_GROUPED_BY_OWNERS = CONFIG['debug']['grouped_by_owners'].get(bool)
 DEBUG_FILTERED_BY_ORGS = CONFIG['debug']['filtered_by_org'].get(bool)
 
+@functions_framework.http
+def http_request(request):
+    now = dt.now()
+    date_value = dt.strftime(now, "%Y-%m-%dT%H:%M:%S.%fZ")
+    message = ''
+    status_code = ''
+    try:
+        message, status_code =  main()
+    except Exception as err:
+        logging.exception(err)
+        exception_message =  tb.format_exc().splitlines()
+        message = exception_message[-1].capitalize()
+        message = message.replace('<',' ')
+        message = message.replace('>',' ')
+        message = 'An error occurred. Details: ' + message
+        for item in message.split():
+            if item.isnumeric():
+               status_code = item
+            else:
+                status_code = 500
+    finally:
+        message = (message + ' on ' +  date_value + '!')
+    return message, status_code
+
 def main():
     client = _get_resource_manager_client()
     client_v1 = _get_resource_manager_client_v1() 
@@ -115,7 +141,7 @@ def main():
 
     if SLACK_ACTIVATED:
         logger.info('Sending Slack messages.')
-        send_messages(projects_by_owner)
+        send_messages_to_slack(projects_by_owner)
         logger.info('All messages sent.')
     else:
         logger.info('Slack integration is not active.')
@@ -128,6 +154,11 @@ def main():
         logger.info('Chat integration is not active.')
 
     logger.info('Happy Friday! :)')
+
+    response_message = "Success "
+    response_code = 200
+
+    return response_message, response_code
 
 
 def _get_projects(client):

diff --git a/requirements.txt b/requirements.txt
@@ -4,4 +4,4 @@ google-cloud-bigquery==1.11.1
 confuse==0.5.0
 slackclient==1.3.1
 requests==2.22.0
-
+functions-framework==3.*
diff --git a/slack.py b/slack.py
@@ -3,6 +3,7 @@
 from pprint import pformat
 from slackclient import SlackClient
 from config import CONFIG
+from datetime import datetime as dt
 
 logger = logging.getLogger(__name__)
 
@@ -24,7 +25,23 @@
 COST_MIN_TO_NOTIFY = CONFIG['slack']['cost_min_to_notify'].get(float)
 
 
-def send_messages(projects_by_owner):
+def prepare_message(slack_user, message):
+    slack_channel = "@{}".format(slack_user)
+    if SEND_TO_TEAM_CHANNEL:
+        slack_channel = "#{}".format(TEAM_CHANNEL)
+    resp = _send_message(slack_channel, message)
+    if resp and not resp.get('ok'):
+        if resp.get('error') == 'channel_not_found':
+            logger.error('Error: %s, Channel: %s', resp.get('error'), slack_channel)
+            if TEAM_CHANNEL_FALLBACK:
+                resp = _send_message("#{}".format(TEAM_CHANNEL), message)
+                if resp and not resp.get('ok'):
+                    logger.error('Error in fallback to team channel: %s, Channel: %s, Response: %s', resp.get('error'), slack_channel, pformat(resp))
+            else:
+                logger.error('Error: %s, Channel: %s, Response: %s', resp.get('error'), slack_channel, pformat(resp))
+
+def send_messages_to_slack(projects_by_owner):
+    number_of_notified_projects = 0
     if not SLACK_ACTIVATED:
         logger.info('Slack integration is not active.')
         return
@@ -47,22 +64,16 @@ def send_messages(projects_by_owner):
                     emoji = ' ' + COST_ALERT_EMOJI
                 send_message_to_this_owner = True
                 message += "- `{}/{}` created `{} days ago`, costing *`{}`* {}.{}\n".format(org, project_id, created_days_ago, cost, currency, emoji)
+                number_of_notified_projects = number_of_notified_projects + 1       
         message += "If these projects are not being used anymore, please consider `deleting them to reduce infra costs` and clutter. :rip:"
+
         if send_message_to_this_owner:
-            slack_channel = "@{}".format(slack_user)
-            if SEND_TO_TEAM_CHANNEL:
-                slack_channel = "#{}".format(TEAM_CHANNEL)
-            resp = _send_message(slack_channel, message)
-            if resp and not resp.get('ok'):
-                if resp.get('error') == 'channel_not_found':
-                    logger.error('Error: %s, Channel: %s', resp.get('error'), slack_channel)
-                    if TEAM_CHANNEL_FALLBACK:
-                        resp = _send_message("#{}".format(TEAM_CHANNEL), message)
-                        if resp and not resp.get('ok'):
-                            logger.error('Error in fallback to team channel: %s, Channel: %s, Response: %s', resp.get('error'), slack_channel, pformat(resp))
-                    else:
-                        logger.error('Error: %s, Channel: %s, Response: %s', resp.get('error'), slack_channel, pformat(resp))
+            prepare_message(slack_user, message)
 
+    today_weekday=dt.today().strftime('%A')
+    final_of_execution_message = f'Happy {today_weekday}!!! \nZombie Projects Watcher ran successfully \
+        and found {number_of_notified_projects} projects with costs higher than the defined notification threshold ${COST_MIN_TO_NOTIFY}.'           
+    prepare_message(TEAM_CHANNEL, final_of_execution_message)
 
 def _send_message(channel, message):
     if TEST_USER: