v0.7.0 - refactored webhook module

Author: Justintime50

.coveragerc

@@ -2,3 +2,6 @@
 exclude_lines =
     if __name__ == '__main__':
     main()
+
+    # TODO: TEMPORARY, Remove when replaced with a long-term solution
+    retrieve_pipeline

CHANGELOG.md

@@ -1,5 +1,14 @@
 # CHANGELOG
 
+## v0.7.0 (2020-10-26)
+
+* Refactored the `webhook` module and added unit tests
+* Refactored webhook logic to return appropriate JSON messages and HTTP status codes
+* Fixed bugs where Harvey would blow up when no JSON, malformed JSON, or other webhook details weren't correct
+* Fixed a bug that would not catch when a bad pipeline name would be given
+* Various other bug fixes and improvements
+* Added basic tests to API routes
+
 ## v0.6.0 (2020-10-25)
 
 * Added unit tests for the `container` module and refactored various code there

README.md

@@ -19,14 +19,15 @@ Harvey was born because Rancher has too much overhead and GitLab is too RAM hung
 
 ## How it Works
 
-Harvey receives a webhook from GitHub, pulls in the changes, tests them, builds them, then deploys them. When all is said and done - we'll even provide you a message to say "job's done".
+Harvey receives a webhook from GitHub, pulls in the changes, tests them, builds them, then deploys them. If you have Slack enabled, Harvey will send you the pipeline summary.
 
-1. GitHub webhook fires to your self-hosted endpoint stating that a new commit hit an enabled repo
-    - Your server pulls in the new changes for that repo via Git
-1. Next we test your code based on the criteria provided
-1. Then we build your docker image locally
-1. Next we spin up the new docker container and tear down the old one once it's up and running, making downtime barely a blip
-1. Finally we shoot off a message to notify users the build was a success or not
+1. GitHub webhook fires and is received by Harvey stating that a new commit hit an enabled repo
+    * Harvey pulls in the new changes for that repo via Git
+1. Next Harvey tests your code based on the criteria provided
+1. Then Harvey builds your docker image locally
+1. Next Harvey spins up the new docker container and tears down the old one once it's up and running
+1. Then Harvey will run a container healthcheck to ensure your container is up and running and didn't exit on startup
+1. Finally, (if enabled) Harvey will shoot off a message to notify users the build was a success or not
 
 Harvey has lightweight testing functionality which is configurable via shell scripts. Harvey builds a unique self-isolated Docker container to test your code and returns the logs from your tests when finished.
 
@@ -48,11 +49,9 @@ make help
 
 1. Install Docker & login
 1. Ensure you've added your ssh key to the ssh agent: `ssh-add` followed by your password
-1. Enable logging (see below)
-1. Setup enviornment variables in `.env`
-1. Add webhooks for all your repositories you want to use Harvey with (point them to `http://example.com:5000/pipelines/start`, send the payload as JSON)
-
-**NOTE:** It is not recommended to use Harvey alongside other CI/CD or Docker orchestration platforms on the same machine.
+1. Enable logging (see `Logs` below)
+1. Setup enviornment variables as needed
+1. Enable GitHub webhooks for all your repositories you want to use Harvey with (point them to `http://example.com:5000/pipelines/start`, send the payload as JSON)
 
 ### Logs
 
@@ -70,9 +69,17 @@ The [following](https://docs.docker.com/config/containers/logging/json-file/#usa
 
 ## Usage
 
+```bash
+# Run locally
+make run
+
+# Run in production
+harvey-ci
+```
+
 Find the full [docs here](docs/README.md). 
 
-Harvey's entrypoint is a webhook (eg: `127.0.0.1:5000/pipelines/start`). Pass GitHub data to Harvey and let it do the rest. If you'd like to simulate a GitHub webhook, simply pass a JSON file like the following example to the Harvey webhook endpoint (ensure you have an environment variable `MODE=test` to bypass the need for a webhook secret):
+Harvey's entrypoint (eg: `127.0.0.1:5000/pipelines/start`) accepts a webhook from GitHub. If you'd like to simulate a GitHub webhook, simply pass a JSON file like the following example to the Harvey webhook endpoint (ensure you have an environment variable `MODE=test` to bypass the need for a webhook secret and GitHub headers):
 
 ```javascript
 {
@@ -101,12 +108,6 @@ Environment Variables:
     DEBUG           Whether the Flask API will run in debug mode or not
 ```
 
-### Start API Server (for Webhook)
-
-```bash
-make run
-```
-
 ### Example Python Functions
 
 See `examples.py` for all available methods of each class. Almost every usage example is contained in this file.

examples/examples.py

@@ -90,9 +90,10 @@
 
 """API Entrypoint (Webhook)"""
 with open('examples/git_webhook.json', 'r') as file:
+    data = json.load(file)
     request = requests.post(
         'http://127.0.0.1:5000/pipelines/start',
-        data=file,
+        json=data,
         headers=harvey.Global.JSON_HEADERS
     )
     print(request.json())

harvey/app.py

@@ -13,16 +13,14 @@
 PORT = os.getenv('PORT', '5000')
 DEBUG = os.getenv('DEBUG', 'True')
 
-
-# TODO: Move all of the logic out of this file and into Harvey itself
-# This file should only route requests to the proper functions
+# TODO: Add authentication to each endpoint
 
 
 @API.route('/pipelines/start', methods=['POST'])
 def start_pipeline():
     """Start a pipeline based on webhook data
     """
-    return Webhook.parse_webhook(request=request, target=Webhook.start_pipeline)
+    return Webhook.parse_webhook(request=request, use_compose=False)
 
 
 # @API.route('/pipelines/stop', methods=['POST'])
@@ -35,7 +33,7 @@ def start_pipeline_compose():
     """Start a pipeline based on webhook data
     But build from compose file.
     """
-    return Webhook.parse_webhook(request=request, target=Webhook.start_pipeline_compose)
+    return Webhook.parse_webhook(request=request, use_compose=True)
 
 
 # @API.route('/pipelines/stop/compose', methods=['POST'])

harvey/git.py

@@ -9,6 +9,7 @@ class Git():
     def update_git_repo(cls, webhook):
         """Clone or pull repo using Git depending on if it exists or not
         """
+        # TODO: Fail fast if a repo doesn't exist
         project_path = os.path.join(
             Global.PROJECTS_PATH, Global.repo_full_name(webhook)
         )

harvey/globals.py

@@ -3,7 +3,7 @@ class Global():
     """
     DOCKER_VERSION = 'v1.40'  # Docker API version
     # TODO: Figure out how to sync this version number with the one in `setup.py`
-    HARVEY_VERSION = '0.5.0'  # Harvey release
+    HARVEY_VERSION = '0.7.0'  # Harvey release
     PROJECTS_PATH = 'projects'
     PROJECTS_LOG_PATH = 'logs/projects'
     HARVEY_LOG_PATH = 'logs/harvey'

harvey/utils.py

@@ -14,6 +14,7 @@ def kill(cls, final_output, webhook):
         if os.getenv('SLACK'):
             Message.send_slack_message(final_output)
         sys.exit()
+        return True
 
     @classmethod
     def success(cls, final_output, webhook):
@@ -22,6 +23,7 @@ def success(cls, final_output, webhook):
         Logs.generate_logs(final_output, webhook)
         if os.getenv('SLACK'):
             Message.send_slack_message(final_output)
+        return True
 
 
 class Logs():

harvey/webhook.py

@@ -3,112 +3,120 @@
 from datetime import datetime
 import hmac
 import hashlib
-from flask import abort
 from threading import Thread
 from harvey.pipeline import Pipeline
 from harvey.git import Git
 from harvey.globals import Global
 from harvey.utils import Utils
 
 
+WEBHOOK_SECRET = os.getenv('WEBHOOK_SECRET')
+APP_MODE = os.getenv('MODE')
+
+
 class Webhook():
     @classmethod
-    def init(cls, webhook):
+    def initialize_pipeline(cls, webhook):
         """Initiate the logic for webhooks and pull the project
         """
         start_time = datetime.now()
-        preamble = f'Running Harvey v{Global.HARVEY_VERSION}\n' + \
-            f'Pipeline Started: {start_time}'
+        preamble = f'Running Harvey v{Global.HARVEY_VERSION}\nPipeline Started: {start_time}'
         pipeline_id = f'Pipeline ID: {Global.repo_commit_id(webhook)}\n'
         print(preamble)
-        git_message = (f'New commit by: {Global.repo_commit_author(webhook)}. \
-            \nCommit made on repo: {Global.repo_full_name(webhook)}.')
+        git_message = (f'New commit by: {Global.repo_commit_author(webhook)}.'
+                       f'\nCommit made on repo: {Global.repo_full_name(webhook)}.')
         git = Git.update_git_repo(webhook)
+        config = cls.open_project_config(webhook)
+        execution_time = f'Startup execution time: {datetime.now() - start_time}\n'
+        output = (f'{preamble}\n{pipeline_id}Configuration:\n{json.dumps(config, indent=4)}'
+                  f'\n\n{git_message}\n{git}\n{execution_time}')
+        print(execution_time)
+        return config, output
 
-        # Open the project's config file to assign pipeline variables
+    @classmethod
+    def open_project_config(cls, webhook):
+        """Open the project's config file to assign pipeline variables
+        """
+        # TODO: Add the ability to configure projects on the Harvey side
+        # instead of only from within a JSON file in the repo
         try:
-            filename = os.path.join(Global.PROJECTS_PATH, Global.repo_full_name(webhook),
-                                    'harvey.json')
+            filename = os.path.join(
+                Global.PROJECTS_PATH, Global.repo_full_name(webhook), 'harvey.json'
+            )
             with open(filename, 'r') as file:
                 config = json.loads(file.read())
                 print(json.dumps(config, indent=4))
-        except FileNotFoundError as fnf_error:
-            final_output = f'Error: Harvey could not find "harvey.json" file in \
-                {Global.repo_full_name(webhook)}.'
-            print(fnf_error)
+            return config
+        except FileNotFoundError:
+            final_output = f'Error: Harvey could not find a "harvey.json" file in {Global.repo_full_name(webhook)}.'
+            print(final_output)
             Utils.kill(final_output, webhook)
 
-        execution_time = f'Startup execution time: {datetime.now() - start_time}\n'
-        output = f'{preamble}\n{pipeline_id}Configuration:\n{json.dumps(config, indent=4)}' + \
-            f'\n\n{git_message}\n{git}\n{execution_time}'
-        print(execution_time)
-
-        return config, output
-
     @classmethod
-    def parse_webhook(cls, request, target):
+    def parse_webhook(cls, request, use_compose):
         """Initiate details to receive a webhook
         """
-        data = request.data
+        success = False
+        message = 'Server-side error.'
+        status_code = 500
+        data = request.json if request.json else None
         signature = request.headers.get('X-Hub-Signature')
-        parsed_data = json.loads(data)  # TODO: Is this necessary?
-        # TODO: Allow the user to configure whatever branch they'd like to pull from
-        if parsed_data['ref'] in ['refs/heads/master', 'refs/heads/main']:
-            if os.getenv('MODE') == 'test':
-                Thread(target=target, args=(parsed_data,)).start()
-                return "200"
-            if cls.decode_webhook(data, signature):
-                Thread(target=target, args=(parsed_data,)).start()
-                return "200"
-            return abort(403)
-        return abort(500, 'Harvey can only pull from the "master" or "main" branch.')
+
+        if request.data and data:
+            # TODO: Allow the user to configure whatever branch they'd like to pull from
+            if data['ref'] in ['refs/heads/master', 'refs/heads/main']:
+                if APP_MODE == 'test' or cls.decode_webhook(data, signature):
+                    Thread(target=cls.start_pipeline, args=(data, use_compose,)).start()
+                    message = f'Started pipeline for {data["repository"]["name"]}'
+                    status_code = 200
+                    success = True
+                if APP_MODE != 'test' and not cls.decode_webhook(data, signature):
+                    message = 'The X-Hub-Signature did not match the WEBHOOK_SECRET.'
+                    status_code = 403
+            else:
+                message = 'Harvey can only pull from the "master" or "main" branch of a repo.'
+                status_code = 422
+        else:
+            message = 'Malformed or missing JSON data in webhook.'
+            status_code = 422
+        response = {
+            'success': success,
+            'message': message,
+        }, status_code
+        return response
 
     @classmethod
     def decode_webhook(cls, data, signature):
         """Decode a webhook's secret key
         """
-        secret = bytes(os.getenv('WEBHOOK_SECRET'), 'UTF-8')
-        mac = hmac.new(secret, msg=data, digestmod=hashlib.sha1)
-        return hmac.compare_digest('sha1=' + mac.hexdigest(), signature)
+        if signature:
+            secret = bytes(WEBHOOK_SECRET, 'UTF-8')
+            mac = hmac.new(secret, msg=data, digestmod=hashlib.sha1)
+            return hmac.compare_digest('sha1=' + mac.hexdigest(), signature)
+        else:
+            return False
 
     @classmethod
-    def start_pipeline(cls, webhook):
+    def start_pipeline(cls, webhook, use_compose=False):
         """Receive a webhook and spin up a pipeline based on the config
         """
-        webhook_config, webhook_output = Webhook.init(webhook)
+        webhook_config, webhook_output = cls.initialize_pipeline(webhook)
+        webhook_pipeline = webhook_config['pipeline']
 
-        if webhook_config['pipeline'] == 'test':
-            pipeline = Pipeline.test(webhook_config, webhook, webhook_output)
-        elif webhook_config['pipeline'] == 'deploy':
-            pipeline = Pipeline.deploy(webhook_config, webhook, webhook_output)
-        elif webhook_config['pipeline'] == 'full':
-            pipeline = Pipeline.full(webhook_config, webhook, webhook_output)
-        elif webhook_config['pipeline'] == 'pull':
+        if webhook_pipeline == 'pull':
             pipeline = Utils.success(webhook_output, webhook)
-        elif not webhook_config['pipeline']:
-            final_output = webhook_output + '\nError: Harvey could not run, \
-                there was no pipeline specified.'
-            Utils.kill(final_output, webhook)
-
-        return pipeline
-
-    @classmethod
-    def start_pipeline_compose(cls, webhook):
-        """Receive a webhook and spin up a pipeline based on the config
-        """
-        webhook_config, webhook_output = Webhook.init(webhook)
-
-        if webhook_config['pipeline'] == 'test':
+        elif webhook_pipeline == 'test':
             pipeline = Pipeline.test(webhook_config, webhook, webhook_output)
-        elif webhook_config['pipeline'] == 'deploy':
-            pipeline = Pipeline.deploy_compose(webhook_config, webhook, webhook_output)
-        elif webhook_config['pipeline'] == 'full':
+        elif webhook_pipeline == 'deploy' and use_compose is False:
+            pipeline = Pipeline.deploy(webhook_config, webhook, webhook_output)
+        elif webhook_pipeline == 'full' and use_compose is True:
             pipeline = Pipeline.full_compose(webhook_config, webhook, webhook_output)
-        elif webhook_config['pipeline'] == 'pull':
-            pipeline = Utils.success(webhook_output, webhook)
-        elif not webhook_config['pipeline']:
-            final_output = webhook_output + '\nError: Harvey could not run, \
-                there was no pipeline specified.'
-            Utils.kill(final_output, webhook)
+        elif webhook_pipeline == 'deploy' and use_compose is True:
+            pipeline = Pipeline.deploy_compose(webhook_config, webhook, webhook_output)
+        elif webhook_pipeline == 'full' and use_compose is False:
+            pipeline = Pipeline.full(webhook_config, webhook, webhook_output)
+        else:
+            final_output = webhook_output + '\nError: Harvey could not run, there was no acceptable pipeline specified.'
+            pipeline = Utils.kill(final_output, webhook)
 
         return pipeline

setup.py

@@ -13,7 +13,7 @@
 
 setuptools.setup(
     name='harvey-ci',
-    version='0.6.0',
+    version='0.7.0',
     description='Your personal CI/CD and Docker orchestration platform.',
     long_description=long_description,
     long_description_content_type="text/markdown",