Merge pull request #338 from envato/add-parameter-files-and-parameter…

…s-options-to-stack-definition

Add parameter_files and parameters configuration options to stack definition

CHANGELOG.md

@@ -8,7 +8,16 @@ The format is based on [Keep a Changelog], and this project adheres to
 [Keep a Changelog]: https://keepachangelog.com/en/1.0.0/
 [Semantic Versioning]: https://semver.org/spec/v2.0.0.html
 
-## [Unreleased]
+## [2.7.0] - 2020-06-15
+
+### Added
+
+- `parameters_dir` is now configurable to match the existing `template_dir`.
+- `parameter_files` configures an array of parameter files relative to
+  `parameters_dir` that will be used instead of automatic parameter file globs
+  based on region and stack name.
+- `parameters` configures stack parameters directly on the stack definition
+  rather than requiring an external parameter file.
 
 ### Fixed
 

README.md

@@ -123,6 +123,7 @@ stack_defaults:
 ```
 
 Additional files can be configured to be uploaded to S3 alongside the templates:
+
 ```yaml
 stacks:
   production:
@@ -131,6 +132,7 @@ stacks:
       files:
         - userdata.sh
 ```
+
 ## Directories
 
 - `templates` - CloudFormation, SparkleFormation or CfnDsl templates.
@@ -155,7 +157,8 @@ template_compilers:
 
 ## Parameters
 
-Parameters are loaded from multiple YAML files, merged from the following lookup paths from bottom to top:
+By default, parameters are loaded from multiple YAML files, merged from the
+following lookup paths from bottom to top:
 
 - parameters/[stack_name].yaml
 - parameters/[stack_name].yml
@@ -170,6 +173,30 @@ A simple parameter file could look like this:
 key_name: myapp-us-east-1
 ```
 
+Alternatively, a `parameter_files` array can be defined to explicitly list
+parameter files that will be loaded. If `parameter_files` are defined, the
+automatic search locations will not be used.
+
+```yaml
+parameters_dir: parameters # the default
+stacks:
+  us-east-1:
+    my-app:
+      parameter_files:
+        - my-app.yml # parameters/my-app.yml
+```
+
+Parameters can also be defined inline with stack definitions:
+
+```yaml
+stacks:
+  us-east-1:
+    my-app:
+      parameters:
+        VpcId:
+          stack_output: my-vpc/VpcId
+```
+
 ### Compile Time Parameters
 
 Compile time parameters can be used for [SparkleFormation](http://www.sparkleformation.io) templates. It conforms and

features/apply_with_explicit_parameter_files.feature

@@ -0,0 +1,65 @@
+Feature: Apply command with explicit parameter files
+
+  Background:
+    Given a file named "stack_master.yml" with:
+      """
+      stack_defaults:
+        tags:
+          Application: myapp
+      stacks:
+        us-east-1:
+          myapp-web:
+            template: myapp.rb
+            parameter_files:
+            - myapp-web-parameters.yml
+      """
+    And a file named "parameters/us-east-1/myapp-web.yml" with:
+      """
+      Color: blue
+      """
+    And a file named "parameters/myapp-web-parameters.yml" with:
+      """
+      KeyName: my-key
+      Color: red
+      """
+    And a directory named "templates"
+    And a file named "templates/myapp.rb" with:
+      """
+      SparkleFormation.new(:myapp) do
+        description "Test template"
+
+        parameters.key_name do
+          description 'Key name'
+          type 'String'
+        end
+
+        parameters.color do
+          description 'Color'
+          type 'String'
+        end
+
+        resources.instance do
+          type 'AWS::EC2::Instance'
+          properties do
+            image_id 'ami-0080e4c5bc078760e'
+            instance_type 't2.micro'
+          end
+        end
+      end
+      """
+
+  Scenario: Run apply and create stack with explicit parameter files
+    Given I stub the following stack events:
+      | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type              | timestamp           |
+      | 1        | 1        | myapp-web  | myapp-web           | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 |
+    When I run `stack_master apply us-east-1 myapp-web --trace`
+    Then the output should match /2020-10-29 00:00:00 (\+|\-)[0-9]{4} myapp-web AWS::CloudFormation::Stack CREATE_COMPLETE/
+    And the output should contain all of these lines:
+      | Stack diff:          |
+      | +    "Instance": {   |
+      | Parameters diff:     |
+      | KeyName: my-key      |
+      | Proposed change set: |
+    And the output should not contain "Color: blue"
+    And the output should contain "Color: red"
+    And the exit status should be 0

features/apply_with_s3.feature

@@ -69,7 +69,7 @@ Feature: Apply command
       | Parameters diff:                                                               |
       | KeyName: my-key                                                                |
     And the output should match /2020-10-29 00:00:00 (\+|\-)[0-9]{4} myapp-vpc AWS::CloudFormation::Stack CREATE_COMPLETE/
-    And an S3 file in bucket "my-bucket" with key "cfn_templates/my-app/myapp_vpc.json" exists with content:
+    And an S3 file in bucket "my-bucket" with key "cfn_templates/my-app/myapp_vpc.json" exists with JSON content:
       """
       {
         "Description": "Test template",

features/apply_with_stack_definition_parameters.feature

@@ -0,0 +1,46 @@
+Feature: Apply command with stack definition parameters
+
+  Background:
+    Given a file named "stack_master.yml" with:
+      """
+      stacks:
+        us-east-1:
+          myapp_web:
+            template: myapp.rb
+            parameters:
+              KeyName: my-key
+      """
+    And a directory named "templates"
+    And a file named "templates/myapp.rb" with:
+      """
+      SparkleFormation.new(:myapp) do
+        description "Test template"
+
+        parameters.key_name do
+          description 'Key name'
+          type 'String'
+        end
+
+        resources.instance do
+          type 'AWS::EC2::Instance'
+          properties do
+            image_id 'ami-0080e4c5bc078760e'
+            instance_type 't2.micro'
+          end
+        end
+      end
+      """
+
+  Scenario: Run apply with parameters contained in
+    Given I stub the following stack events:
+      | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type              | timestamp           |
+      | 1        | 1        | myapp-web  | myapp-web           | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 |
+    When I run `stack_master apply us-east-1 myapp-web --trace`
+    Then the output should match /2020-10-29 00:00:00 (\+|\-)[0-9]{4} myapp-web AWS::CloudFormation::Stack CREATE_COMPLETE/
+    And the output should contain all of these lines:
+      | Stack diff:          |
+      | +    "Instance": {   |
+      | Parameters diff:     |
+      | KeyName: my-key      |
+      | Proposed change set: |
+    And the exit status should be 0

features/step_definitions/stack_steps.rb

@@ -69,7 +69,13 @@ def extract_hash_from_kv_string(string)
   allow(StackMaster.cloud_formation_driver.class).to receive(:new).and_return(StackMaster.cloud_formation_driver)
 end
 
-When(/^an S3 file in bucket "([^"]*)" with key "([^"]*)" exists with content:$/) do |bucket, key, body|
+Then(/^an S3 file in bucket "([^"]*)" with key "([^"]*)" exists with content:$/) do |bucket, key, body|
   file = StackMaster.s3_driver.find_file(bucket: bucket, object_key: key)
   expect(file).to eq body
 end
+
+Then(/^an S3 file in bucket "([^"]*)" with key "([^"]*)" exists with JSON content:$/) do |bucket, key, body|
+  file = StackMaster.s3_driver.find_file(bucket: bucket, object_key: key)
+  parsed_file = JSON.parse(file)
+  expect(parsed_file).to eq JSON.parse(body)
+end

lib/stack_master/commands/tidy.rb

@@ -12,7 +12,7 @@ def perform
         parameter_files = Set.new(find_parameter_files())
 
         status = @config.stacks.each do |stack_definition|
-          parameter_files.subtract(stack_definition.parameter_files)
+          parameter_files.subtract(stack_definition.parameter_files_from_globs)
           template = File.absolute_path(stack_definition.template_file_path)
 
           if template

lib/stack_master/config.rb

@@ -17,6 +17,7 @@ def self.load!(config_file = 'stack_master.yml')
     attr_accessor :stacks,
                   :base_dir,
                   :template_dir,
+                  :parameters_dir,
                   :stack_defaults,
                   :region_defaults,
                   :region_aliases,
@@ -39,6 +40,7 @@ def initialize(config, base_dir)
       @config = config
       @base_dir = base_dir
       @template_dir = config.fetch('template_dir', nil)
+      @parameters_dir = config.fetch('parameters_dir', nil)
       @stack_defaults = config.fetch('stack_defaults', {})
       @region_aliases = Utils.underscore_keys_to_hyphen(config.fetch('region_aliases', {}))
       @region_to_aliases = @region_aliases.inject({}) do |hash, (key, value)|
@@ -115,6 +117,7 @@ def load_stacks(stacks)
             'stack_name' => stack_name,
             'base_dir' => @base_dir,
             'template_dir' => @template_dir,
+            'parameters_dir' => @parameters_dir,
             'additional_parameter_lookup_dirs' => @region_to_aliases[region])
           stack_attributes['allowed_accounts'] = attributes['allowed_accounts'] if attributes['allowed_accounts']
           @stacks << StackDefinition.new(stack_attributes)

lib/stack_master/parameter_loader.rb

@@ -5,18 +5,17 @@ class ParameterLoader
 
     COMPILE_TIME_PARAMETERS_KEY = 'compile_time_parameters'
 
-    def self.load(parameter_files)
+    def self.load(parameter_files: [], parameters: {})
       StackMaster.debug 'Searching for parameter files...'
-      parameter_files.reduce({template_parameters: {}, compile_time_parameters: {}}) do |hash, file_name|
-        parameters = load_parameters(file_name)
+      all_parameters = parameter_files.map { |file_name| load_parameters(file_name) } + [parameters]
+      all_parameters.reduce({template_parameters: {}, compile_time_parameters: {}}) do |hash, parameters|
         template_parameters = create_template_parameters(parameters)
         compile_time_parameters = create_compile_time_parameters(parameters)
 
         merge_and_camelize(hash[:template_parameters], template_parameters)
         merge_and_camelize(hash[:compile_time_parameters], compile_time_parameters)
         hash
       end
-
     end
 
     private

lib/stack_master/parameter_validator.rb

@@ -9,15 +9,14 @@ def initialize(stack:, stack_definition:)
 
     def error_message
       return nil unless missing_parameters?
-      message = "Empty/blank parameters detected. Please provide values for these parameters:"
+      message = "Empty/blank parameters detected. Please provide values for these parameters:\n"
       missing_parameters.each do |parameter_name|
-        message << "\n - #{parameter_name}"
+        message << " - #{parameter_name}\n"
       end
-      message << "\nParameters will be read from files matching the following globs:"
-      base_dir = Pathname.new(@stack_definition.base_dir)
-      @stack_definition.parameter_file_globs.each do |glob|
-        parameter_file = Pathname.new(glob).relative_path_from(base_dir)
-        message << "\n - #{parameter_file}"
+      if @stack_definition.parameter_files.empty?
+        message << message_for_parameter_globs
+      else
+        message << message_for_parameter_files
       end
       message
     end
@@ -28,6 +27,24 @@ def missing_parameters?
 
     private
 
+    def message_for_parameter_files
+      "Parameters are configured to be read from the following files:\n".tap do |message|
+        @stack_definition.parameter_files.each do |parameter_file|
+          message << " - #{parameter_file}\n"
+        end
+      end
+    end
+
+    def message_for_parameter_globs
+      "Parameters will be read from files matching the following globs:\n".tap do |message|
+        base_dir = Pathname.new(@stack_definition.base_dir)
+        @stack_definition.parameter_file_globs.each do |glob|
+          parameter_file = Pathname.new(glob).relative_path_from(base_dir)
+          message << " - #{parameter_file}\n"
+        end
+      end
+    end
+
     def missing_parameters
       @missing_parameters ||=
         @stack.parameters_with_defaults.select { |_key, value| value.nil? }.keys

lib/stack_master/stack.rb

@@ -56,7 +56,7 @@ def self.find(region, stack_name)
     end
 
     def self.generate(stack_definition, config)
-      parameter_hash = ParameterLoader.load(stack_definition.parameter_files)
+      parameter_hash = ParameterLoader.load(parameter_files: stack_definition.all_parameter_files, parameters: stack_definition.parameters)
       template_parameters = ParameterResolver.resolve(config, stack_definition, parameter_hash[:template_parameters])
       compile_time_parameters = ParameterResolver.resolve(config, stack_definition, parameter_hash[:compile_time_parameters])
       template_body = TemplateCompiler.compile(config, stack_definition.compiler, stack_definition.template_dir, stack_definition.template, compile_time_parameters, stack_definition.compiler_options)
@@ -76,7 +76,7 @@ def self.generate(stack_definition, config)
     end
 
     def self.generate_without_parameters(stack_definition, config)
-      parameter_hash = ParameterLoader.load(stack_definition.parameter_files)
+      parameter_hash = ParameterLoader.load(parameter_files: stack_definition.all_parameter_files, parameters: stack_definition.parameters)
       compile_time_parameters = ParameterResolver.resolve(config, stack_definition, parameter_hash[:compile_time_parameters])
       template_body = TemplateCompiler.compile(config, stack_definition.compiler, stack_definition.template_dir, stack_definition.template, compile_time_parameters, stack_definition.compiler_options)
       template_format = TemplateUtils.identify_template_format(template_body)