diff --git a/CHANGELOG.md b/CHANGELOG.md index a122740e..35de9059 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,16 @@ The format is based on [Keep a Changelog], and this project adheres to [Keep a Changelog]: https://keepachangelog.com/en/1.0.0/ [Semantic Versioning]: https://semver.org/spec/v2.0.0.html -## [Unreleased] +## [2.7.0] - 2020-06-15 + +### Added + +- `parameters_dir` is now configurable to match the existing `template_dir`. +- `parameter_files` configures an array of parameter files relative to + `parameters_dir` that will be used instead of automatic parameter file globs + based on region and stack name. +- `parameters` configures stack parameters directly on the stack definition + rather than requiring an external parameter file. ### Fixed diff --git a/README.md b/README.md index e451d2c0..dd90d731 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,7 @@ stack_defaults: ``` Additional files can be configured to be uploaded to S3 alongside the templates: + ```yaml stacks: production: @@ -131,6 +132,7 @@ stacks: files: - userdata.sh ``` + ## Directories - `templates` - CloudFormation, SparkleFormation or CfnDsl templates. @@ -155,7 +157,8 @@ template_compilers: ## Parameters -Parameters are loaded from multiple YAML files, merged from the following lookup paths from bottom to top: +By default, parameters are loaded from multiple YAML files, merged from the +following lookup paths from bottom to top: - parameters/[stack_name].yaml - parameters/[stack_name].yml @@ -170,6 +173,30 @@ A simple parameter file could look like this: key_name: myapp-us-east-1 ``` +Alternatively, a `parameter_files` array can be defined to explicitly list +parameter files that will be loaded. If `parameter_files` are defined, the +automatic search locations will not be used. + +```yaml +parameters_dir: parameters # the default +stacks: + us-east-1: + my-app: + parameter_files: + - my-app.yml # parameters/my-app.yml +``` + +Parameters can also be defined inline with stack definitions: + +```yaml +stacks: + us-east-1: + my-app: + parameters: + VpcId: + stack_output: my-vpc/VpcId +``` + ### Compile Time Parameters Compile time parameters can be used for [SparkleFormation](http://www.sparkleformation.io) templates. It conforms and diff --git a/features/apply_with_explicit_parameter_files.feature b/features/apply_with_explicit_parameter_files.feature new file mode 100644 index 00000000..4679ea60 --- /dev/null +++ b/features/apply_with_explicit_parameter_files.feature @@ -0,0 +1,65 @@ +Feature: Apply command with explicit parameter files + + Background: + Given a file named "stack_master.yml" with: + """ + stack_defaults: + tags: + Application: myapp + stacks: + us-east-1: + myapp-web: + template: myapp.rb + parameter_files: + - myapp-web-parameters.yml + """ + And a file named "parameters/us-east-1/myapp-web.yml" with: + """ + Color: blue + """ + And a file named "parameters/myapp-web-parameters.yml" with: + """ + KeyName: my-key + Color: red + """ + And a directory named "templates" + And a file named "templates/myapp.rb" with: + """ + SparkleFormation.new(:myapp) do + description "Test template" + + parameters.key_name do + description 'Key name' + type 'String' + end + + parameters.color do + description 'Color' + type 'String' + end + + resources.instance do + type 'AWS::EC2::Instance' + properties do + image_id 'ami-0080e4c5bc078760e' + instance_type 't2.micro' + end + end + end + """ + + Scenario: Run apply and create stack with explicit parameter files + Given I stub the following stack events: + | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type | timestamp | + | 1 | 1 | myapp-web | myapp-web | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 | + When I run `stack_master apply us-east-1 myapp-web --trace` + Then the output should match /2020-10-29 00:00:00 (\+|\-)[0-9]{4} myapp-web AWS::CloudFormation::Stack CREATE_COMPLETE/ + And the output should contain all of these lines: + | Stack diff: | + | + "Instance": { | + | Parameters diff: | + | KeyName: my-key | + | Proposed change set: | + And the output should not contain "Color: blue" + And the output should contain "Color: red" + And the exit status should be 0 diff --git a/features/apply_with_s3.feature b/features/apply_with_s3.feature index 5f957b25..f6683a0f 100644 --- a/features/apply_with_s3.feature +++ b/features/apply_with_s3.feature @@ -69,7 +69,7 @@ Feature: Apply command | Parameters diff: | | KeyName: my-key | And the output should match /2020-10-29 00:00:00 (\+|\-)[0-9]{4} myapp-vpc AWS::CloudFormation::Stack CREATE_COMPLETE/ - And an S3 file in bucket "my-bucket" with key "cfn_templates/my-app/myapp_vpc.json" exists with content: + And an S3 file in bucket "my-bucket" with key "cfn_templates/my-app/myapp_vpc.json" exists with JSON content: """ { "Description": "Test template", diff --git a/features/apply_with_stack_definition_parameters.feature b/features/apply_with_stack_definition_parameters.feature new file mode 100644 index 00000000..32d9f9e8 --- /dev/null +++ b/features/apply_with_stack_definition_parameters.feature @@ -0,0 +1,46 @@ +Feature: Apply command with stack definition parameters + + Background: + Given a file named "stack_master.yml" with: + """ + stacks: + us-east-1: + myapp_web: + template: myapp.rb + parameters: + KeyName: my-key + """ + And a directory named "templates" + And a file named "templates/myapp.rb" with: + """ + SparkleFormation.new(:myapp) do + description "Test template" + + parameters.key_name do + description 'Key name' + type 'String' + end + + resources.instance do + type 'AWS::EC2::Instance' + properties do + image_id 'ami-0080e4c5bc078760e' + instance_type 't2.micro' + end + end + end + """ + + Scenario: Run apply with parameters contained in + Given I stub the following stack events: + | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type | timestamp | + | 1 | 1 | myapp-web | myapp-web | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 | + When I run `stack_master apply us-east-1 myapp-web --trace` + Then the output should match /2020-10-29 00:00:00 (\+|\-)[0-9]{4} myapp-web AWS::CloudFormation::Stack CREATE_COMPLETE/ + And the output should contain all of these lines: + | Stack diff: | + | + "Instance": { | + | Parameters diff: | + | KeyName: my-key | + | Proposed change set: | + And the exit status should be 0 diff --git a/features/step_definitions/stack_steps.rb b/features/step_definitions/stack_steps.rb index c55a6849..d40540bb 100644 --- a/features/step_definitions/stack_steps.rb +++ b/features/step_definitions/stack_steps.rb @@ -69,7 +69,13 @@ def extract_hash_from_kv_string(string) allow(StackMaster.cloud_formation_driver.class).to receive(:new).and_return(StackMaster.cloud_formation_driver) end -When(/^an S3 file in bucket "([^"]*)" with key "([^"]*)" exists with content:$/) do |bucket, key, body| +Then(/^an S3 file in bucket "([^"]*)" with key "([^"]*)" exists with content:$/) do |bucket, key, body| file = StackMaster.s3_driver.find_file(bucket: bucket, object_key: key) expect(file).to eq body end + +Then(/^an S3 file in bucket "([^"]*)" with key "([^"]*)" exists with JSON content:$/) do |bucket, key, body| + file = StackMaster.s3_driver.find_file(bucket: bucket, object_key: key) + parsed_file = JSON.parse(file) + expect(parsed_file).to eq JSON.parse(body) +end diff --git a/lib/stack_master/commands/tidy.rb b/lib/stack_master/commands/tidy.rb index d7fc63b1..24b440e8 100644 --- a/lib/stack_master/commands/tidy.rb +++ b/lib/stack_master/commands/tidy.rb @@ -12,7 +12,7 @@ def perform parameter_files = Set.new(find_parameter_files()) status = @config.stacks.each do |stack_definition| - parameter_files.subtract(stack_definition.parameter_files) + parameter_files.subtract(stack_definition.parameter_files_from_globs) template = File.absolute_path(stack_definition.template_file_path) if template diff --git a/lib/stack_master/config.rb b/lib/stack_master/config.rb index f85e797f..3426ed52 100644 --- a/lib/stack_master/config.rb +++ b/lib/stack_master/config.rb @@ -17,6 +17,7 @@ def self.load!(config_file = 'stack_master.yml') attr_accessor :stacks, :base_dir, :template_dir, + :parameters_dir, :stack_defaults, :region_defaults, :region_aliases, @@ -39,6 +40,7 @@ def initialize(config, base_dir) @config = config @base_dir = base_dir @template_dir = config.fetch('template_dir', nil) + @parameters_dir = config.fetch('parameters_dir', nil) @stack_defaults = config.fetch('stack_defaults', {}) @region_aliases = Utils.underscore_keys_to_hyphen(config.fetch('region_aliases', {})) @region_to_aliases = @region_aliases.inject({}) do |hash, (key, value)| @@ -115,6 +117,7 @@ def load_stacks(stacks) 'stack_name' => stack_name, 'base_dir' => @base_dir, 'template_dir' => @template_dir, + 'parameters_dir' => @parameters_dir, 'additional_parameter_lookup_dirs' => @region_to_aliases[region]) stack_attributes['allowed_accounts'] = attributes['allowed_accounts'] if attributes['allowed_accounts'] @stacks << StackDefinition.new(stack_attributes) diff --git a/lib/stack_master/parameter_loader.rb b/lib/stack_master/parameter_loader.rb index 452be95a..64dacfce 100644 --- a/lib/stack_master/parameter_loader.rb +++ b/lib/stack_master/parameter_loader.rb @@ -5,10 +5,10 @@ class ParameterLoader COMPILE_TIME_PARAMETERS_KEY = 'compile_time_parameters' - def self.load(parameter_files) + def self.load(parameter_files: [], parameters: {}) StackMaster.debug 'Searching for parameter files...' - parameter_files.reduce({template_parameters: {}, compile_time_parameters: {}}) do |hash, file_name| - parameters = load_parameters(file_name) + all_parameters = parameter_files.map { |file_name| load_parameters(file_name) } + [parameters] + all_parameters.reduce({template_parameters: {}, compile_time_parameters: {}}) do |hash, parameters| template_parameters = create_template_parameters(parameters) compile_time_parameters = create_compile_time_parameters(parameters) @@ -16,7 +16,6 @@ def self.load(parameter_files) merge_and_camelize(hash[:compile_time_parameters], compile_time_parameters) hash end - end private diff --git a/lib/stack_master/parameter_validator.rb b/lib/stack_master/parameter_validator.rb index 1c405d15..b7b36b06 100644 --- a/lib/stack_master/parameter_validator.rb +++ b/lib/stack_master/parameter_validator.rb @@ -9,15 +9,14 @@ def initialize(stack:, stack_definition:) def error_message return nil unless missing_parameters? - message = "Empty/blank parameters detected. Please provide values for these parameters:" + message = "Empty/blank parameters detected. Please provide values for these parameters:\n" missing_parameters.each do |parameter_name| - message << "\n - #{parameter_name}" + message << " - #{parameter_name}\n" end - message << "\nParameters will be read from files matching the following globs:" - base_dir = Pathname.new(@stack_definition.base_dir) - @stack_definition.parameter_file_globs.each do |glob| - parameter_file = Pathname.new(glob).relative_path_from(base_dir) - message << "\n - #{parameter_file}" + if @stack_definition.parameter_files.empty? + message << message_for_parameter_globs + else + message << message_for_parameter_files end message end @@ -28,6 +27,24 @@ def missing_parameters? private + def message_for_parameter_files + "Parameters are configured to be read from the following files:\n".tap do |message| + @stack_definition.parameter_files.each do |parameter_file| + message << " - #{parameter_file}\n" + end + end + end + + def message_for_parameter_globs + "Parameters will be read from files matching the following globs:\n".tap do |message| + base_dir = Pathname.new(@stack_definition.base_dir) + @stack_definition.parameter_file_globs.each do |glob| + parameter_file = Pathname.new(glob).relative_path_from(base_dir) + message << " - #{parameter_file}\n" + end + end + end + def missing_parameters @missing_parameters ||= @stack.parameters_with_defaults.select { |_key, value| value.nil? }.keys diff --git a/lib/stack_master/stack.rb b/lib/stack_master/stack.rb index 29f12025..99357dad 100644 --- a/lib/stack_master/stack.rb +++ b/lib/stack_master/stack.rb @@ -56,7 +56,7 @@ def self.find(region, stack_name) end def self.generate(stack_definition, config) - parameter_hash = ParameterLoader.load(stack_definition.parameter_files) + parameter_hash = ParameterLoader.load(parameter_files: stack_definition.all_parameter_files, parameters: stack_definition.parameters) template_parameters = ParameterResolver.resolve(config, stack_definition, parameter_hash[:template_parameters]) compile_time_parameters = ParameterResolver.resolve(config, stack_definition, parameter_hash[:compile_time_parameters]) template_body = TemplateCompiler.compile(config, stack_definition.compiler, stack_definition.template_dir, stack_definition.template, compile_time_parameters, stack_definition.compiler_options) @@ -76,7 +76,7 @@ def self.generate(stack_definition, config) end def self.generate_without_parameters(stack_definition, config) - parameter_hash = ParameterLoader.load(stack_definition.parameter_files) + parameter_hash = ParameterLoader.load(parameter_files: stack_definition.all_parameter_files, parameters: stack_definition.parameters) compile_time_parameters = ParameterResolver.resolve(config, stack_definition, parameter_hash[:compile_time_parameters]) template_body = TemplateCompiler.compile(config, stack_definition.compiler, stack_definition.template_dir, stack_definition.template, compile_time_parameters, stack_definition.compiler_options) template_format = TemplateUtils.identify_template_format(template_body) diff --git a/lib/stack_master/stack_definition.rb b/lib/stack_master/stack_definition.rb index 587afb77..939e0c4c 100644 --- a/lib/stack_master/stack_definition.rb +++ b/lib/stack_master/stack_definition.rb @@ -16,7 +16,10 @@ class StackDefinition :additional_parameter_lookup_dirs, :s3, :files, - :compiler_options + :compiler_options, + :parameters_dir, + :parameters, + :parameter_files attr_reader :compiler @@ -32,8 +35,12 @@ def initialize(attributes = {}) @compiler = nil super @additional_parameter_lookup_dirs ||= [] + @base_dir ||= "" @template_dir ||= File.join(@base_dir, 'templates') + @parameters_dir ||= File.join(@base_dir, 'parameters') @allowed_accounts = Array(@allowed_accounts) + @parameters ||= {} + @parameter_files ||= [] end def ==(other) @@ -56,13 +63,9 @@ def ==(other) @compiler_options == other.compiler_options end - def compiler=(compiler) - @compiler = compiler.&to_sym - end - def template_file_path return unless template - File.expand_path(File.join(template_dir, template)) + File.expand_path(template, template_dir) end def files_dir @@ -85,7 +88,15 @@ def s3_template_file_name Utils.change_extension(template, 'json') end - def parameter_files + def all_parameter_files + if parameter_files.empty? + parameter_files_from_globs + else + parameter_files + end + end + + def parameter_files_from_globs parameter_file_globs.map(&Dir.method(:glob)).flatten end @@ -101,20 +112,26 @@ def s3_configured? !s3.nil? end + def parameter_files + Array(@parameter_files).map do |file| + File.expand_path(file, parameters_dir) + end + end + private def additional_parameter_lookup_globs additional_parameter_lookup_dirs.map do |a| - File.join(base_dir, 'parameters', a, "#{stack_name_glob}.y*ml") + File.join(parameters_dir, a, "#{stack_name_glob}.y*ml") end end def region_parameter_glob - File.join(base_dir, 'parameters', "#{region}", "#{stack_name_glob}.y*ml") + File.join(parameters_dir, "#{region}", "#{stack_name_glob}.y*ml") end def default_parameter_glob - File.join(base_dir, 'parameters', "#{stack_name_glob}.y*ml") + File.join(parameters_dir, "#{stack_name_glob}.y*ml") end def stack_name_glob diff --git a/lib/stack_master/version.rb b/lib/stack_master/version.rb index 9ae3d87c..b2e2bc44 100644 --- a/lib/stack_master/version.rb +++ b/lib/stack_master/version.rb @@ -1,3 +1,3 @@ module StackMaster - VERSION = "2.6.1" + VERSION = "2.7.0" end diff --git a/spec/stack_master/parameter_loader_spec.rb b/spec/stack_master/parameter_loader_spec.rb index 9d6015b1..65739e9e 100644 --- a/spec/stack_master/parameter_loader_spec.rb +++ b/spec/stack_master/parameter_loader_spec.rb @@ -2,7 +2,7 @@ let(:stack_file_name) { '/base_dir/parameters/stack_name.yml' } let(:region_file_name) { '/base_dir/parameters/us-east-1/stack_name.yml' } - subject(:parameters) { StackMaster::ParameterLoader.load([stack_file_name, region_file_name]) } + subject(:parameters) { StackMaster::ParameterLoader.load(parameter_files: [stack_file_name, region_file_name]) } before do file_mock(stack_file_name, **stack_file_returns) @@ -60,7 +60,7 @@ let(:region_yaml_file_returns) { {exists: true, read: "Param1: value1\nParam2: valueX"} } let(:region_yaml_file_name) { "/base_dir/parameters/us-east-1/stack_name.yaml" } - subject(:parameters) { StackMaster::ParameterLoader.load([stack_file_name, region_yaml_file_name, region_file_name]) } + subject(:parameters) { StackMaster::ParameterLoader.load(parameter_files: [stack_file_name, region_yaml_file_name, region_file_name]) } before do file_mock(region_yaml_file_name, **region_yaml_file_returns) diff --git a/spec/stack_master/parameter_validator_spec.rb b/spec/stack_master/parameter_validator_spec.rb index 1faf6746..79d5cdd7 100644 --- a/spec/stack_master/parameter_validator_spec.rb +++ b/spec/stack_master/parameter_validator_spec.rb @@ -2,7 +2,8 @@ subject(:parameter_validator) { described_class.new(stack: stack, stack_definition: stack_definition) } let(:stack) { StackMaster::Stack.new(parameters: parameters, template_body: '{}', template_format: :json) } - let(:stack_definition) { StackMaster::StackDefinition.new(base_dir: '/base_dir', region: 'ap-southeast-2', stack_name: 'stack_name') } + let(:parameter_files) { nil } + let(:stack_definition) { StackMaster::StackDefinition.new(base_dir: '/base_dir', region: 'ap-southeast-2', stack_name: 'stack_name', parameter_files: parameter_files) } describe '#missing_parameters?' do subject { parameter_validator.missing_parameters? } @@ -27,7 +28,7 @@ let(:parameters) { {'Param1' => true, 'Param2' => nil, 'Param3' => 'string', 'Param4' => nil} } it 'returns a descriptive message' do - expect(error_message).to eq(<<~MESSAGE.chomp) + expect(error_message).to eq(<<~MESSAGE) Empty/blank parameters detected. Please provide values for these parameters: - Param2 - Param4 @@ -38,6 +39,21 @@ end end + context 'when the stack definition is using explicit parameter files' do + let(:parameters) { {'Param1' => true, 'Param2' => nil, 'Param3' => 'string', 'Param4' => nil} } + let(:parameter_files) { ["params.yml"] } + + it 'returns a descriptive message' do + expect(error_message).to eq(<<~MESSAGE) + Empty/blank parameters detected. Please provide values for these parameters: + - Param2 + - Param4 + Parameters are configured to be read from the following files: + - /base_dir/parameters/params.yml + MESSAGE + end + end + context 'when no parameers have a nil value' do let(:parameters) { {'Param' => '1'} } diff --git a/spec/stack_master/stack_definition_spec.rb b/spec/stack_master/stack_definition_spec.rb index b9deb16e..f0af5f07 100644 --- a/spec/stack_master/stack_definition_spec.rb +++ b/spec/stack_master/stack_definition_spec.rb @@ -5,7 +5,8 @@ stack_name: stack_name, template: template, tags: tags, - base_dir: base_dir) + base_dir: base_dir, + parameter_files: parameter_files) end let(:region) { 'us-east-1' } @@ -13,6 +14,7 @@ let(:template) { 'template.json' } let(:tags) { {'environment' => 'production'} } let(:base_dir) { '/base_dir' } + let(:parameter_files) { nil } before do allow(Dir).to receive(:glob).with( @@ -35,7 +37,7 @@ end it 'has default and region specific parameter file locations' do - expect(stack_definition.parameter_files).to eq([ + expect(stack_definition.all_parameter_files).to eq([ "/base_dir/parameters/#{stack_name}.yaml", "/base_dir/parameters/#{stack_name}.yml", "/base_dir/parameters/#{region}/#{stack_name}.yaml", @@ -75,7 +77,7 @@ end it 'includes a parameter lookup dir for it' do - expect(stack_definition.parameter_files).to eq([ + expect(stack_definition.all_parameter_files).to eq([ "/base_dir/parameters/#{stack_name}.yaml", "/base_dir/parameters/#{stack_name}.yml", "/base_dir/parameters/#{region}/#{stack_name}.yaml", @@ -109,4 +111,12 @@ it 'defaults ejson_file_kms to true' do expect(stack_definition.ejson_file_kms).to eq true end + + context "with explicit parameter_files" do + let(:parameter_files) { ["my-stack.yml", "../my-stack.yml"] } + + it "ignores parameter globs and resolves them relative to parameters_dir" do + expect(stack_definition.all_parameter_files).to eq ["/base_dir/parameters/my-stack.yml", "/base_dir/my-stack.yml"] + end + end end