Spike/WIP: Stack definition yaml files. Explicit parameter_files & parameters in stack definitions.

[stevehodgkiss]

Some experimental changes to simplify StackMaster usage.

Stack definitions move from stack_master.yml to their own stack-name.yml

Stacks can be defined in their own YAML file and applied with stack_master apply stack-name.yml. region must be defined in stack-name.yml or stack_master.yml's stack_defaults.

This adds flexibility in terms of how stacks are organised on the filesystem, and prevents stack_master.yml bloat.

Explicitly define parameter files locations with parameter_files

Adds the ability to explicitly define parameter file locations in stack definitions parameter_files: [my-parameters.yml]. The benefit here is less magic about where they're sourced from, and the ability to share parameter files in any way you see fit. For example, if we have myapp-blue and myapp-green, they could share a parameter file, whereas currently that's not possible without duplicating the parameters.

Add the ability to define parameters in stack definitions

Used with stacks in stack_master.yml, this would make the file extremely bloated. When stacks are in their own yaml file though, it makes them easier to read and understand in context rather than being split over multiple files.

These changes may remove the need to have any concept of environment/region alias as discussed as part of StackMaster 2.0.

Example

Here's a made up example using all these features:

cat myapp-web-blue.yml
template: myapp.rb
tags:
  Name: myapp-blue
parameter_files:
  - shared_parameters/myapp-web.yml
  - shared_parameters/myapp-web-production.yml
parameters:
  name: myapp-blue
  bucket_name:
    stack_output: myapp-blue/bucket_name

cat shared-parameters/myapp-web.yml
some_param: 'value'

cat shared-parameters/myapp-web-production.yml
some_other_param: 'value'

stack_master apply myapp-web-blue.yml

Thoughts?

[joesustaric]

Interesting @stevehodgkiss .. so just clarifying some observations.

In the example, the myapp-web-blue.yml file specifies 2 parameter files. I'm assuming that behaviour is the same as current. eg they inherit and will subsequently override any params.

There are production params being inherited in the example (determined via - shared_parameters/myapp-web-production.yml), so I'm assuming this example is a prod stack which would then need to be organised in some logical file system grouping . i.e myapp-web-blue/production

If thats true then there would be something like myapp-web-blue/staging . Would that mean the stack definition for myapp-web-blue.yml would need to exist in both folders?

[stevehodgkiss]

In the example, the myapp-web-blue.yml file specifies 2 parameter files. I'm assuming that behaviour is the same as current. eg they inherit and will subsequently override any params.

Yep, they'd be merged from first to last.

There are production params being inherited in the example (determined via - shared_parameters/myapp-web-production.yml), so I'm assuming this example is a prod stack which would then need to be organised in some logical file system grouping . i.e myapp-web-blue/production

Yep, that's the idea. You can structure the stack yaml files on the filesystem however you see fit - for example, separate directories for staging/production. And you can share parameter files in any way that makes sense. More flexibility, less magic.

If thats true then there would be something like myapp-web-blue/staging . Would that mean the stack definition for myapp-web-blue.yml would need to exist in both folders?

Yep, in the same way that there'd be 2 definitions in a stack_master.yml file with the current approach.

The PR atm supports the existing approach and this one.

[joesustaric]

Yep great gotcha, thanks for clarifying.

Seems like a good idea.
I've certainly seen some people new to stack_master have some confusion when navigating the templates, tracing back parameters and navigating environments from alias' etc..
So a big +1 to removing some of the magic.

[patrobinson]

this is going in the right direction

I like the idea that environments/region_defaults can be implemented without a massive refactor like i was thinking. I agree that it's better to be explicit about which parameter files are loaded.

However i dislike allowing both approaches and merging the results from the existing hierarchical lookup with explicit parameter files as well as parameters defined in a stack definiition file. This could be very confusing and difficult to maintain

I also am not sure if adding another file, the stack definition file, to the mix helps, as now you can have 3 files define the behaviour of a stack instead of just 2. This implies some more magic, while removing some other.

I'd really like to nail down a name for and scope for a concept like what @grassdog referred to in #256 . This could be defined in a single file, like you've implemented. Then some "targets" for that stack, with the stack name and region as well as override parameters.

The more I think about it, the more getting rid of a single stack_master.yml file with a directory of parameter files makes more sense.

Imagine your scenario:

myapp-web.yml:

template: myapp.rb
parameter_files:
  - shared_parameters/myapp-web.yml
targets:
  blue:
    region: us-east-1
    allowed_accounts: '987654321'
    tags:
      Name: myapp-blue
    parameter_files:
      - shared_parameters/myapp-web-production.yml
  green:
    region: us-east-1
    allowed_accounts: '987654321'
    tags:
      Name: myapp-blue
    parameter_files:
      - shared_parameters/myapp-web-production.yml
  staging:
    region: us-east-1
    allowed_accounts: '123456789'
    tags:
      Name: myapp-blue
    parameters:
      staging: 'true'
      bucket_name:
        stack_output: foo/bucket_name

stack_master apply myapp-web blue

But this still involves the giant refactor I've been procrastinating over for 18 months

[stevehodgkiss]

However i dislike allowing both approaches and merging the results from the existing hierarchical lookup with explicit parameter files as well as parameters defined in a stack definiition file. This could be very confusing and difficult to maintain

Yeah, I was thinking about removing support for defining stacks in stack_master.yml and the magic parameter lookups. I haven't used this approach enough to know for sure that's the direction I'd like, but I think it is.

Interesting idea (the target example), but I wonder if the added complexity is worth it. That could be expressed without StackMaster needing to have the concept of a target though:

cat blue/myapp-web.yml
region: us-east-1
allowed_accounts: '987654321'
tags:
  Name: myapp-blue
parameter_files:
  - ../shared_parameters/myapp-web.yml
  - ../shared_parameters/myapp-web-production.yml
cat green/myapp-web.yml
region: us-east-1
allowed_accounts: '987654321'
tags:
  Name: myapp-green
parameter_files:
  - ../shared_parameters/myapp-web.yml
  - ../shared_parameters/myapp-web-production.yml

stack_master apply blue/myapp-web.yml

What benefit would modelling targets inside StackMaster bring vs allowing the user to lay it out like that manually?

[patrobinson]

Yeah, I was thinking about removing support for defining stacks in stack_master.yml and the magic parameter lookups. I haven't used this approach enough to know for sure that's the direction I'd like, but I think it is.

Should we enable it behind a feature flag? Have the ability to turn on/off explicit parameters, defaulting to off.

With "stack definition" files my loosely held opinion is I'd rather have them be standalone, i.e. if a stack definition file is specified the stack_master.yml file is not referred to at all. But then again I haven't thought about your use case deeply

What benefit would modelling targets inside StackMaster bring vs allowing the user to lay it out like that manually?

Nothing tangible at present, but I believe in the future when we can define multiple stacks a "target" applies then it'll prove more useful. For the purposes of this discussion it's not particularly useful as you have more concrete problems you want to solve.