Create dataflow flex template module

[daniel-cit]

Fixes #163

It's a good idea to open an issue first for discussion.

• Tests pass
• Appropriate changes to README are included in PR
• replace resource calls in examples

[erlanderlo]

Wow! I like how you approached this and can be quite helpful.

[erlanderlo]

LGTM on the examples using the flex-template module.

[bharathkkb]

Thanks for the PR. I was wondering if we can upstream necessary changes to https://github.com/terraform-google-modules/terraform-google-dataflow and use that instead of having maintain this in two places?

[daniel-cit]

Thanks for the PR. I was wondering if we can upstream necessary changes to https://github.com/terraform-google-modules/terraform-google-dataflow and use that instead of having maintain this in two places?

I am not sure on upstreaming the code because we are a bit more opinionated about some items:

• we don't have the default network as a valid default value, we require a explicit subnetwork self link
• we require the user to provide a KMS key
• we require the user to provide a dataflow controller service account
• we require the to provide a staging location
• we have explicit support fort network tags

@bharathkkb @erlanderlo

[erlanderlo]

It's likely a future item to move it upstream, and for now we wanted to modularize things first and observe how the module is used with these very specific items that were mentioned.

[erlanderlo]

LGTM; I'm guessing the tests are decoupled because we will be added using the newer go framework

[bharathkkb]

@daniel-cit @erlanderlo yup if we have opinionated defaults lets use this as a thin wrapper around the upstream module to enforce specific patterns. We can do this as a follow up. Ideally we want the upstream module to support security focused usage patterns similar to safer cluster wrapping main GKE module.

[daniel-cit]

@daniel-cit @erlanderlo yup if we have opinionated defaults lets use this as a thin wrapper around the upstream module to enforce specific patterns. We can do this as a follow up. Ideally we want the upstream module to support security focused usage patterns similar to safer cluster wrapping main GKE module.

Opened issue terraform-google-modules/terraform-google-dataflow#32 in the dataflow module to track this