YAML import support

[pidpawel]

Hi!
First of all thanks for creating a very promising piece of software!

This post is more of an idea description rather than a bug report.

I'd like to have as much of the definitions in the repository as possible and I'm looking for a way of incorporating some setups that utilize YAML filed provided by software authors into kuberig's config. Unfortunately those YAML files still are a de facto standard and in order to be more compliant with the rest of the world a pattern for dealing with those seems necessary.
Solving this problem by writing a tool to generate Kotlin code based on the YAML file sounds interesting but like a lot of work at the same time. Solution that'd probably be a little bit less involving would be to add a support for reading YAML files, unpacking them as proper resource objects and allowing the user to modify them inflight if needed.

Example usecase/scenario:

• take a file from https://kubernetes.github.io/ingress-nginx/deploy/
• save proper deploy.yml file as ingress-nginx.yml in a specific place
• add a method decorated with EnvResources
• read that YAML file from there
• modify i.e. namespaces to your desire/requirements with a some kind of filter or other tooling
• emit proper resources

One of the main benefits (to me) of that solution is that in case upstream updates their manifests, you'd only need to wget the new version, verify that new state matches what you wanted (i.e. namespaces in all new objects are modified too), and deploy changes. Think of this workflow like a much cleaner version of Helm charts. Still not super pretty, but better. :-P

Please do let me know what do you think about this idea… or if it's already implemented - how do I find it - as I failed to find it myself. I'd start implementing a pull request with that feature myself but I'm still a little bit too new to JVM world.

--
Have a wonderful new year
pidpawel

[teyckmans]

Hi @pidpawel ,

First of all thanks for the nice feedback.

You did not miss it in the docs, it is not available at the moment.

I think this is a good idea for several reasons:

• Indeed YAML files are the default. They won't be going away any time soon.
• It may be the only alternative in case the Kuberig DSL is not available for CRDs. Leaving you with no other option but to use another tool to deploy those. Which is obviously not what we want. Because this may be a reason to not use Kuberig at all.

Having a converter from YAML to Kuberig DSL code is something to think about for YAML files that you are managing yourself. It adds an additional step to apply it for externally maintained YAML files. So at this moment I don't think it makes sense to require this.

We do need to put some thought into it.

First idea would be to have something like this:

@EnvYaml(group / ...)
fun ingressNginx(): EnvYamlSource {
    return envYamlSource("src/main/yaml/ingress-nginx.yml") { // this block provides the namespace filter 
             useEnvironmentDefaultNamespace() 
             // or 
             useNamespace("my-ingress-ns") 
             // most likely only useful when you have dedicated clusters per environment 
             // or you would suffix the namespace with the environment name in case of shared clusters.
    }
    // or
    // No namespace filter block will result in using the namespace defined in the yaml file.  
    // By default in case no namespace is specified on resources they will get the default namespace for the environment. 
    // So I think we need to make sure the same thing happens for all the resources from a YAML file or give an error otherwise.
}

I think it would be a good convention to have the YAML files in the src/main/yaml directory. So not sure if it really needs to be specified.
I am limiting filtering options to the namespace only because we may not have a DSL class to read to in case of missing CRD DSLs. So filtering would currently work on the raw JSON objects.

I am working on #33 now that touches a lot of the deploy code and how resources are picked up from annotated methods so I need to finish up on that one first. Work for this enhancement will touch a lot of the same code.

[teyckmans]

We also need to consider this new @EnvYaml annotation needs to be compatible with the @EnvFilter annotation.

[pidpawel]

Hi!
I am really glad that you like the idea!
Your proposal to make the filtering apply only to raw JSON objects makes more sense than what I imagined. Thanks for giving it a thorough thought!

[teyckmans]

While working on this I've discovered that kuberig is always adding the namespace attribute in metadata. Which obviously is not correct. It has to consider the namespaced flag.

This does however mean that the generateYaml task is going to need to access the API server in order to retrieve the namespaced flag. It also means that when there are CRDs used that are also part of the deployment it will fail to find the needed info on the server. Not really a fan of this situation but lets have it this way anyway and see if there are actual issues with it.

[teyckmans]

[teyckmans]

In order to deal with this properly, we need to introduce a new stage to go from the initial generated JSON to what it needs to be. Using a combination of sources to get the APIResource information from (both API server and CRD definitions from the resources that are going to be deployed).