Workflows: Support recursive directories (#795)

* Fix workflow files without path

On commit f909009 the workflow code was
cleaned up, but it broke files that had no `path`.

Error shown is similar to this:

```
    destdir = f"{destdir}/{os.path.dirname(entry['origin'])}"
                 ^^^^^^^
UnboundLocalError: cannot access local variable 'destdir' where it is not associated with a value
```

That's because in the previously mentioned commit variable
`default_destdir` that was removed was accidentally changed to `destdir`
instead of `outputdir`.

This patch fixes this problem using the right variable.

* Workflows: Support recursive directories

Current workflow code only supports files within a single directory.

This patch adds the functionality to support recursive directories, so
a workflow like this:

```
ctrlplane:
  type: workflow
  destdir: ./out
  scripts:
  - scripts/ctrlplane.sh
  files:
  - origin: ctrl
```

Will work with the file structure:

```
├── ctrl
│   ├── 00-openstack-ns.yaml
│   ├── 00-operators
│   │   ├── 00-operator-cert-manager.yaml
│   │   ├── 00-operator-cluster-observability.yam
│   │   ├── 00-operator-metallb.yaml
│   │   ├── 00-operator-nmstate.yaml
│   │   └── 00-operator-openstack.yaml
```

Author: Akrog

docs/index.md

@@ -1344,6 +1344,38 @@ Will create the following file structure:
 ./outdir/frout.j2
 ```
 
+When using a directory in a `files` section the structure will be recreated and all files within it will be rendered.
+
+If we have this file structure:
+
+```
+./arch/frout.sh
+./arch/frout.txt
+./arch/frout.j2
+./arch/subdir/anotherfile.sh
+```
+
+And we use this workflow:
+
+```YAML
+myworkflow:
+  type: workflow
+  destdir: outdir
+  scripts:
+  - arch/frout.sh
+  files:
+  - origin: arch
+```
+
+We'll end up with the following:
+
+```
+./outdir/frout.sh
+./outdir/arch/frout.txt
+./outdir/arch/frout.j2
+./outdir/arch/subdir/anotherfile.sh
+```
+
 ### vms
 You can point at an existing profile in your plans, define all parameters for the vms, or combine both approaches. You can even add your own profile definitions in the plan file and reference them within the same plan:
 

docs/index.rst

@@ -1446,6 +1446,40 @@ Will create the following file structure:
    ./outdir/frout.txt
    ./outdir/frout.j2
 
+
+When using a directory in a ``files`` section the structure will be recreated and all files within it will be rendered.
+
+If we have this file structure:
+
+::
+
+   ./arch/frout.sh
+   ./arch/frout.txt
+   ./arch/frout.j2
+   ./arch/subdir/anotherfile.sh
+
+
+And we use this workflow:
+
+.. code:: yaml
+
+   myworkflow:
+     type: workflow
+     destdir: outdir
+     scripts:
+     - arch/frout.sh
+     files:
+     - origin: arch
+
+We'll end up with the following:
+
+::
+
+   ./outdir/frout.sh
+   ./outdir/arch/frout.txt
+   ./outdir/arch/frout.j2
+   ./outdir/arch/subdir/anotherfile.sh
+
 vms
 ~~~
 

kvirt/baseconfig.py

@@ -1212,29 +1212,32 @@ def create_workflow(self, workflow, overrides={}, outputdir=None, run=True, base
             if 'path' in entry:
                 destdir = os.path.dirname(entry['path'])
             else:
-                destdir = f"{destdir}/{os.path.dirname(entry['origin'])}"
+                destdir = f"{outputdir}/{os.path.dirname(entry['origin'])}"
             if not os.path.exists(destdir):
                 pprint(f"Creating directory {destdir}")
                 os.makedirs(destdir)
-            content = entry.get('content')
-            render = entry.get('render', True)
-            if isinstance(render, str):
-                render = render.lower() == 'true'
-            if origin in directories:
+            # If we've already processed the directory tree then there's no
+            # point in looking at individual files.
+            if any(origin.startswith(d) for d in directories):
                 continue
             elif not os.path.exists(origin):
                 msg = f"Origin file {origin} not found"
                 error(msg)
                 return {'result': 'failure', 'reason': msg}
             elif os.path.isdir(origin):
-                origin = entry.get('origin')
-                if not os.path.exists(f"{destdir}/{os.path.basename(origin)}"):
-                    pprint(f"Creating directory {destdir}/{os.path.basename(origin)}")
-                    os.makedirs(f"{destdir}/{os.path.basename(origin)}")
-                    directories.append(origin)
-                for _fic in os.listdir(origin):
-                    directoryfiles.append(f'{origin}/{_fic}')
+                for path, _, files in os.walk(origin):
+                  new_destdir = f"{destdir}{path}"
+                  if not os.path.exists(new_destdir):
+                      pprint(f"Creating directory {new_destdir}")
+                      os.makedirs(new_destdir)
+                  directories.append(path)
+                  for _fic in files:
+                      directoryfiles.append(f'{path}/{_fic}')
                 continue
+            content = entry.get('content')
+            render = entry.get('render', True)
+            if isinstance(render, str):
+                render = render.lower() == 'true'
             filename = os.path.basename(origin)
             if not render:
                 content = open(origin, 'r').read()