Remove memory limit pod (redhat-performance#537)

* add workflow dispatch (redhat-performance#476)

* add workflow dispatch (redhat-performance#476)

* - Eliminate memory limit for runc and Kata pods.
  Memory limits with Kata (sandboxed container) pods do not calculate the size
  of the VM correctly and can fail with an out of memory condition as a result.
  The limits are not needed to run correctly.
- Include the failing template file if a template operation fails.
- Add a top level Makefile to rebuild and test the golden files to simplify
  development.

* - Apply fix to Kata pod memory overhead to make limits safe.
- Restore Kata limit to the original 4 GiB.

* Remove conditionals on limits/requests

* add workflow dispatch (redhat-performance#476)

* add workflow dispatch (redhat-performance#476)

* runtimeclass patch needs to be executable

---------

Co-authored-by: ebattat <[email protected]>

Author: RobertKrawitz

HOW_TO.md

@@ -45,23 +45,19 @@ compares these files against expected files found in
 `tests/unittest/benchmark_runner/common/workloads_flavors/golden_files` and fails if
 any golden files have been added, modified, or removed.
 
-_*If you add or modify any YAML files, you must run the following commands:*_
+*If you remove any YAML files, you must identify the changed files and
+`git rm` them before committing the result.*
 
-```
-PYTHONPATH=. python3 tests/unittest/benchmark_runner/common/template_operations/generate_golden_files.py
-git add tests/unittest/benchmark_runner/common/templates/golden_files
-git commit -m "Update golden files"
-```
+_*If you add or modify any YAML files, you must run the following
+commands from top level to regenerate and test the golden files:*_
 
-If you remove any YAML files, you must identify the changed files and
-`git rm` them before committing the result.
 
-The check is run automatically as part of the unit tests; if you want
-to run it manually, you can do so as follows.  The test should take
-only a few seconds to run.
+The `make` command will run a check automatically to check to verify
+the golden files; this check is also run as part of the unit tests.
+The output will look like this:
 
 ```
-$ PYTHONPATH=. python3 -m pytest -v tests/unittest/benchmark_runner/common/template_operations/
+$ make all
 ============================== test session starts ===============================
 platform linux -- Python 3.9.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
 cachedir: .pytest_cache
@@ -74,6 +70,13 @@ tests/unittest/benchmark_runner/common/templates/test_golden_files.py::test_gold
 =============================== 1 passed in 1.85s ================================
 ```
 
+If the check succeeds, you need to add and commit the golden files:
+
+```
+$ git add tests/unittest/benchmark_runner/common/templates/golden_files
+$ git commit -m "Update golden files"
+```
+
 This test uses synthetic environment variables that you do not need to
 modify, and you should never have to manually modify the golden files
 except to manually remove any that are no longer required.

Makefile

@@ -0,0 +1,9 @@
+.PHONY: golden_files
+
+all: golden_files test_golden_files
+
+golden_files:
+	PYTHONPATH=. python3 tests/unittest/benchmark_runner/common/template_operations/generate_golden_files.py
+
+test_golden_files:
+	PYTHONPATH=. python3 -m pytest -v tests/unittest/benchmark_runner/common/template_operations/

benchmark_runner/common/ocp_resources/create_kata.py

@@ -71,6 +71,6 @@ def create_kata(self):
                 completed_nodes_count = self.__oc.run(cmd="oc get kataconfig -ojsonpath='{.items[0].status.installationStatus.completed.completed_nodes_count}'")
                 if total_nodes_count != completed_nodes_count:
                     raise KataInstallationFailed(f'not all nodes installed successfully total {total_nodes_count} != completed {completed_nodes_count}')
-            elif '03_ocp48_patch.sh' == resource:
+            elif resource.endswith('.sh'):
                 self.__oc.run(cmd=f'chmod +x {os.path.join(self.__path, resource)}; {os.path.join(self.__path, resource)}')
         return True

benchmark_runner/common/ocp_resources/kata/template/04_ocp_patch_runtimeclass_memory_overhead.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+oc patch runtimeclass kata -p '{"overhead":{"podFixed":{"memory":"2398Mi"}}}'

benchmark_runner/common/template_operations/template_operations.py

@@ -6,7 +6,7 @@
 import benchmark_runner
 from multiprocessing import Process
 
-from jinja2 import Template
+from jinja2 import Template, TemplateSyntaxError
 from benchmark_runner.common.template_operations.render_yaml_from_template import render_yaml_file
 from benchmark_runner.common.logger.logger_time_stamp import logger_time_stamp, logger
 from benchmark_runner.main.environment_variables import environment_variables
@@ -156,8 +156,11 @@ def __generate_yamls_internal(self, scale: str = None, scale_num: str = None, sc
             else:
                 file_components = os.path.splitext(filename)
                 template_file = f'{file_components[0]}_template{file_components[1]}'
-            with open(os.path.join(workload_dir_path, 'internal_data', template_file)) as f:
-                template = Template(f.read())
+            try:
+                with open(os.path.join(workload_dir_path, 'internal_data', template_file)) as f:
+                    template = Template(f.read())
+            except TemplateSyntaxError as err:
+                raise SyntaxError(f"Error while rendering {template_file}: {err}")
             answer[filename] = f"{template.render(render_data)}\n"
             if scale:
                 answer['namespace.yaml'] = render_yaml_file(dir_path=os.path.join(self.__dir_path, 'scale'), yaml_file='namespace.yaml', environment_variable_dict=self.__environment_variables_dict)

benchmark_runner/common/template_operations/templates/vdbench/internal_data/vdbench_pod_template.yaml

@@ -114,9 +114,7 @@ spec:
           memory: {{ requests_memory }}
         limits:
           cpu: {{ limits_cpu }}
-{%- if not kind == 'kata' %}
           memory: {{ limits_memory }}
-{%- endif %}
       {%- if cluster == "kubernetes" %}
       volumeMounts:
         - name: vdbench-pod-vol

benchmark_runner/common/template_operations/templates/vdbench/internal_data/vdbench_vm_template.yaml

@@ -83,7 +83,7 @@ spec:
             memory: {{ requests_memory }}
           limits:
             cpu: {{ limits_cpu }}
-            memory: {{ requests_memory }}
+            memory: {{ limits_memory }}
       terminationGracePeriodSeconds: 180
       volumes:
 {%- if odf_pvc == True %}

benchmark_runner/common/template_operations/templates/vdbench/vdbench_data_template.yaml

@@ -41,7 +41,7 @@ template_data:
       # size in MB
       SIZE_PER_FILE: 10
       limits_cpu: 2
-      limits_memory: 64Gi
+      limits_memory: 4Gi
       requests_cpu: 2
       requests_memory: 4Gi
       storage: 64Gi

tests/unittest/benchmark_runner/common/template_operations/golden_files/func_ci_vdbench_kata_ODF_PVC_False/vdbench_pod.yaml

@@ -34,6 +34,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       env:
         - name: BLOCK_SIZES
           value: "64,oltp1"

tests/unittest/benchmark_runner/common/template_operations/golden_files/func_ci_vdbench_kata_ODF_PVC_True/vdbench_pod.yaml

@@ -46,6 +46,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       volumeMounts:
         - name: vdbench-pod-pvc-claim
           mountPath: "/workload"

tests/unittest/benchmark_runner/common/template_operations/golden_files/perf_ci_vdbench_kata_ODF_PVC_False/vdbench_pod.yaml

@@ -34,6 +34,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       env:
         - name: BLOCK_SIZES
           value: "oltp1,oltp2,oltphw,odss2,odss128,4_cache,64_cache,4,64,4_cache,64_cache,4,64"

tests/unittest/benchmark_runner/common/template_operations/golden_files/perf_ci_vdbench_kata_ODF_PVC_True/vdbench_pod.yaml

@@ -46,6 +46,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       volumeMounts:
         - name: vdbench-pod-pvc-claim
           mountPath: "/workload"

tests/unittest/benchmark_runner/common/template_operations/golden_files/perf_ci_vdbench_pod_ODF_PVC_False/vdbench_pod.yaml

@@ -32,7 +32,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
-          memory: 64Gi
+          memory: 4Gi
       env:
         - name: BLOCK_SIZES
           value: "oltp1,oltp2,oltphw,odss2,odss128,4_cache,64_cache,4,64,4_cache,64_cache,4,64"

tests/unittest/benchmark_runner/common/template_operations/golden_files/perf_ci_vdbench_pod_ODF_PVC_True/vdbench_pod.yaml

@@ -44,7 +44,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
-          memory: 64Gi
+          memory: 4Gi
       volumeMounts:
         - name: vdbench-pod-pvc-claim
           mountPath: "/workload"

tests/unittest/benchmark_runner/common/template_operations/golden_files/release_vdbench_kata_ODF_PVC_False/vdbench_pod.yaml

@@ -34,6 +34,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       env:
         - name: BLOCK_SIZES
           value: "64,oltp1"

tests/unittest/benchmark_runner/common/template_operations/golden_files/release_vdbench_kata_ODF_PVC_True/vdbench_pod.yaml

@@ -46,6 +46,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       volumeMounts:
         - name: vdbench-pod-pvc-claim
           mountPath: "/workload"

tests/unittest/benchmark_runner/common/template_operations/golden_files/test_ci_vdbench_kata_ODF_PVC_False/vdbench_pod.yaml

@@ -34,6 +34,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       env:
         - name: BLOCK_SIZES
           value: "64,oltp1"

tests/unittest/benchmark_runner/common/template_operations/golden_files/test_ci_vdbench_kata_ODF_PVC_True/vdbench_pod.yaml

@@ -46,6 +46,7 @@ spec:
           memory: 4Gi
         limits:
           cpu: 2
+          memory: 4Gi
       volumeMounts:
         - name: vdbench-pod-pvc-claim
           mountPath: "/workload"