[Benchmark] Add mechanism to retreive the logs from test-execution.json #4686
Conversation
Signed-off-by: Divya Madala <[email protected]>
Divyaasm
requested review from
dblock,
peterzhuamazon,
bbarani,
gaiksaya,
rishabh6788,
zelinh,
prudhvigodithi and
tianleh
as code owners
| with open(results) as file: | ||
| data = json.load(file) | ||
| formatted_data = pd.json_normalize(data["results"]["op_metrics"]) | ||
| formatted_data.to_csv(os.path.join(os.getcwd(), "test_execution.csv"), index=False) |
There was a problem hiding this comment.
rename the csv file to test_execution_<suffix>.json
| @@ -53,4 +56,7 @@ def run_tests(self) -> None: | |||
| with WorkingDirectory(current_workspace): | |||
| with BenchmarkCreateCluster.create(self.args, self.test_manifest, config, current_workspace) as test_cluster: | |||
| benchmark_test_suite = BenchmarkTestSuite(test_cluster.endpoint_with_port, self.security, self.args, test_cluster.fetch_password()) | |||
| retry_call(benchmark_test_suite.execute, tries=3, delay=60, backoff=2) | |||
| try: | |||
| benchmark_test_suite.execute() | |||
There was a problem hiding this comment.
why remove retry_call?
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4686 +/- ##
==========================================
- Coverage 92.40% 92.29% -0.11%
==========================================
Files 193 193
Lines 6317 6411 +94
==========================================
+ Hits 5837 5917 +80
- Misses 480 494 +14 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Divya Madala <[email protected]>
| try: | ||
| retry_call(benchmark_test_suite.execute, tries=3, delay=60, backoff=2) | ||
| finally: | ||
| subprocess.check_call(f"docker rm docker-container-{self.args.stack_suffix}", cwd=os.getcwd(), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
There was a problem hiding this comment.
Can we move the docker rm command to BenchmarkTestSuite class to a method like remove_container or whatever you like and then use that?
| log_info = f"Executing {self.command.replace(self.endpoint, len(self.endpoint) * '*').replace(self.args.username, len(self.args.username) * '*')}" | ||
| logging.info(log_info.replace(self.password, len(self.password) * '*') if self.password else log_info) | ||
| subprocess.check_call(f"{self.command}", cwd=os.getcwd(), shell=True) | ||
| with TemporaryDirectory() as work_dir: |
There was a problem hiding this comment.
move this file processing logic a new method and call that method only in the case endpoint is provided. This way the existing nightly runs will not be affected and we can enable it whenever we are ready.
Signed-off-by: Divya Madala <[email protected]>
Description
Get the logs from the test_executions folder and log for user's reference. Eventually the converted test_execution.csv file is archived to the respective jenkins build.
Issues Resolved
#4634
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.