A more detailed summary of performance runs available with 'get performance' command

[bmsgit1]

• changed scenario_performance_map to include multiple performance metrics from mlperf_log_summary.txt file
• simplified scenario_performance_map (the beautify_summary function in code_axs already does all the formatting of units etc.. required)
• added 99p latency: target latency ratio to 'get performance' summary for server runs

[bmsgit1]

example: for a server performance run, 'get performance' would return a list of strings,
something like:

['VALID :', 'target_qps=152500', '99.00_percentile_latency=7.805 milliseconds', 'target_latency=15.000 milliseconds', 'latency_cutoff_ratio=0.5203223333333333', 'Completed_samples_per_second=152498.92']

[bmsgit1]

sorry @xihajun I accidentally removed saheli !!

[kaidrake]

I have tested the changes on my experiment entries for classic benchmarks for Offline and Server. It looks great:

axs byname generated_by_image_classification_using_qaic_kilt_on_get_8cd307069273465a828fc88f09904754 , get performance
['VALID :', 'Samples_per_second=425210', 'target_qps=435000']

axs byname generated_by_bert_using_qaic_kilt_on_get_d649d2abe5bc43b7a240ab255830fc4a , get performance
['VALID :', 'Samples_per_second=14821.5', 'target_qps=14600']

axs byname generated_by_object_detection_using_qaic_kilt_on_get_2b473f54a8fe45e395d714b4cf6eab62 , get performance
['VALID :', 'Samples_per_second=7081.97', 'target_qps=7500'] 

axs byname generated_by_image_classification_using_qaic_kilt_on_get_cc564210f6fa4999a07d78e78eb2071b , get performance
['VALID :', 'target_qps=426600', '99.00_percentile_latency=13.664 milliseconds', 'target_latency=15.000 milliseconds', 'latency_cutoff_ratio=0.9109276666666667', 'Completed_samples_per_second=426699.58']

axs byname generated_by_bert_using_qaic_kilt_on_get_de793ffb35d04cc8aa22e6bdc48fb8f0 , get performance
['VALID :', 'target_qps=6850', '99.00_percentile_latency=87.096 milliseconds', 'target_latency=130.000 milliseconds', 'latency_cutoff_ratio=0.6699658615384615', 'Completed_samples_per_second=6851.47']

axs byname generated_by_object_detection_using_qaic_kilt_on_get_bb1bbdcd0ed94e289ab8be7b3c78960e , get performance
['VALID :', 'target_qps=6350', '99.00_percentile_latency=86.401 milliseconds', 'target_latency=100.000 milliseconds', 'latency_cutoff_ratio=0.86400786', 'Completed_samples_per_second=6351.55']

With respect to the code, I spotted some extra spaces: base_loadgen_experiment/code_axs.py
L49,L82
base_loadgen_experiment/data_axs.json
L37

[bmsgit1]

yep I will remove those spaces and the unsightly colon after the validity!

[sahelib25]

It would look good if latency_cutoff_ratio is rounded to 2 decimal places.

[bmsgit1]

sure!

[bmsgit1]

The command output now looks like this: ['VALID', 'target_qps=152000', '99.00_percentile_latency=7.722 milliseconds', 'target_latency=15.000 milliseconds', 'latency_cutoff_ratio=0.51', 'Completed_samples_per_second=150593.59']