Added option to avoid ranging mode run, fixes #270 #273
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
Some sample test outputs are added below. Uncertainty command is working as expected both on the upper and lower sides. Actual power usage of SUT: 10-25W. Output with ranging + testing runs (no manual setting):Output:Ranging mode: Output with direct testing run (manual setting in the expected range):Output:Output with direct testing run (manual setting with wrong range on the higher side):Output:Output with direct testing run (manual setting with wrong range on the lower side):Output:More on Uncertainty SamplesAny sample where the measured uncertainty is above 1% get counted as an uncertain measurement like below. If required we can immediately stop such measurements as the result will be invalid. |
|
@arjunsuresh I'm a bit concerned about the output in the case when the range is set on the lower side. The measurement is clearly wrong, yet the uncertainty is 0.34%, less than 0.60% in the valid cases. How do we know that it is incorrect? More generally, do you have a description of the returned fields? |
|
@psyhtest Below are the descriptions of the return fields. The first 3 fields are average,min and max uncertainties which are not important to us as if for any given sample if the uncertainty is above 1%, it'll be counted as an "invalid sample" - third last entry. If measurement is not done, it'll be added to the "unknown sample count". If "total sample count" = "valid sample count" we can consider the result as valid. |
|
@psyhtest Based on your feedback during yesterday's WG meeting regarding different voltage ranges in US and Europe/Asia and also different power factor in different countries I guess it makes sense to allow |
|
I've got a perfect workload/sut combination where the power draw is just on the border of 200 mA, momentarily crossing it and then going under e.g.:
|
|
Thank you Anton for checking. Are the results fine at 100mA/50mA as well? I mean are the samples correctly classified as invalid/power reading is as expected? |
|
Adding some more data points as discussed during the last WG meeting. This is an example power meter reading. Current usage is 0.136 A Power measured is 16.34W Yokogawa 310E current ranges for crest factor 3 are as below WT310E Since the next current range above 0.136A is 0.2, the PTDaemon should be setting this and this is done as can be seen here: Now, lets see what happens when the user sets the current range to 0.5A and 1A and 0.05A. 0.5APower measured is 16.35W Result is good 1 APower measured is Result is good 0.1 APower measured is 0.05 APower measured is Result is questionable 0.02 APower measured is marked invalid (all samples being invalid) Result is rightly marked invalid 2APower measured is having uncertainty above the required threshold Result is rightly marked invalid |
|
@arjunsuresh / @psyhtest can you attach spl.txt excerpts for 0.1/0.05 (questionable measurements) |
|
Hi @dmiskovic-NV I don't have the full spl.txt now, but these are some entries from them I had saved. I can try to get the full spl.txt tomorrow. This is the output when current range is 1A (0.2A or 0.5A also gives similar output) This is the output when the current range is set to 0.05A |
|
@psyhtest Would it be possible to test this behaviour on the second system Krai had submitted with since it has 0.99 power factor?
|
|
@dmiskovic-NV When I set the current range to 200 mA and my actual usage is around 160 mA, I can see the power analyzer light for "Check Range Current" flashing once in a while meaning the range must be updated (this never turns on if I set the range to 500 mA). If I set to 100 mA/50 mA, this light is always on. So, the problem could be with the SPEC PTDaemon or how we are using it and not the power analyzer. |
I'm afraid not. Apart from a logistics nightmare, we simply do not have space to place the power analyser in the same room as the R282. All desks are now busy! |
|
No problem @psyhtest. I was curious as to whether it had to do with the power factor being low or the power being low. I checked the Uncertainty calculation and it is not surprising that we don't get uncertain samples when we give lower range than the actual. This is because the uncertainty calculation does 0.2*Voltage range * Current range, and so lower these ranges, lower will be the uncertainty. I believe we still have a way to invalidate wrong results. @psyhtest In the spl.txt you had shared, current range goes to 0.18 mA, but the manually set limit is 0.1 mA. In fact in all the problematic cases we have maximum current usage > set current range. |
looking at @psyhtest 's spl and previous comment, this spl.txt seems as 200mA range. (judging by timestamp) this looks quite worrisome since samples are between 40% and 95% of the range where meter is supposed to be quite trustworthy. Also, regarding allowable sample range, cutoff for WT3xxE series is at 130% of the range (with some additional reading error) |
|
Thank you @dmiskovic-NV for verifying. I mistook Anton's spl.txt as for 100mA measurement. Now, we have a mismatch of behvaiour between the same Yokogawa 310E meters too. In the meter I'm using, if I set maxAmps to 50mA, it does go to around 65mA which is around the 130% range. But the result Anton shared is not even touching 100%. |
|
As suggested by Greg (from SPEC team), I changed the crest factor to 6 as shown here and then setting any lower current range than the actual is correctly captured as an error. Just to conclude the findings so far
|
But not as high as 20 Amps if the SUT is under 200 mAmps, as my experiments show :) |
|
@psyhtest That is an unsupported case (actual current range being lower than 1% of the set range) and is rightly flagged by the output of the uncertainty command. So we can easily invalidate any such results from any submitter. |
|
We got confirmation from SPEC team that this problem happens for low power devices (<70W) when the power factor is below 80% and crest factor is 3. We are expecting a new PTDaemon release which can capture these in the Uncertainty result. Until then, this PR can be on hold. |
|
Hi @araghun As discussed in the WG meeting I have now added a warning for the manual range setting saying that it is only for "experimental purpose". Please review. |
|
Approved per alignment in 1/31 PowerWG. |
|
@arjunsuresh , @pgmpablo157321 @psyhtest : |
|
@araghun I have fixed the conflicts. Please merge it. |
|
@araghun I have updated the source checksums in this PR itself. I have removed the old checksums as our measurement methodology has been updated in recent accepted PRs. |
This PR provides an option to do power runs without having a ranging mode. As of now any power run has a ranging mode followed by a testing mode and both doing a full performance run. With this PR user will be able to manually provide Volt and Ampere ranges and thus will avoid the ranging mode run and directly do a testing run. If the user gives wrong ranges (tolerance for this range is quite high like giving 0.5 A instead of 0.1 is fine but giving 1A instead of 0.1A is not) uncertainty output will be produced and this will invalidate the measured result. Once this PR is merged the inference submission checker will need an update for taking care of this invalidation.
Instruction to test this PR
If you already have the setup to run MLPerf inference using power, just use this repository for the power source code and add the below options while calling the power client code
client.py. The given values of0.1and250must be adjusted according to the current and voltage usage of the SUT. Current range in between 1/5 to 5X of the actual usage is typically fine.An extra log file
run_1/ptd_out.txtwill be produced which will have the number of total power samples and the number of uncertain samples which should be 0 for a valid run. Some sample test outputs can be seen here.If you are not having a power setup for MLPerf inference but have access to a power analyzer you can easily test this PR using a dummy script and using CM as detailed here