Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime stats for application versions #3356

Open
drivere opened this issue Oct 29, 2019 · 7 comments · May be fixed by #3362
Open

runtime stats for application versions #3356

drivere opened this issue Oct 29, 2019 · 7 comments · May be fixed by #3362

Comments

@drivere
Copy link

drivere commented Oct 29, 2019

It would be nice to keep track of average run times and cpu times for each application version in the database.

This information could then be displayed directly in apps.php (and possibly other places). This would give desirable information for both project developers and users. For example, you could compare average run times between 32 bit app versions and 64 bit app versions; or between a CPU version and a GPU version.

To implement this, 4 new fields would need to be added to both the app_version table and the host_app_version table:
runtime_n
runtime_avg
cputime_n
cputime_avg
These would then be tracked in the validator. Impact to server performance would be negligible.

SETIguy created a branch called runtime_stats, where the above solution was implemented as a proof of concept. I have been using it for several months in a private test server and it works very well. Here is a screenshot showing the runtimes in apps.php:
appVersionsScreenshot

@RichardHaselgrove
Copy link
Contributor

This concept could be useful where task runtimes are solely or predominantly determined by application version.

But for many projects, task runtime is determined by the data being analysed - sometimes deterministically, sometimes with a strong degree of randomness.

I'm surprised to see you quoting SETI in this context: the SETI project comes into the 'deterministic variablility' class (varies by angle range and search parameters). Has that been taken into account by your algorithm?

@drivere
Copy link
Author

drivere commented Oct 29, 2019

That is true, it would only be useful if the average runtimes were consistent for a given batch. It would be a moving average, so it would adjust over time as batches changed. Also note that randomness is not a problem, as long as the mean of the runtimes makes sense. If the underlying distribution of runtimes is multi-modal then the mean would not be a good measure of anything.

I actually was not quoting SETI in my original comment. I mentioned that "SETIguy" has already made the required changes, for no other reason than to give him credit, as he is the one who did the work. However, the fact that he was interested in this tells me that there might be an application for it at SETI, but I will let him speak to that.

@SETIguy
Copy link
Contributor

SETIguy commented Oct 29, 2019

The version in the runtime_stats branch, it uses the same average as the PFC code, (an exponential running average) For the version of this currently running on the SETI@home Beta, I'm using a running median with roughly the same time constant as the exponential average, but it is far more stable, and does not require outlier exclusion except in the case that you could get several thousand outliers in a row. We could include that in this PR or have a separate PR.

Of course, neither handles the case of a widely bifurcated distribution, but nothing would short of custom code to recognize that case (ala VLAR at S@H).

@davidpanderson
Copy link
Contributor

davidpanderson commented Oct 29, 2019 via email

@SETIguy
Copy link
Contributor

SETIguy commented Oct 29, 2019 via email

@drivere
Copy link
Author

drivere commented Oct 29, 2019

@davidpanderson
Interesting approach. I was actually unaware of the gpu comparison web code (I just found gpu_list.php which I assume is what you are referring to). This is a neat feature, I wish I had known about it sooner.
The one concern I might have is a hit on server performance - it took about a minute for me to load the page, if many users were accessing this page it could bog down the server. I guess it could be run as a periodic task and dumped to an html file to reduce the load.

@RichardHaselgrove
Copy link
Contributor

@SETIguy - try #2949

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

Successfully merging a pull request may close this issue.

5 participants