Gigantic scores of DB12 recorded by DIRAC Pilots (from 50 up to 2250) #7760
Comments
|
I also have to notice that the working nodes are virtualized with KVM. On the second screenshot there are results of 40 dirac-wms-cpu-normalization which were received after the test. As you can see, the normal benchmark is within 35-37 range. When "something" happens, it may grow up to 55. And even more when we have longer test done by real dirac pilots. |
|
And I also created a test version of DB12 benchmark to test it. And so far it looks like the os.times function works wrong: The norm calculated as: To record times os.times() function is used. Start values are recorded in the beginnig of benchmark workload iterations. End values are recorded after benchmark workload. In the following listing CPU represents exactly that: (end u+s+cu+cs) - (start u+s+cu+cs). And I was able to record really strange values: That is listing when estimations are done sequentially: Normally CPU is around 6.25. But for the biggest benchmark from the first listing the CPU value may be 2. Which means that the benchmark workload was executed 3 times faster than it should, which is impossible. It is the first time I observe this within DIRAC environment. And it is only on one particular resource. It is counter intuitive, because benchmark results are used to calculate normalized CPU time. But looks that if on some resource more than 1 job running on one core, the benchmark could be higher! I will try to make more tests on the other server which is neither KVM virtualized, nor Alma Linux. BTW: DIRAC version v8.0.36 |
|
Thank you @IgorPelevanyuk for the notification and the extensive tests. At a minimum, they show that there some fixes to do. Please report back your findings and your suggestion. EDIT: we do not have much development workforce at the moment, so you are of course welcome to take care of what will need to be done. |
|
Hello, @fstagni! So far, I think that the main culprit of the issues is ProxmoxVE on Debian. No similar issues found on the same cluster but with Libvirt on Fedora. We will try to make more tests to understand the reason(in two weeks), why Proxmox behaves so strange. It is hard to tell somethin about solution. The issue appears only in a very specific virtual environment. I hope that it can be fixed by some setting in Proxmox. |
|
Just for info, are you also using cgroups on these cluster? |
|
Slurm on this particular resource is configured to use cgroups. But other Slurm resources also use it. |
|
After several other tests the issue still persist. Right now, advantage (or not?) of DIRACBenchmark is the fact, that you can run it even on 100% loaded cluster, and because the benchmark do not rely on walltime during its estimations, you can get rather accurate result. Example: on a server that are free from load you may get DB12 result for 30 seconds. After that you run some processes and overload the server so that 100% of all cores are used and after that start DB12 benchmark. In that case it would take longer for benchmark to perform test because from time to time it will be interupted by other processes on the server, so the benchmark test would last for 60 seconds. But in the end, you will get the same results of DB12. In this example I am neglecting the fact that frequency of an empty processor is usually much higher. If we would like to use walltime in Python there are time.perf_counter() function. In that case we would get benchmark results more related to the real speed of the server. But it will not be possible to get more or less accurate result in case if you want to benchmark server that is already 100% loaded. Honestly, I do not fully understand why using cputime is better for benchmarking comparing with walltime. The only advantage is a possibility to run benchmark in parallel with some other processes running on the same CPU core. I think that in our DIRAC installation we may try to tweak DIRACBenchmark script in CVMFS to use walltime and see how it works. What may be done for the DIRAC in general is writing to JobParametersDB not only DB12 calculated with cputime, but also DB12 calculated with walltime and call it DB12WT or something like this. In that case it will be possible to make some extansive analysis on different DIRAC installations and computing resources. And after that decide if the move from one approach to another is justified. |
|
Hi Igor, as discussed in Thursday's meeting: we can insert an alternate CPU power in the Job parameters, please start with this commit: 2de8586 On the DB12 side, you define a new method or a new flag, I let it up to you. |
|
@fstagni, Thank you! Do you propose to put this code in DIRAC master branch, or it is only a hint for me to change our local DIRAC installation directory placed in CVMFS? |
|
You can add it to |
Hello,
Recently I have noticed unrealisticly big values of DB12 benchmark. Normally it shold be somewhere between 0 and 45. But I noticed that during this summer we had pilots with jobs that recorded values up to 2250.
So far it is observed only on one computing resource with AlmaLinux and CPU Model Intel Gold 6242R.
Here are zoomed version from our local monitoring. Every dot is a job, color is CPU model, X is DB12 value, Y is job walltime.
I have checked the databases:
Unfortunately, jobs with the craziest DB12 benchmark was cleand from DIRAC and there are only resent jobs with wrong DB12 benchmark. But I was able to grep DIRAC Pilot logs on the computing resource to get this:
Does anybody saw with this before?
The text was updated successfully, but these errors were encountered: