trainSVM scaling with multiple CPUs

[tobigithub]

Hi,
this is for finger1d 1.4 and shen_ISMB2014.py
Scaling for trainSVM(ckm_f, fingerprints, np=....) (np= number of processors)

1 CPU:
real 0m19.516s
user 0m30.252s
sys 0m2.136s

2 CPUs
real 0m12.498s
user 0m30.048s
sys 0m1.856s

4 CPUs
real 0m9.657s
user 0m29.472s
sys 0m1.800s

8 CPUs
real 0m8.742s
user 0m31.916s
sys 0m1.828s

16 CPUs
real 0m7.935s
user 0m34.060s
sys 0m2.280s

So there is basically for this small set fgtree_folder = "test_data/train_trees/" there is no scaling beyond 4 CPUs. There is some multiprocessing at the first 5-10 seconds of the code. The most time consuming part of the code (again only single CPU at 100% use) is then Writing LI kernel to LI_kernel.txt and Writing RLI kernel to RLI_kernel.txt and the general output.

Cheers
Tobias

[huibinshen]

I think may be before new job started, the old one has finished so it use the old CPU not the new ones. This training tasks are so small and it runs on the fly.

[huibinshen]

Also, in the example, it has 10 fingerprints to predict, so more than 10 processes would not be useful.