Use logical cpus instead of physical cpus

[mdeboer]

To get the most out of CPUs with hyperthreading, use /usr/sbin/sysctl -n hw.logicalcpu on OSX instead of /usr/sbin/sysctl -n hw.physicalcpu.

[alexislefebvre]

That's an interesting point. Hyperthreading offer about 30 % more power than the same core without Hyperthreading. If you launch two processes for two threads that are treated by one core, you're requiring a 200 % load while the CPU can only provide about 130 %.

I suggest you to compare time taken by tests when you use more or less threads with the --process option. Could you please give us the results of your tests?

[mdeboer]

I think it’s not solely about the usage per core but also having a small queue “waiting”. The hyper threaded threads may not be as fast as a physical core but why waste a bit of cpu time when we can use it to create a thread and start testing. Then when the first thread on the core is done, there’s already one that can get our full attention and no time is wasted on garbage collection, starting a new thread and possibly fixtures.

I’ll cook up some numbers :)

[alexislefebvre]

When I tested with one process per core vs. two processes per core, it looked like each test process was saturating one core, so there was no free resource for another process.

[mdeboer]

Some results from a project that I'm working which is a Symfony 3.3 project with functional tests and database fixtures. I ran these tests on a Macbook Pro (2017) with dual-core i5 (2 physical cores, 4 logical cores with HT). I wrote a script to run these tests after clearing the cache and running it once to warm the (op)cache.

2 threads:
Test 1: Time: 18674 ms, Memory: 6291456 b
Test 2: Time: 17154 ms, Memory: 6291456 b
Test 3: Time: 17677 ms, Memory: 6291456 b
Average: 17835 ms

4 threads:
Test 1: Time: 13476 ms, Memory: 6291456 b
Test 2: Time: 13808 ms, Memory: 6291456 b
Test 3: Time: 14027 ms, Memory: 6291456 b
Average: 13770 ms

So yeah, that's 22.79% faster on average at least on this workload with over 90% of the tests being functional tests, requiring to bootstrap the whole application and load fixtures. I'll test this with some other projects that feature more unit tests to see if there's a difference when each test finishes a lot quicker.

[alexislefebvre]

Thanks for the tests, these are good results!

IIRC tests took longer time with Hyperthreading, I tested it on a quad core / 8 processes i7 2 years ago. Too bah I didn't save the results of my experiment.

[mdeboer]

You're welcome! 👍 I've heard that for gaming it's often preferred to have HT disabled so I guess it really depends on the workload.

Also I noticed that for linux you already count the logical number of CPU's by counting occurrences of processor. In the text fixture it shows 4 processors which according to the Intel specs equal the logical number of cores. In each processor entry it also states "cpu cores: 2".

So yeah, things are a bit skewed now as for all platforms you use the number of logical processors except for OSX 😄

[francoispluchino]

Just for information, for the Windows platform, the number of logical processors is already used.

[mdeboer]

Yes, just like for Linux 👍 It's just Mac that uses the physical processor count.