avajs/ava

Incredibly high CPU load during test running #2600

bhallstein posted onGitHub

Hi, I'm encountering some strange behaviour. I'm using an (I think) fairly standard ava + enzyme + babel setup.

I initially noticed my machine was running poorly — almost totally locked up — while ava tests were running. Investigating, I was astonished to find that CPU load was continuously over 100 during test running. The machine is so intensely locked up that Activity Monitor won't refresh any of its CPU or memory info until ava has finished running the tests. That 100 figure isn't a percentage, that's the actual CPU load figure, as a % it would be 10,000% CPU load. I thought I must be dreaming it, but it really is true.

My first thought was to try without concurrency (-c 1), and that results in CPU load around 1 (100%) as expected. Test running takes ~40s with -c 1 and CPU load of about 1, compare with ~30s without -c and CPU load of about 100, system all but unusable.

Thinking about this, and 3 things seem possible explanations:

  1. ava is getting the number of CPU cores available on the machine wrong, spawning too many threads/processes.
  2. a 'thread bomb' situation, where lots of threads are spawned continuously.
  3. inefficient inter-process communication is occurring. I wondered if poor multithreading support in one of the libraries, e.g. Enzyme, could have this effect? Or if there is a locking system for non-multithreaded shared libraries or some part of the test execution process that might be spin locking?

Or it could be something else. As I've never seen anything quite like this before, I'd really appreciate anyone's thoughts if they have further ideas.

Code: link to repo

The machine is a MacBook Air, 1.3GHz, 2 real cores and SMT.


To try and get more info about what's going on, I just tried a few different levels of concurrency:

  • without -c: 30s, CPU load ~= 100
  • -c 1: 40s, CPU load ~= 1
  • -c 2: 30s, CPU load ~= 50
  • -c 4: 30s, CPU load ~= 100 (perhaps a little lower than with concurrency auto-set, it's a bit hard to tell)

This suggests first that ava might be spawning too many processes: perhaps the number of logical rather than physical cores, while the workload does not spread well over SMT cores. However, even that wouldn't fully explain such high load figures: the jump from 1 at -c 1 to 50 at -c 2. What I think this suggests is that there is also some kind of inter-process communication issue, perhaps such as as busy waiting, but I could be very mistaken.

ava is by far my favourite testing library and I'd love to carry on using & recommending it! For now I have disabled concurrency on this test suite.

cpu-load-ava

posted by bhallstein about 4 years ago

@babel/register might result in each worker process recompiling the code (because IIRC @babel/register compiles the first time a file is loaded in that process). That'll hit your CPU.

We use node -pe 'os.cpus().length' to determine concurrency. What is that on your node machine?

posted by novemberborn about 4 years ago

Thanks @novemberborn. os.cpus().length returns 4, which roughly gels with what's above. I wonder if using the physical number of cores would dramatically improve efficiency without losing much (any?) performance on machines with SMT. Looks like that can be queried but it's not exactly pretty.

Thanks again, I'll see if I can get some figures on @babel/register's contribution to this.

posted by bhallstein about 4 years ago

Looks like that can be queried but it's not exactly pretty.

No, that looks pretty bad.

For I/O heavy tests, using all (hyper-threaded) cores can work well. But if the tests are extremely CPU heavy that might backfire. I suspect that's relatively rare though.

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going.)

posted by novemberborn about 4 years ago

I'm also hitting what I think is the same issue, I noticed it on https://github.com/xojs/xo/pull/606#issuecomment-928858131.

Basically whatever value of concurrency bigger than 1 spawns too many processes making the system unresponsive...

I'm using a VM with 4 threads, but even if I use -c=2 I'm getting tons of spawned processes.

<img src="https://user-images.githubusercontent.com/349621/135029254-10695160-c44f-45f8-b68d-634bd3d829f2.png" alt="2021-09-28_08-31-53" width="450">

posted by XhmikosR over 3 years ago

That's very curious.

Re my original issue, I do often see CPU load of around 100; though I see this with a lot of node/npm related tasks, such as parcel builds, npm install, and so on, so I think it's unlikely it has much to do with ava.

posted by bhallstein over 3 years ago

No, this is clearly due to ava. I only ran ava when I took the screenshot.

posted by XhmikosR over 3 years ago

Fund this Issue

$0.00
Funded

Pull requests