How the OS Scheduler Works

Why the background job hurts

Now the earlier problem makes sense.

Your background job is CPU-hungry and stays runnable almost all the time. Every time your web request becomes runnable, it might find every core busy with that job. So it waits in the run queue for a time slice.

Each wait is small, maybe just a few milliseconds. But a single request touches the CPU many times. It has to parse the input, run your logic, and serialize the response. Each step can wait. Add them up and you get the latency spikes on your dashboard, even while average CPU usage looks calm.

How the scheduler shares a core

Since one core runs one thread at a time, the scheduler gives each runnable thread a small slice of time. Then it pauses the thread and switches to the next one. This is a context switch. If the scheduler does this fast enough, it looks like everything runs at once. In reality, the core is taking turns.

Threads do not all have to be treated equally. You can tell the scheduler that some work is less important. This makes it step aside when something more important wants to run. On Linux, this is the nice value. A higher number means lower priority:

This is the direct fix for the situation above. Mark the background job as low priority. When it competes with a web request for a core, the request wins and the job waits. The job takes a little longer to finish, but your users do not notice. This is exactly the trade you want.

The same idea works in reverse for your own service. Keep the number of runnable threads near your core count so the run queue stays short. For a Node.js app, that means one worker process per core with the cluster module. You want to avoid hundreds of threads fighting over the same cores.

The mental model

Stop reading CPU usage as one number. Start thinking in terms of runnable threads and a run queue.

CPU usage tells you how busy the cores were on average. It does not tell you whether your work waited to get on a core. Latency lives in that wait. Once you think about the queue instead of the percentage, the confusing cases make sense. A load average past your core count is just a long queue. A background job hurting your API is just competition for cores. A container throttled with "idle" cores is just its CPU quota running out mid-slice.

The scheduler is not misbehaving in any of these situations. It is doing its job. It shares a fixed set of cores among more threads than fit at once. One reminder when you chase tail latency: the scheduler is not the only source of pauses. A stop-the-world garbage collection pause can stall a request the exact same way a full run queue can.