What is the difference between latency and throughput?
Latency is how long one request takes; throughput is how many requests the system finishes per unit of time. You can improve one without the other, and pushing throughput too hard often raises latency.
Explain it simply
Think of a highway toll booth. Latency is how long one car waits from arriving to leaving. Throughput is how many cars pass per minute. Opening more booths raises throughput, but each car still spends the same time at its booth, so latency is unchanged.
Latency is measured in milliseconds per request. Throughput is measured in requests per second (RPS or QPS), or bytes per second for data transfer. They are linked: as a system nears its maximum throughput, requests queue up and latency climbs sharply, like a toll plaza at Diwali rush.
Example
An API server handles each request in 50 ms using one worker.
1 worker : 1000 ms / 50 ms = 20 requests per second
10 workers: 10 x 20 = 200 requests per second
Latency per request: still about 50 msAdding workers raised throughput tenfold with no latency gain. To cut latency you would need a faster query, a cache, or a closer server.
What to say in the interview
"Latency is the time for one request, throughput is the number of requests handled per second. Adding servers mostly raises throughput; caching and shorter network paths lower latency. Near saturation, queues build and latency rises, so I design with headroom and track p95 and p99 latency, not only the average."
Common mistakes
- Using the two words as if they meant the same thing.
- Quoting only average latency; slow outliers hide in the average.