Network and Bandwidth Considerations

In distributed testing, the network becomes your most important -- and most overlooked -- infrastructure component. I have seen teams spend weeks perfecting their test scripts only to get garbage results because they did not account for network topology. Let me share the pitfalls and the math.

Two Networks, Two Concerns

In distributed testing, there are two separate networks to worry about. First, the control plane: the RMI communication between master and workers (test plan distribution, result streaming). Second, the data plane: the actual test traffic from workers to the target application. These two should ideally be on separate network paths so they do not compete for bandwidth.

Bandwidth Math: Do It Before the Test

Here is a quick formula that can save you from embarrassing failures. If each virtual user makes 5 requests per second, and the average response size is 50 KB, then each user consumes 250 KB/s of download bandwidth. For 1,000 users on one worker, that is 250 MB/s -- already exceeding a 1 Gbps NIC (which gives about 125 MB/s actual throughput). You hit the network ceiling before you hit CPU or RAM limits.

Per-Worker Bandwidth = Users x Requests/sec x Avg_Response_Size

Example:
- 500 users per worker
- 4 requests per second per user
- Average response size: 30 KB
- Bandwidth = 500 x 4 x 30 KB = 60,000 KB/s = 60 MB/s
- A 1 Gbps NIC handles ~125 MB/s, so this is fine.

Total Test Bandwidth = Per-Worker Bandwidth x Number of Workers
- With 5 workers: 60 x 5 = 300 MB/s hitting target
- Target application must handle this incoming traffic!

Do not forget:
- SSL/TLS adds ~5-10% overhead
- TCP headers add ~40 bytes per packet
- Result streaming adds bandwidth on the control plane

Network Latency Between Workers and Target

If your workers are on the same LAN as the target application, network latency is sub-millisecond and essentially a non-factor. But if your workers are in a different data center or cloud region, network latency directly adds to every response time measurement. This is not necessarily bad -- it may actually reflect real user experience -- but you need to account for it.

• Same data center (LAN): <1ms latency -- isolates application performance from network effects
• Same cloud region, different availability zone: 1-5ms -- realistic for most deployments
• Cross-region (e.g., US-East to US-West): 50-80ms -- adds 100-160ms to each request (round trip)
• Cross-continent: 100-300ms -- your response times will be dominated by network, not application
• Best practice: place workers in the same region as the target unless you specifically want to test geographic distribution

Common Network Pitfalls

• Shared network switch between workers and target -- all traffic competes for the same backplane bandwidth
• Workers behind a NAT -- the target sees all traffic from one IP, which may trigger rate limiting or load balancer stickiness
• DNS resolution delays -- configure workers to use the target IP directly, or verify DNS caching is working
• TCP connection limits on workers -- increase ulimit for open files (ulimit -n 65536)
• Ephemeral port exhaustion -- reduce TIME_WAIT with net.ipv4.tcp_tw_reuse=1 on Linux

# Increase max open files (for more concurrent connections)
ulimit -n 65536

# Or permanently in /etc/security/limits.conf:
# jmeter_user soft nofile 65536
# jmeter_user hard nofile 65536

# Enable TCP reuse to avoid port exhaustion
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# Increase local port range
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

# Increase connection backlog
sudo sysctl -w net.core.somaxconn=65535

# Increase network buffer sizes
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216