Troubleshooting Distributed Testing

If you have made it this far and your distributed test worked on the first try, you are either very lucky or very experienced. For the rest of us, here is the troubleshooting guide I wish I had when I started. Every error in this list is one I have personally encountered and debugged with chai in hand and frustration in heart.

Error 1: Connection Refused

The single most common error. You start the test and see: "java.rmi.ConnectException: Connection refused to host: 192.168.1.101"

# Step 1: Is jmeter-server running on the worker?
ssh user@192.168.1.101 "ps aux | grep jmeter-server"

# Step 2: Can you reach the worker port from master?
telnet 192.168.1.101 1099
# Or if telnet is not installed:
nc -zv 192.168.1.101 1099
# Expected: "Connected" or "Connection succeeded"
# If timeout/refused: firewall or process not running

# Step 3: Check what IP jmeter-server is bound to (on worker)
netstat -tlnp | grep 1099
# Look for 0.0.0.0:1099 (listening on all interfaces) or specific IP

# Step 4: Verify no DNS resolution issues
nslookup 192.168.1.101
ping -c 3 192.168.1.101

Error 2: SSL Handshake Failure

You see: "javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake" or "java.rmi.ConnectIOException: non-JRMP server at remote endpoint"

• SSL is enabled on master but disabled on worker (or vice versa) -- make server.rmi.ssl.disable consistent on ALL machines
• rmi_keystore.jks is missing on one machine -- copy the same keystore to all machines
• Different keystores on different machines -- generate once, distribute the same file everywhere
• Keystore expired -- regenerate with ./create-rmi-keystore.sh and redistribute
• JMeter version mismatch -- older versions may not have SSL enabled by default. Use the same version everywhere

# Check if SSL is enabled/disabled on each machine
grep "server.rmi.ssl.disable" /opt/jmeter/bin/jmeter.properties
# Should return the same value on ALL machines

# Check if keystore exists on each machine
ls -la /opt/jmeter/bin/rmi_keystore.jks
# Should exist on every machine with the same file size/checksum

# Compare keystore checksums across machines
md5sum /opt/jmeter/bin/rmi_keystore.jks
# All machines must return the same hash

# Quick fix for test environments: disable SSL everywhere
for host in master worker1 worker2 worker3; do
  ssh user@$host "echo 'server.rmi.ssl.disable=true' >> /opt/jmeter/bin/jmeter.properties"
done

Error 3: Workers Not Responding After Test Starts

The test starts, master says "Starting remote workers," but then nothing happens. No results come back. The test seems hung.

• Timeout too short: increase remote.connect.timeout and remote.response.timeout in jmeter.properties
• Dynamic RMI port blocked: pin server.rmi.localport and open it in the firewall
• Worker ran out of memory during test plan transfer: increase JVM heap on workers
• Test plan is very large (many samplers): workers take time to parse. Wait longer or simplify the plan.
• Network partition: master and workers lost connectivity mid-test. Check network monitoring.

Error 4: OutOfMemoryError on Workers

Workers crash with java.lang.OutOfMemoryError: Java heap space. This means you are running more threads than the worker can handle with its current memory allocation.

# Increase JMeter heap memory on workers
# Option 1: Set environment variable before starting
export JVM_ARGS="-Xms2g -Xmx4g -XX:MaxMetaspaceSize=256m"
./jmeter-server

# Option 2: Edit jmeter script directly
# In bin/jmeter file, find the HEAP line and change:
HEAP="-Xms2g -Xmx4g"

# Option 3: For very heavy tests
export JVM_ARGS="-Xms4g -Xmx8g -XX:MaxMetaspaceSize=512m -XX:+UseG1GC"
./jmeter-server

# Rule of thumb:
# 1 GB heap per 500 medium-complexity threads
# Monitor with: jcmd <pid> GC.heap_info

Error 5: Inconsistent Results Across Workers

One worker shows 200ms average response time while another shows 2,000ms. Before blaming the application, check these worker-side issues.

• Unequal hardware: one worker has 4 cores while another has 8. The weaker machine is overloaded.
• Different network paths: workers on different subnets may have different latency to the target
• One worker has more threads than it can handle (check if its CPU is maxed)
• CSV data distribution issue: one worker is using the same data as another, causing cache hits on the server for one and cache misses for another
• Clock drift between workers: if timestamps are off, aggregated results will look wrong. Sync clocks with NTP.

Error 6: CSV File Not Found on Worker

ERROR - jmeter.threads.JMeterThread: Test failed!
java.lang.IllegalArgumentException: File /home/user/test-data/users.csv must exist and be readable
  at org.apache.jmeter.services.FileServer.resolveFileFromPath

• The CSV file path in the test plan does not exist on the worker machine
• Fix: Use relative paths (place CSVs in JMeter bin directory) or ensure the same absolute path exists on all machines
• Better fix: Create a standardized directory structure on all machines (e.g., /opt/jmeter/test-data/) and always put CSV files there
• Best fix: Use an automation script that deploys test resources to all workers before every test run

Master Troubleshooting Checklist