The sweet spot of any Magento project from hosting standpoint is when traffic patterns allow that all services store in question needs, fit on a single more or less powerful machine. Having that singe IP address you connect to in order to deploy code, do system upgrades when traffic is low and monitor when traffic is high is super convenient. But as store grows in terms of traffic, sooner or later you'll be forced to think horizontally. This usually means adding load balancer, couple of frontend nodes as well as dedicated servers for other services your store requires. In other words, you need more machines configured in private network working together in order to serve more traffic. There are many challenges to setting up and maintaining such system, with having fast network through which your nodes communicate being the most important component of your infrastructure.

First thing you need to watch out for is the network latency. So when you go ahead and ping nodes over private network, anything over half of millisecond indicates this network is practically useless for the purpose of wiring your infrastructure. Lets do some math here. Lets assume you'll have separate Redis server in order to store Magento cache, and dedicated MySQL server as your persistent data storage. These figures can vary greatly depending on Magento version and custom code, but lets assume that average PHP request must do 300 calls to cache storage and 100 queries to database depending on the area of store being accessed. Therefore if network through which your frontend node communicates with Redis and MySQL has an average latency of 1 ms, you'll lose 400 ms on your TTFB simply due to network latency between nodes in your private network. Ouch. Additionally, if you don't have enough luck in the bandwidth department as well, as your private network approaches its limits, your network latency will increase further.

All of these things are something you must investigate as part of deciding where you'll be assembling your private cloud, for the purpose of basic testing, simple iperf and ping will do. So the idea is to push some traffic between your nodes to both test bandwidth limits, as well as relation between latency and bandwidth in your private network. Lets assume we have 3 nodes:

1. 192.168.0.1 - Frontend node 1 (e.g. PHP-FPM)
2. 192.168.0.2 - Cache storage node (e.g. Redis)
3. 192.168.0.3 - Persistent storage node (MySQL)

To begin with you'll need to SSH into each node, and make sure you have iperf3 there (consult your Linux distro docs on how to do this).

Step 1 - put up an iperf servers on frontend nodes

Frontend node 1

We put up an iperf3 servers on our frontend nodes, -s switch means server:

ssh root@192.168.0.1
root@frontend1:~# iperf3 -s

Step 2 - Push traffic to your frontend nodes from both Redis and MySQL nodes

Here we attempt to simulate what will happen when our store goes to production, and when our web nodes start receiving traffic from cache storage and database nodes. In order to do that, we bring up the iperf3 clients, and configure them for 50 concurrent connections each doing 10Mb of traffic per second for 10 minutes in total. And we do this for both Redis and MySQL nodes and every frontend node we'll have, in our case just one frontend node. Basically we'll be pushing a lot of traffic around and measuring network latency which should give us general idea how network behaves under stress. Don't forget to adjust the -P (concurrent connections), and -b (bandwidth each) parameters according to your network bandwidth limits and total number of frontend nodes at your disposal.

Redis node

The -c switch to iperf3 means we're starting in client mode, lets do that for our Redis node:

ssh root@192.168.0.2
root@redis:~# iperf3 -c 192.168.0.1 -P 50 -b 10m -t 1200 > /dev/null 2>&1 &

MySQL node

We also bring up client on our MySQL node:

ssh root@192.168.0.3
root@mysql:~# iperf3 -c 192.168.0.1 -P 50 -b 10m -t 1200 > /dev/null 2>&1 &

Step 3 - Ping Redis and MySQL nodes

Now ping your Redis and MySQL nodes from one of your frontend nodes, and save the output to files for later inspection:

ssh root@192.168.0.1
ping -w 600 192.168.0.2 > redis.ping 2>&1 &
ping -w 600 192.168.0.3 > mysql.ping 2>&1 &

Examining the output of ping while your network is under stress is extremely important before going forward. It's much better knowing that private network is poorly assembled or congested early in the process, then it is with infrastructure being hot and with customers on frontend.