diff options
Diffstat (limited to 'docs/network.txt')
| -rw-r--r-- | docs/network.txt | 23 |
1 files changed, 23 insertions, 0 deletions
diff --git a/docs/network.txt b/docs/network.txt index a164d36..bcd45e1 100644 --- a/docs/network.txt +++ b/docs/network.txt @@ -56,3 +56,26 @@ Hostnames The linux host name (uname -a) should match the hostnames assigned to openshift nodes. Otherwise, the certificate verification will fail. It seems minor issue as system continue functioning, but better to avoid. The check can be performed with etcd: etcdctl3 --key=/etc/etcd/peer.key --cacert=/etc/etcd/ca.crt --endpoints="192.168.213.1:2379,192.168.213.3:2379,192.168.213.4:2379" + +Performance +=========== + - Redhat recommends using Native Container Routing for speeds above 1Gb/s. It creates a new bridge connected to fast fabric and docker + configured to use it instead of docker0 bridge. The docker0 is routed trough the OpenVSwich fabric and the new bridge should go directly. + Unfortunatelly, this is not working with Infiniband. IPoIB is not fully Ethernet compatible and is not working as slave in bridges. + * There is projects for full Ethernet compatibility (eipoib) providing Ethernet L2 interfaces. But it seems there is no really mature + solution ready for production. It also penalyzes performance (about 2x). + * Mellanox cards working in both Ethernet and Infiniband modes. No problem to select the current mode with: + echo "eth|ib|auto" > /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port1 + However, while the switch support Ethernet, it requires additional license basically for 50% of the original switch price (it is about + 4 kEUR for SX6018). License is called: UPGR-6036-GW. + + - Measured performance + Standard: ~ 3.2 Gb/s + Standard (pods on the same node) ~ 20 - 30 Gb/s + hostNet (using cluster IP ) ~ 3.6 Gb/s + hostNet (using host IP) ~ 12 - 15 Gb/s + + - So, I guess the optimal solution is really to introduce a second router for the cluster, but with Ethernet interface. Then, we can + reconfigure the second Infiniband adapter for the Ethernet mode. The switch to native routing should be possible also with running + cluster with short downtime. As temporary solution, we may use hostNetwork. + |