Linux host to host IP over infiniband

Some tips to configure host to host ip over infiniband

Infiniband allows different server to communicate high speed between each other. When you don’t use a switch to control your network you need to run opensm service on one of the host to manage the network.

# apt-get install opensm
# systemctl enable opensm
# systemctl start opensm

Some modules are required to run infiniband and ip over ib. Add them into /etc/modules

mlx4_ib
ib_ipoib
rdma_ucm
ib_umad

Normally you get interface like this one:

# ip a
ibp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:1e:67:03:00:58:08:39 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

You can setup an IP address

# ip a add 192.168.10.1/24 dev ibp1s0

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.