My infiniband + sandybridge experience, presented at InfiniBand Day 03. Held on 9th May, 2011.

1. DIY
InfiniBand Networking
InfiniBand Day 03 : 9th May, 2011
@syoyo

3. Global illumination
Local Illumination Global Illumination
x100 CPU power
CONFIDENTIAL
x10 I/O

4. We need BandWidth!

5. • 4 SandyBridge CPUs
• 4 2-port IB QDR HCA
• Scientific Linux 6
QDR
QDR QDR
QDR
QDR
To server

6. Why InfiniBand?
• Faster networking
• x10 ~ x40 faster than 1GbE.
• Cost-effective!
• $100 for IB 10Gbps(SDR) HCA
• We need at least 1GB/s transfer

7. Why DIY?
• No vendor seems did not yet provide
support for SandyBridge(AVX) + IB QDR
combination.
• Kernel 2.6.32~(RHEL6 or Ubu 10.4 ~) is
required for AVX support.
• SandyBridge IB
.

8. What you need
• HW
• IB HCA
• IB Cable
• IB Switch(optional for small network)
• SW
• Driver suite(OFED, WinOF)

9. OFED
• OSS IB driver suite
• 1.5.3.1 as of May 2011.
• IPoIB CM is now default connection
mode
• Accelerate TCP/IP app without
modification.

10. Linux + OFED 1.5.3.1
confirmed distro
• RHEL6
• Scientific Linux6
• SRPT needs recent scst svn trunk
• Ubuntu
• 10.04, 10.10

11. Performance
&
Functionalities
&
Interoperability

12. MPI
• Confirmed
• 3.2 GB/s unidirectional bandwidth!
• Hits 5GT/s x 8 PCI-ex BW peak.
• 1 ~ 2 us latency!

13. IPoIB
• IP over InfiniBand
• Accelerate existing TCP/IP app without
modification
• Still requires larger CPU time.
• 930 MB/s BW confirmed with IB SDR
HCA(With Connected Mode).

14. SRP(iSCSI + RDMA)
• 700 ~ 800 MB/s BW confirmed on IB SDR
HCA
• iSCSI + IPoIB CM also achieves same
performance.
• Even though larger CPU time was
consumed.

15. WinOF 2.3 OFED 1.5.3.1
Confirmed :
IPoIB, SRP(iSCSI)

16. SRP iSCSI
Client: Win7
SRPT : Linux
IB SDR
tmpfs

17. FlexBoot
• Network boot from IB HCA
• Faster boot image xfer?
• We just confirmed FlexBoot works on
ConnectX HCA.

18. Conclution

19. Pros
• Fast!
• Cost effective!
• IPoIB CM shows good performance.
• Accelerates TCP/IP app without
modification.
• Win <-> Lin seems now stable in OFED
1.5.3.1.

20. Cons
• Hard to install OFED drivers
• On newer distribution, newer system
configuration.
• Kernel recompilation required for
• SRPT
• IPoIB CM seems not available on Windows
yet

21. ToDo
• InfiniBand virtualization
• e.g. 4 VMs sharing One 40 Gbps IB HCA
possible(10 Gbps per VM)
• InfiniBand-ready distributed storage system
• Glusterfs, NFS/RDMA
• Diskless boot with FlexBoot.

22. InfiniBand @ HOME
• InfiniBand?

23. mini ITX ...

24. IB QDR
...

25. IB (4 )

26. Many thanks goes to:
• @hiroysato FlexBoot advice
• @yitabashi IB
• @takahashi9854 InfiniBand