What's Infiniband?

InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. Infiniband host bus adapters and network switches are manufactured by Mellanox and Intel (which acquired Qlogic's infiniband business in January 2012[1]).


InfiniBand forms a superset of the Virtual Interface Architecture (VIA).
Description


http://www.tmcscsi.com/QSFP_cables.shtml

Effective theoretical throughput

(actual data rate, not signaling rate)

SDR DDR QDR FDR EDR

1X 2 Gbit/s 4 Gbit/s 8 Gbit/s 13.64 Gbit/s 25 Gbit/s

4X 8 Gbit/s 16 Gbit/s 32 Gbit/s 54.54 Gbit/s 100 Gbit/s

12X 24 Gbit/s 48 Gbit/s 96 Gbit/s 163.64 Gbit/s 300 Gbit/s

Like Fibre Channel, PCI Express, Serial ATA, and many other modern interconnects, InfiniBand offers point-to-point bidirectional serial links intended for the connection of processors with high-speed peripherals such as disks. On top of the point to point capabilities, InfiniBand also offers multicast operations as well. It supports several signaling rates and, as with PCI Express, links can be bonded together for additional throughput.

[edit]Signaling rate

An Infiniband link is a serial link operating at one of five data rates, named SDR ("single data rate"), DDR ("double data rate"), QDR ("quad data rate"), FDR, and EDR.

The SDR connection's signalling rate is 2.5 gigabit per second (Gbit/s) in each direction per connection. DDR is 5 Gbit/s and QDR is 10 Gbit/s. FDR is 14.0625 Gbit/s and EDR is 25.78125 Gbit/s per lane.

For SDR, DDR and QDR, links use 8B/10B encoding — every 10 bits sent carry 8bits of data — making the effective data transmission rate four-fifths the raw rate. Thus single, double, and quad data rates carry 2, 4, or 8 Gbit/s useful data, respectively. For FDR and EDR, links use 64B/66B encoding — every 66 bits sent carry 64 bits of data. (Neither of these calculations takes into account the additional physical layer overhead requirements for comma characters or protocol requirements such as StartOfFrame and EndOfFrame).

Implementers can aggregate links in units of 4 or 12, called 4X or 12X. A 12X QDR link therefore carries 120 Gbit/s raw, or 96 Gbit/s of useful data. As of 2009 most systems use a 4X aggregate, implying a 10 Gbit/s (SDR), 20 Gbit/s (DDR) or 40 Gbit/s (QDR) connection. Larger systems with 12X links are typically used for cluster and supercomputer interconnects and for inter-switch connections.

The InfiniBand future roadmap also has "HDR" (High Data rate), due in 2014, and "NDR" (Next Data Rate), due "some time later", but as of June 2010, these data rates were not yet tied to specific speeds.[2]

[edit]Latency

The single data rate switch chips have a latency of 200 nanoseconds, DDR switch chips have a latency of 140 nanoseconds and QDR switch chips have a latency of 100 nanoseconds. The end-to-end latency range spans from 1.07 microseconds MPI latency (Mellanox ConnectX QDR HCAs) to 1.29 microseconds MPI latency (Qlogic InfiniPath HCAs) to 2.6 microseconds (Mellanox InfiniHost DDR III HCAs).[citation needed] As of 2009 various InfiniBand host channel adapters (HCA) exist in the market, each with different latency and bandwidth characteristics. InfiniBand also provides RDMA capabilities for low CPU overhead. The latency for RDMA operations is less than 1 microsecond (Mellanox ConnectX HCAs).

[edit]Topology

InfiniBand uses a switched fabric topology, as opposed to a hierarchical switched network like traditional Ethernet architectures, although emerging Ethernet fabric architectures propose many benefits which could see Ethernet replace InfiniBand.[3] Most of the network topologies are Fat-Tree, mesh or 3D-Torus. Recent papers (ISCA'10) demonstrated butterfly topologies (Clos) as well.[citation needed]

All transmissions begin or end at a "channel adapter." Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service (QoS).什麼是Infiniband？


Infiniband是一種高速匯流排標準，曾被認為是將廣泛用於各種系統的下一代互連技術，但現在其應用侷限於伺服器叢集和儲存網路；在這些領域，Infiniband因為得到包括IBM、SGI和Sun等在內的一些公司的支援而獲得了發展動力。該標準主要是希望能取代PCI匯流排，可支援的資料頻寬從500MB/sec變化到6GB/sec，線速為2.5Gbps。
 
從InfiniBand到PCIe


InfiniBand曾一度被視為FC介面的有力競爭者，但經過多年發展，InfiniBand的儲存應用仍未能打開局面，僅在高效能運算領域較為成功（作為叢集伺服器之間、以及伺服器與儲存設備間的連接），未能成功進入企業應用。
儘管InfiniBand未獲企業普遍應用，但企業對於更高頻寬儲存匯流排的需求仍然存在，如一些希望透過導入固態硬碟藉以提高I/O效能的用戶，以及一些從事影音後製或串流服務的用戶，都需要更大的儲存傳輸頻寬，但透過匯集多條8G FC或10GbE成為單一大頻寬通道所帶來的高成本，又讓人難以接受。