QLogic 10GbE network solutions from HP delivers significant operational savings, high performance, reduced management complexity and improved scalability for virtualized data center deployments.
2. ii SN0054584-00 A
Reference Guide—Efficient Data Center Virtualization with QLogic 10GbE Solutions from HP®
Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no
responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its
use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications
described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no
representation nor warranty that such applications are suitable for the specified use without further testing or
modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document.
Document Revision History
Revision A, December 15, 2011
Changes Sections Affected
Initial release All
5. SN0054584-00 A v
Preface
Overview
Designing an efficient data center to support high performance and scalable
virtualized server deployment creates a number of challenges when using a
legacy 1-gigabit Ethernet (GbE) network I/O infrastructure. This document
identifies the challenges of the 1GbE legacy network infrastructures for virtualized
data centers and presents a reference architecture solution using QLogic’s 10Gb
Intelligent Ethernet Adapter with HP infrastructure.
The reference architecture discussed in this document uses proven HP and
QLogic 10GbE networking technology solutions currently available in the market.
System architects can leverage the reference architecture to plan, migrate, and
optimize their designs, maximizing data center operational efficiency and savings.
As data centers continue to increase the number of virtualized hosts on a physical
server, the demands for higher efficiency, performance, and consolidation creates
critical challenges with 1GbE legacy networks:
Low network bandwidth caps the scaling of virtualized hosts and services.
Inefficient usage of server PCI slots limits I/O consolidation.
Increased cabling, switch ports, and adapters add management complexity
and costs.
Peak network performance is reduced for on-demand applications and
services.
The HP-QLogic 10GbE network solution eliminates the I/O bottlenecks, delivering
significant operational savings, high performance, reduced management
complexity, and improved scalability for virtualized data center deployments.
Key Benefits
Efficient data center virtualization has the following key benefits:
Reduced management complexity
Improved operational efficiency
High-performance 10GbE network
Scalable virtual hosts and services
7. SN0054584-00 A 1-1
1 Reference Architecture
Model
System Architecture
Figure 1-1 compares a legacy deployment with 1GbE versus the efficient
deployment with 10GbE.
Figure 1-1. Comparison of a Legacy Deployment Versus an Efficient Deployment
8. 1–Reference Architecture Model
Network Topology
1-2 SN0054584-00 A
Table 1-1 compares the components used for the legacy deployment with the
components used for the efficient deployment.
Network Topology
There are two network topologies described in this document:
“Typical Legacy 1GbE Topology” on page 1-3
“Efficient 10GbE Topology” on page 1-5
Table 1-1. 1GbE and 10GbE Components for Reference Network
Topology
Component
Legacy Deployment with
1GbE
Efficient Deployment with 10GbE
Model Units Model Units
Server HP DL380 G7 1 HP DL380 G7 1
NICs Legacy 1 GbE
NIC ports
14 HP NC523 SFP (QLogic)
ports
2
LAN switch Legacy 1GbE
Ethernet switch
2 switches;
14 switch
ports
HP A5820X-24XG-SFP+
10GbE
2 switches;
2 switch
ports
9. 1–Reference Architecture Model
Network Topology
SN0054584-00 A 1-3
Typical Legacy 1GbE Topology
The 1GbE network topology shown in Figure 1-2 uses the legacy 1GbE network
interfaces with VMware®
ESX®
virtualization environment and multiple 1GbE links.
An HP DL380 G7 server with four 1GbE LAN-on-motherboard (LOM) ports and
five dual-port 1GbE NICs uses five of the available six PCIe®
slots on the server.
Figure 1-2. Typical Legacy 1GbE Topology
Twelve virtual machines (VMs) or hosts running various application services (such
as a Web server, a file server, and database applications) were considered for this
reference architecture model (see Figure 1-2). ESX back-end services (such as
NFS, IP storage, and VMotion®
) provide access to the storage and high
availability required for enterprise deployments.
10. 1–Reference Architecture Model
Network Topology
1-4 SN0054584-00 A
The ESX virtual standard switch is configured with multiple port groups (1–6) to
support the various networking functions and application groupings. Each port
group is supplied with one or more physical 1GbE connections. Virtual LAN
(VLAN) tags (A–F) are implemented on the port groups for segmenting network
traffic. Each VM connects to the data center network through one or more ESX
virtual NICs (VNICs), and each VNIC connects to one or more virtual switch
(vswitch) port groups.
Network bandwidth and high-availability requirements of the various traffic types
(see Table 1-2) on the 1GbE physical uplinks are either configured for failover for
redundancy or static link aggregation control protocol (LACP) for higher
bandwidth.
The management network is served by the ESX VMkernel network interface
connected to a pair of 1GbE uplinks connected by port group 6 on the ESX virtual
standard switch. The management traffic is carried over VLAN F through a
redundant pair of 1GbE links (failover configuration) connected to a redundant set
of 1GbE switches. While the solid green line is the primary I/O path, the dashed
green line depicts a failover path for the network traffic (see Figure 1-2).
Two legacy 1GbE switches interconnected to each other create a high-availability
configuration that prevents network traffic disruption if one of the 1GbE switches
fail. The 1GbE architecture has limited network bandwidth; it is unable to scale to
meet the peak network bandwidth requirements for on-demand applications. The
reason for this is that most PCIe slots are already occupied, and the network
cannot scale with more VMs and application services, which demand additional
network bandwidth that cannot increase because of the limitation of additional
PCIe slots.
Table 1-2. Network Bandwidth Sizing for Legacy 1GbE Reference
Architecture
Traffic Types for VMware ESX
Provisioned
1GbE Network
Ports
Available
Bandwidth
Target
Network
Bandwidth
Management traffic 2 (with failover) 1Gbps 1Gps
VMotion traffic 1 1Gbps 4Gbps
IP storage traffic 1 1Gbps 2Gbps
NFS traffic 1 1Gbps 2Gbps
Fault tolerance (FT) traffic 1 1Gbps 2Gbps
VM traffic (for 12 VMs) 8 (with failover) 6Gbps 8Gbps
Total 14 11Gbps 19Gbps
11. 1–Reference Architecture Model
Network Topology
SN0054584-00 A 1-5
There are 14 cables originating from one server using seven switch ports per
switch, increasing operational cost and management complexity.
Efficient 10GbE Topology
The 10GbE network topology with ESX deployment shown in Figure 1-3 uses two
10GbE uplinks. The HP DL380 G7 server with one dual-port 10GbE NIC uses
only one of the available six PCIe slots on the server instead of the five slots used
in the legacy network.
Figure 1-3. Efficient 10GbE Topology
The legacy 1GbE topology and efficient 10GbE topology both use the same
virtualized model with 12 virtual machines and associated ESX back-end
services, vswitch, port groups, and VLAN. The 10GbE network model yields an
efficient and scalable network for a virtualized data center.
12. 1–Reference Architecture Model
Network Topology
1-6 SN0054584-00 A
The architecture uses the two 10GbE uplinks aggregated (static LACP) to provide
a single consolidated 20GbE uplink for the ESX standard virtual switch. The
network bandwidth sizing table (see Table 1-3) categorizes various traffic types
and the allocation of the 20Gb network bandwidth.
VMware ESX traffic types (such as VMotion) that were limited to 1-2Gbps in the
legacy 1GbE deployment can now leverage the increased available bandwidth,
resulting in a more efficient VMotion for live migration. IP storage, NFS traffic, and
the storage access needs of the virtual machines are no longer limited to the
number of 1GbE adapters dedicated to the traffic type.
Two HP 5800 series 10GbE network switches replace the legacy 1GbE switches
and provide high-speed and high-availability network traffic configuration (see
Figure 1-3).
The obvious advantage of the 10GbE solution is that it reduces the overall
physical connections, simplifying infrastructure management. Fewer physical
ports reduce the wiring complexity and improve infrastructure reliability.
With just one PCIe slot occupied and significantly higher bandwidth than the
1GbE legacy solution, the 10GbE architecture can scale to accommodate more
VMs and higher network bandwidth requirements for peak demands. This
consolidation improves the efficiency of the architecture.
Table 1-3. Network Bandwidth Sizing for the Efficient 10GbE
Reference Architecture
Traffic Types for VMware ESX Provisioned 10GbE
Network Ports
Available
Bandwidth
Management traffic
2
1Gps
VMotion traffic 4Gps
IP storage traffic 2Gps
NFS traffic 2Gps
FT traffic 2Gps
Virtual machine traffic (for 12 VMs) 8Gps
Total 2 19Gps
13. SN0054584-00 A 2-1
2 Reference Component List
Table 2-1, Table 2-2, and Table 2-3 provide lists of hardware, software, and
management components required for this reference architecture. See
Appendix A for relevant references.
Table 2-1. Hardware Components
Component HP Part Number Description
Server
HP DL380 G7 583914-B21 Base configuration model; various models are
available with a different selection of system
configurations.
Intelligent Ethernet Adapter
HP NC523SFP 593717-B21 HP ProLiant
®
Ethernet Server adapter
Switch
HP 5800-24G JC100A Fixed four RJ-45 10/100/1000 ports and
four SFP+ 10GbE ports
Table 2-2. Software Components
Component Description Version
Server
VMware ESX Operating system 4.1
Intelligent Ethernet Adapter
Driver-network HP NC-Series QLogic P3P Multifunction
Driver for VMware ESX
4.0.719
Switch
Platform firmware HP ProCurve
®
switch firmware and boot
code
A5800_5.20.R1211
14. 2–Reference Component List
2-2 SN0054584-00 A
Table 2-3. Management Components
Component Description Version
Server
HP CIM Provider HP Common Information Model 1.1(a) (1
Mar 2011)
VMware vCenterTM
VMware ESX Management Suite 4.1.0 Build
258902
Intelligent Ethernet Adapter
QLogic vCenter
Plugin
vCenter Plugin TBD (Yet to
be posted)
QLogic CIM Pro-
vider
QConvergeConsole
®
Management CIM Provider
for HP-branded QLogic-based Intelligent Ethernet
Adapters
TBD (Yet to
be posted)
HP NC-Series
QLogic P3P Flash
Update Kit
A bootable FreeDOSTM
operating system along
with utilities required to Flash HP P3+-based
adapters
1.00
Switch
Microsoft
®
Win-
dows
®
-based net-
work management
HP Intelligent Management Center (IMC) platform
for HP network switches
5.0
15. SN0054584-00 A 3-1
3 Optimizing the Virtualized
Data Center
The HP-QLogic 10GbE Solution
The HP 10GbE solution comprises the HP-QLogic NC523SFP Dual Port 10GbE
Server Adapter hosted in a HP ProLiant DL380 G7 enterprise class server
connected to a set of HP A5820 switch series.
HP-QLogic NC523SFP Dual Port 10GbE Server Adapter
The NC523SFP is an eight-lane (x8) PCI Express®
Generation 2 (PCIe Gen2)
10Gb network solution offering superior bandwidth in a ProLiant Ethernet adapter.
This dual-port PCI Express Gen2 adapter supports SFP+ connectors, requiring
either direct attach cable (DAC) for copper environments or fiber transceivers and
fiber cables for fiber optic environments.
The NC523SFP ships with advanced server features such as support for TCP
checksum and large segment offload (LSO) capability, VLAN tagging, jumbo
frames, IPv6, and more.
The two ports on the NC523SFP have a theoretical maximum of 40Gbps
full-duplex, bidirectional throughput.
Two ports save valuable I/O slots for other cards and reduce cabling
compared to the 1GbE legacy architecture.
Ports can be configured for failover, load balancing, or connecting to a
separate network.
The NC523SFP is ideal for enterprise customers in a VMware ESX environment
because it provides the following advantages:
The best combination of high speed and low latency for virtualization makes
it the ideal choice when migrating from a 1GbE solution to a 10GbE solution.
It reduces the number of cables and switch ports needed by using a larger
10GbE pipe in the infrastructure.
16. 3–Optimizing the Virtualized Data Center
The HP-QLogic 10GbE Solution
3-2 SN0054584-00 A
The ability to provide a higher virtual machine density by meeting and
exceeding the network bandwidth demands of an increasing number of
virtual machines.
For more information, visit the following Web sites:
http://www.hp.com/go/nc523sfp
http://www.qlogic.com/go/hp
The HP 5800 Switch Series
HP 5800 switches offer a combination of 1Gb and 10GbE port density,
high-availability architecture, and full Layer 2 and Layer 3 dual-stack IPv4 and
IPv6 capabilities. Besides wire-speed line-rate performance on all ports, the
switches include patented intelligent resilient framework (IRF) technology and
rapid ring protection protocol (RRPP), which allow local or geographically
distributed 5800 switches to be interconnected for higher resiliency and
performance. Available in power over Ethernet (PoE) and non-PoE models and
1 RU and 2 RU flex chassis configurations, 5800 switches are built on open
standards and include an open application architecture (OAA) module slot that
enables flexible deployment options for new services. These versatile switches
are ideal for use in the network core for a building or department or as a
high-performance switch in the convergence layer or network edge of enterprise
campus networks. For more information, visit the following Web site:
http://h17007.www1.hp.com/us/en/products/switches/HP_5800_Switch_Series/in
dex.aspx
HP ProLiant DL380 G7 Server
The HP ProLiant DL380 G7 Server provides up to two-socket Intel®
Xeon®
performance in a 2U density. With up to six available PCI Express Gen2
expansion slots (including two x8 slots), it provides for the full use of the
high-performance capabilities of the HP-QLogic NC523SFP Dual Port 10GbE
Server Adapter.
For more information, visit the following Web site:
http://h20195.www2.hp.com/v2/GetPDF.aspx/c02171016.pdf
17. SN0054584-00 A 4-1
4 Building an Efficient
Network Design
Design Considerations
This reference architecture is built according to the following design goals:
Performance
The design should provide sufficient network bandwidth to virtual machines
and back-end services and allow for growth of virtual machines without
bandwidth bottlenecks.
Availability
The design should be capable of recovery from any single points of failure in
the network outside of the VMware ESX server. Traffic should continue to
flow if a single switch, cable, or network adapter port fails.
Isolation
Each traffic type should be logically isolated from every other traffic type.
Consolidation
The design should reduce the number of server PCIe slots occupied,
network cables, and switch ports.
Scalability
The design allows the seamless growth of the number of virtual machines
and the demand for network bandwidth.
Networking Requirements for the Virtual Infrastructure
The key to developing and implementing the 10GbE reference architecture is to
understand how the various networking components of VMware vSphere®
ESX
function and what their requirements are for network performance, isolation, and
high availability.
18. 4–Building an Efficient Network Design
Design Considerations
4-2 SN0054584-00 A
VMware vSphere ESX offers multiple components that relate to Ethernet
networking:
Hypervisor
The hypervisor runs on physical servers and abstracts processor, memory,
storage, and networking resources. HP-QLogic NC523SFP 10GbE Server
Adapter drivers run in the context of the hypervisor.
Virtual standard switch
The virtual standard switch is a software component running under the
control of the hypervisor. It provides the necessary network connectivity for
virtual machines (through the VNIC interface) and back-end services
(through the VMkernel network interface).
Virtual center management server
The virtual center management server allows for configuring, provisioning,
and managing virtualized IT infrastructure.
VMware back-end services
These services include VMware VMotion (which enables virtual machine
migration), VMware FT (which provides a higher level of availability), and
VMware IP storage and NFS (which provide access to storage and NFS
servers).
A VMware vSphere ESX environment involves the following traffic types:
Management
Management traffic goes through the VMkernel management interface on
VMware ESXi. This port is used for all management and configuration and is
the port by which VMware ESX communicates with the VMware vCenter
server. This port generally has very low network usage, but it should always
be available and isolated from other traffic types through a management
VLAN.
VMware VMotion
The VMkernel port is used for migrating a running virtual machine from one
VMware ESX host to another. Using a 10GbE connection, VMotion under
vSphere 4.1 can use up to 8Gbps of aggregate bandwidth with multiple
virtual machines being migrated simultaneously. This traffic is typically
implemented on a separate VLAN specific to VMware VMotion, with no
outside communication required.
19. 4–Building an Efficient Network Design
Network Design with 10GbE
SN0054584-00 A 4-3
Fault-tolerant logging
The VMkernel port for fault-tolerant logging transfers the input network I/O
for the fault-tolerant virtual machine plus the read disk traffic to the
secondary fault-tolerant virtual machine. Traffic varies according to the
network and storage behavior of the application. End-to-end latency
between the fault-tolerant virtual machines should be less than 1ms, and the
dedicated 2Gbps of network bandwidth is considered sufficient. This traffic is
typically implemented on a separate VLAN specific to fault-tolerant logging,
with no outside communication required.
IP Storage
The VMkernel port provides access to a disk subsystem over the IP network.
In VMware ESXi 4.0, two VMkernel ports can be bonded to allow IP storage
traffic over both physical network interfaces. Traffic varies according to I/O;
typically 2Gbps of network bandwidth is considered sufficient. This traffic is
typically implemented on a storage-specific VLAN common to initiators and
targets, although targets may reside on another VLAN accessible through a
Layer 3 gateway.
Network file system
The VMkernel port communicates with NFS files in VMware ESX. Traffic
varies according to I/O, and 1–2Gbps of network bandwidth is considered
sufficient. This traffic is typically implemented on an NFS-specific VLAN,
although files may reside on another VLAN accessible through a Layer 3
gateway.
Virtual machines
Virtual machines vary in number, may be distributed over more than one
VLAN, and may be subject to different policies defined in port profiles and
virtual port groups. The reference architecture design assumes an
aggregate virtual machines network bandwidth demand to be 6–8Gbps.
Network Design with 10GbE
When migrating from a 1GbE network to a 10GbE network, the same or enhanced
level of network isolation and high availability requirements must be maintained
across the legacy 1GbE and efficient 10GbE solutions.
20. 4–Building an Efficient Network Design
Network Design with 10GbE
4-4 SN0054584-00 A
With the HP-QLogic 10Gb network infrastructure solution, the transition from the
legacy design is very simple and straightforward. After the VMware ESX driver for
the NC523SFP has been loaded, the only task is to map the new 10GbE uplinks
to the VMware vswitch and create a static LACP (802.3ad) of the two ports of the
NC523SFP. Each of the two ports of the NC523SFP are physically connected to
two independent HP 5800 switches; this architecture provides for resilience to any
single point of failure. While the use of two 10GbE ports helps to reduce the
solution cost, some organizations may prefer the use of four 10GbE ports to
provide additional bandwidth, additional redundancy, or simply to interface with
existing network infrastructure.
The availability of an aggregate bandwidth of 40Gbps bidirectional across the two
ports of the NC523SFP provides for all the network bandwidth requirements of the
virtual machines and back-end services, as described in Table 1-3.
If planned appropriately, the migration from 1GbE to 10GbE in a VMware ESX
environment provides the benefits described in the following sections.
Increased Performance
10GbE delivers 10 times the bandwidth of 1GbE in a single pipe. Now, rather than
using 8 to 10 1GbE ports in addition to on-board LOMs in each server,
organizations can deploy just two 10GbE NICs and meet the network bandwidth
demands of the virtual infrastructure.
Increased bandwidth per virtual machine
10GbE allows each virtual machine to use bandwidth of more than 1Gbps
(and its increments), helping to increase performance and eliminate the
network as a bottleneck.
Faster VMotion
Using a 10GbE connection, VMotion under vSphere 4.1 can use up to
8Gbps of aggregate bandwidth, which enables virtual machines to move to
new hosts, quickly reducing downtime.
Faster access to IP storage
Using a faster 10GbE link, access speeds to IP storage and NFS servers
are enhanced, improving overall virtual machine performance.
NOTE:
HP supports up to a maximum of two physical NC523SFP adapters in a
supported server.
21. 4–Building an Efficient Network Design
Network Design with 10GbE
SN0054584-00 A 4-5
Simplified Management
Adopting 10GbE in virtualized environments reduces the number of Ethernet
adapter ports to be managed, components to troubleshoot, and components that
can fail.
Ease of management
Consolidating a number of gigabit Ethernet links onto a single 10GbE
connection dramatically decreases the number of management points,
including NICs and their firmware/driver, cables, and switch ports. NIC
management can be executed in any of three ways: through
QConvergeConsole, through the command line interface, or through the
Windows network management utility.
Simplified troubleshooting
Decreasing the number of devices reduces the number of failure points and
devices to troubleshoot, contributing to higher availability.
Increased Scalability
Moving to 10gbE enables the data center’s virtual infrastructure to scale and
accommodate a virtual machine sprawl and associated I/O infrastructure
demands.
Higher virtual machine density
10GbE provides increased headroom that gives applications running in
virtual machines room to scale and the ability to host more virtual machines
per ESX host without network bottlenecks. The increased bandwidth of
10GbE improves scalability for both back-end traffic and production traffic to
virtual machines.
Free PCIe slots
Deploying 10GbE frees up server PCIe slots that were occupied by multiple
1GbE adapters, proving room for expansion.
23. SN0054584-00 A 5-1
5 Conclusion
Deploying HP-QLogic 10GbE technology in virtualized environments based on
VMware vSphere allows enterprises to reduce the complexity and increase the
scalability of the network infrastructure. The 10GbE products from HP and QLogic
enable enterprise customers to meet the performance requirements of virtual
machines without the large numbers of physical server connections required in
1GbE networks.
Increased bandwidth; fewer cables, ports, and network adapters to manage;
fewer PCIe slots used; and the capability to consolidate network I/O on a single
wire complete the argument for deploying the HP-QLogic 10GbE technology in an
efficient data center.
25. SN0054584-00 A A-1
A Implementation Resources
Use the following references for more information about the HP server, Intelligent
Ethernet Adapter, and switch.
User Guide
HP NC523SFP 10Gb 2-port Ethernet Server Adapter User Guide:
http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02653034/c02
653034.pdf
HP-QLogic Product Matrix
http://www.qlogic.com/OEMPartnerships/HP/Documents/matrix_interactive_hba.h
tm
Other Resources
QLogic Converged Network Adapter User's Guide (8200 and 3200 Series):
http://filedownloads.qlogic.com/files/Manual/79506/User_Guide_Converged
_Network_Adaper_8200_and_3200_Series_A.pdf
Server—QuickSpecs for HP ProLiant DL380 G7 Server:
http://h20195.www2.hp.com/v2/GetPDF.aspx/c02171016.pdf
Intelligent Ethernet Adapter:
QuickSpecs for HP NC522SFP Dual Port 10GbE Server Adapter:
http://h18000.www1.hp.com/products/quickspecs/13230_div/13230_di
v.pdf
HP ProLiant Ethernet Adapters—Overview and Features:
http://www.qlogic.com/OEMPartnerships/HP/Pages/HPOEMProducts.
aspx
Switch—QuickSpecs for HP 5820 Switch Series:
http://h18000.www1.hp.com/products/quickspecs/13791_na/13791_na.HT
ML