Cisco Nexus 7000 Series Guide

FlexPod Data Center with VMware vSphere 5.1 and
Cisco Nexus 7000 Design Guide
Last Updated: November 22, 2013
Building Architectures to Solve Business Problems

Cisco Validated Design
2

About the Authors
3
About the Authors
Lindsey Street, Systems Architect, Infrastructure and Cloud Engineering, NetApp
Systems
Lindsey Street is a systems architect in the NetApp Infrastructure and Cloud Engineering
team. She focuses on the architecture, implementation, compatibility, and security of innova-
tive vendor technologies to develop competitive and high-performance end-to-end cloud
solutions for customers. Lindsey started her career in 2006 at Nortel as an interoperability
test engineer, testing customer equipment interoperability for certification. Lindsey has her
Bachelors of Science degree in Computer Networking and her Master's of Science in Infor-
mation Security from East Carolina University.
John George, Reference Architect, Infrastructure and Cloud Engineering, NetApp
Systems
John George is a Reference Architect in the NetApp Infrastructure and Cloud Engineering
team and is focused on developing, validating, and supporting cloud infrastructure solutions
that include NetApp products. Before his current role, he supported and administered
Nortel's worldwide training network and VPN infrastructure. John holds a Master's degree in
computer engineering from Clemson University.
Derek Huckaby, Technical Marketing Engineer, Unified Fabric Switching Services
Product Group, Cisco Systems
Derek Huckaby is a Technical Marketing Engineer for the Nexus 7000 Unified Fabric Switch-
ing products focusing on Nexus 7000 integration into FlexPod designs and Nexus 7000 ser-
vices. Prior to joining the Nexus 7000 Product Marketing team, Derek led the team of
Technical Marketing Engineers for the Data Center Application Services BU within Cisco. He
began his work in network services at Cisco over 13 years ago specializing in application
delivery and SSL termination solutions.
Haseeb Niazi, Technical Marketing Engineer, Server Access Virtualization Business
Unit, Cisco Systems
Haseeb has over 13 years of experience at Cisco dealing in Data Center, Security, and WAN
Optimization related technologies. As a member of various solution teams and advanced
services, Haseeb has helped many enterprise and service provider customers evaluate and
deploy a wide range of Cisco solutions. Haseeb holds a master's degree in Computer Engi-
neering from the University of Southern California.

4
Chris O'Brien, Technical Marketing Manager, Server Access Virtualization Business
Unit, Cisco Systems
Chris O'Brien is currently focused on developing infrastructure best practices and solutions
that are designed, tested, and documented to facilitate and improve customer deployments.
Previously, O'Brien was an application developer and has worked in the IT industry for more
than 15 years.
Chris Reno, Reference Architect, Infrastructure and Cloud Engineering, NetApp
Systems
Chris Reno is a reference architect in the NetApp Infrastructure and Cloud Enablement
group and is focused on creating, validating, supporting, and evangelizing solutions based
on NetApp products. Before being employed in his current role, he worked with NetApp
product engineers designing and developing innovative ways to perform Q and A for
NetApp products, including enablement of a large grid infrastructure using physical and vir-
tualized compute resources. In these roles, Chris gained expertise in stateless computing,
netboot architectures, and virtualization.

5
About Cisco Validated Design (CVD) Program
About Cisco Validated Design (CVD) Program
The CVD program consists of systems and solutions designed, tested, and documented to facilitate
faster, more reliable, and more predictable customer deployments. For more information visit:
http://www.cisco.com/go/designzone
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLEC-
TIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUP-
PLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING
FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS
SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES,
INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF
THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED
OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR
THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR
OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT
THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY
DEPENDING ON FACTORS NOT TESTED BY CISCO.
CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco
WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We
Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS,
Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the
Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital,
the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Cen-
ter, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone,
iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace
Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels,
ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to
Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of
Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners.
The use of the word partner does not imply a partnership relationship between Cisco and any other com-
pany. (0809R)
© 2013 Cisco Systems, Inc. All rights reserved.
NetApp, the NetApp logo, Go further, faster, Data ONTAP, FlexClone, FlexPod, MetroCluster, OnCommand,
SnapLock, SnapMirror, Snapshot, and SnapVault are trademarks or registered trademarks of NetApp, Inc.
in the United States and/or other countries.

6
About Cisco Validated Design (CVD) Program

Corporate Headquarters:
Copyright © 2013 Cisco Systems, Inc. All rights reserve
Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA 95134-1706 USA
VMware vSphere 5.1 on FlexPod with Nexus 7000
Using FCoE
Goal of This Document
Cisco® Validated Designs include systems and solutions that are designed, tested, and documented to
facilitate and improve customer deployments. These designs incorporate a wide range of technologies
and products into a portfolio of solutions that have been developed to address the business needs of our
customers.
This document describes the Cisco and NetApp® FlexPod® solution, which is a validated approach for
deploying Cisco and NetApp technologies as a shared cloud infrastructure.
Audience
The intended audience of this document includes, but is not limited to, sales engineers, field consultants,
professional services, IT managers, partner engineering, and customers who want to take advantage of
an infrastructure built to deliver IT efficiency and enable IT innovation.
Changes in FlexPod
The following design elements distinguish this version of FlexPod from previous models:
•Introduction of the Cisco Nexus® 7000 product family, providing the highest level of availability.
•End-to-end Fibre Channel over Ethernet (FCoE) delivering a unified Ethernet fabric.
•Single wire Cisco Unified Computing System™ Manager (Cisco UCS™ Manager) management for
C-Series M3 servers with the VIC 1225 effectively doubling the server density per I/O module while
reducing cabling cost.
•NetApp clustered Data ONTAP® delivering unified scale-out storage.

8
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Solution Overview
FlexPod Solution Overview
Industry trends indicate a vast data center transformation toward shared infrastructure and cloud
computing. Enterprise customers are moving away from isolated centers of IT operation toward more
cost-effective virtualized environments.
The objective of the move toward virtualization, and eventually to cloud computing, is to increase agility
and reduce costs.
Especially because companies must address resistance to change in both their organizational and
technical IT models, achieving this transformation can seem daunting and complex. To accelerate the
process and simplify the evolution to a shared cloud infrastructure, Cisco and NetApp have developed a
solution called VMware vSphere® on FlexPod.
FlexPod® is a predesigned, best practice data center architecture that is built on the Cisco UCS, the
Cisco Nexus® family of switches, and NetApp Fabric Attached storage (FAS) or V-Series systems.
FlexPod is a suitable platform for running a variety of virtualization hypervisors as well as bare metal
operating systems and enterprise workloads. FlexPod delivers a baseline configuration and also has the
flexibility to be sized and optimized to accommodate many different use cases and requirements.
Figure 1 shows the components used in this solution, while also highlighting the inherent flexibility of
FlexPod.
Figure 1 FlexPod Component Families
This document describes VMware vSphere 5.1 built on the FlexPod model from Cisco and NetApp and
discusses design choices and deployment of best practices using this shared infrastructure platform.

9
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Solution Overview
Customer Challenges
As customers transition toward shared infrastructure or cloud computing, they face a number of
questions, such as:
•How do I start the transition?
•What will my return on investment be?
•How do I build a future-proof infrastructure?
•How do I cost-effectively transition from my current infrastructure?
•Will my applications run properly in a shared infrastructure?
•How do I manage the infrastructure?
The FlexPod architecture is designed to help you answer these questions by providing proven guidance
and measurable value. By introducing standardization, FlexPod helps customers mitigate the risk and
uncertainty involved in planning, designing, and implementing a new data center infrastructure. The
result is a more predictive and adaptable architecture capable of meeting and exceeding customers’ IT
demands.
FlexPod Program Benefits
Cisco and NetApp have thoroughly validated and verified the FlexPod solution architecture and its many
use cases while creating a portfolio of detailed documentation, information, and references to assist
customers in transforming their data centers to this shared infrastructure model. This portfolio includes,
but is not limited to the following items:
•Best practice architectural design
•Workload sizing and scaling guidance
•Implementation and deployment instructions
•Technical specifications (rules for what is, and what is not, a FlexPod configuration)
•Frequently asked questions (FAQs)
•Cisco Validated Designs (CVDs) and NetApp Validated Architectures (NVAs) focused on a variety
of use cases
Cisco and NetApp have also built an experienced support team focused on FlexPod solutions, from
customer account and technical sales representatives to professional services and technical support
engineers. The support alliance provided by NetApp and Cisco provides customers and channel services
partners with direct access to technical experts who collaborate with multiple Vendors and have access
to shared lab resources to resolve potential issues.
FlexPod supports tight integration with virtualized and cloud infrastructures, making it the logical
choice for long-term investment. The following IT initiatives are addressed by the FlexPod solution.
Integrated System
FlexPod is a prevalidated infrastructure that brings together computing, storage, and network to simplify,
accelerate, and minimize the risk associated with data center builds and application rollouts. These
integrated systems provide a standardized approach in the data center supports staff expertise,
application onboarding, and automation, as well as operational efficiencies that are important for
compliance and certification.

10
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Solution Overview
Fabric Infrastructure Resilience
FlexPod is a highly available and scalable infrastructure that IT can evolve over time to support multiple
physical and virtual application workloads. FlexPod has no single point of failure at any level, from the
server through the network, to the storage. The fabric is fully redundant and scalable and provides
seamless traffic failover should any individual component fail at the physical or virtual layer.
Fabric Convergence
FlexPod components are interconnected through the Cisco Unified Fabric network architecture, which
supports both traditional LAN traffic and all types of storage traffic, including the lossless requirements
for block-level storage transport over FC or FCoE. The Cisco Unified Fabric creates high-performance,
low-latency, and highly available networks, serving a diverse set of data center needs.
FlexPod uses the Cisco Unified Fabric to offer a wire-once environment that accelerates application
deployment. Cisco Unified Fabric also offers the efficiencies associated with infrastructure
consolidation, including:
•Cost savings from the reduction in switches (LAN/SAN switch ports), associated cabling, rack
space, all of which reduce capital expenditures (capex)
•Cost savings on power and cooling, which reduce operating expenses (opex)
•Migration to the faster 10 Gigabit Ethernet network, and in the future, to 40 Gigabit Ethernet and
100 Gigabit Ethernet
•Evolution to a converged network with little disruption to operations
•FlexPod with Cisco Unified Fabric helps you preserve investments in existing infrastructure,
management tools, and staff training and expertise
•Simplified cabling, provisioning, and network maintenance to improve productivity and operational
models
Network Virtualization
FlexPod delivers the capability to securely connect virtual machines into the network. This solution
allows network policies and services to be uniformly applied within the integrated compute stack using
technologies such as virtual LANs (VLANs), Quality of Service (QoS), and the Cisco Nexus 1000v
virtual distributed switch. This capability enables the full utilization of FlexPod while maintaining
consistent application and security policy enforcement across the stack even with workload mobility.
FlexPod provides a uniform approach to IT architecture, offering a well-characterized and documented
shared pool of resources for application workloads. FlexPod delivers operational efficiency and
consistency with the versatility to meet a variety of SLAs and IT initiatives, including:
•Application rollouts or application migrations
•Business continuity/disaster recovery
•Desktop virtualization
•Cloud delivery models (public, private, hybrid) and service models (IaaS, PaaS, SaaS)
•Asset consolidation and virtualization

11
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
FlexPod Design Details
This section provides an overview on the FlexPod design and also the topology differences between the
FlexPod model with clustered Data ONTAP and Data ONTAP in 7-mode.
System Overview
FlexPod is a best practice data center architecture that includes three components:
•Cisco Unified Computing System (Cisco UCS)
•Cisco Nexus switches
•NetApp Fabric-Attached Storage (FAS) systems
These components are connected and configured according to best practices of both Cisco and NetApp
and provide the ideal platform for running a variety of enterprise workloads with confidence. FlexPod
can scale up for greater performance and capacity (adding compute, network, or storage resources
individually as needed), or it can scale out for environments that need multiple consistent deployments
(rolling out additional FlexPod stacks). FlexPod delivers a baseline configuration and also has the
flexibility to be sized and optimized to accommodate many different use cases.
Typically, the more scalable and flexible a solution is, the more difficult it becomes to maintain a single
unified architecture capable of offering the same features and functionalities across each
implementation. This is one of the key benefits of FlexPod. Each of the component families shown in
Figure 1 (Cisco UCS, Cisco Nexus, and NetApp FAS) offers platform and resource options to scale the
infrastructure up or down, while supporting the same features and functionalities that are required under
the configuration and connectivity best practices of FlexPod.
Design Principles
FlexPod addresses the following design principles and architecture goals:
•Application Availability—Ensures that the services are accessible and ready to use.
•Scalability—Addresses increasing demands with appropriate resources.
•Flexibility—Provides new services or recovers resources without infrastructure modification
requirements.
•Manageability—Facilitates efficient infrastructure operations through open standards and APIs.
Note Performance and comprehensive security are key design criteria that were not directly addressed in this
project but have been addressed in other collateral, benchmarking and solution testing efforts. The
functionality and basic security elements were validated.
FlexPod - Distinct Uplink Design
Figure 2 details FlexPod distinct uplink design with clustered Data ONTAP. As the illustration shows,
the design is fully redundant in the compute, network, and storage layers. There is no single point of
failure from a device or traffic path perspective.

12
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
Figure 2 FlexPod Distinct Uplink Design with Clustered Data ONTAP
*The FAS22xx fully supports IP-based storage, but does not support FCoE.
FlexPod distinct uplink design is an end-to-end Ethernet transport solution supporting multiple LAN and
SAN protocols, most notably FCoE. The solution provides a unified 10GbE-enabled fabric, defined by
dedicated FCoE uplinks and dedicated Ethernet uplinks between the Cisco UCS Fabric Interconnects
and the Cisco Nexus switches, as well as converged connectivity between the NetApp storage devices
and the same multipurpose Cisco Nexus platforms.
The distinct uplink design does not employ a dedicated SAN switching environment and requires no
direct Fibre Channel connectivity. The Cisco Nexus 7000 Series Switches are configured in N-Port ID
Virtualization (NPIV) mode, providing storage services for the FCoE-based traffic traversing its fabric.
As illustrated, link aggregation technologies play an important role, providing improved aggregate
bandwidth and link resiliency across the solution stack. The NetApp storage controllers, Cisco Unified
Computing System, and Cisco Nexus 7000 platforms support active port channeling using 802.3ad
standard Link Aggregation Control Protocol (LACP). Port channeling is a link aggregation technique
offering link fault tolerance, and traffic distribution (load balancing) for improved aggregate bandwidth
across the member ports. In addition, the Cisco Nexus 7000 Series Switches feature virtual PortChannel
(vPC) capabilities. The vPC allows links that are physically connected to two different Cisco Nexus 7000
series devices to appear as a single “logical” port channel to a third device, essentially offering device
fault tolerance. The vPC addresses aggregate bandwidth, link, and device resiliency. The Cisco UCS
Fabric Interconnects and NetApp FAS controllers benefit from the Cisco Nexus vPC abstraction, gaining
link and device resiliency as well as full utilization of an non-blocking Ethernet fabric.

13
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
Note The Spanning Tree protocol does not actively block redundant physical links in a properly configured
vPC-enabled environment, so all ports should forward on the vPC member ports.
This dedicated uplink design leverages FCoE-capable NetApp FAS controllers. From a storage traffic
perspective, both standard LACP and Cisco vPC link aggregation technologies play an important role in
FlexPod distinct uplink design. Figure 2 shows the use of dedicated FCoE uplinks between the Cisco
UCS Fabric Interconnects and Cisco Nexus 7000 Unified Switches. The Cisco UCS Fabric Interconnects
operate in the N-Port Virtualization (NPV) mode, meaning the servers’ FC traffic is either manually or
automatically pinned to a specific FCoE uplink, in this case either of the two FCoE port channels are
pinned. The use of discrete FCoE port channels with distinct VSANs allows an organization to maintain
traditional SAN A/B fabric separation best practices, including separate zone databases. The vPC links
between the Cisco Nexus 7000 switches’ and NetApp storage controllers’ Unified Target Adapters
(UTAs) are utilized for the Ethernet traffic, and dedicated FCoE links between the systems maintain
SAN A/B fabric segregation.
Figure 2 shows the initial storage configuration of this solution, as a two-node HA pair with the NetApp
clustered Data ONTAP. An HA pair consists of like storage nodes such as FAS22xx, 32xx, or 62xx
Series. Scalability is achieved by adding storage capacity (disk/shelves) to an existing HA pair, or by
adding HA pairs into the cluster or storage domain. For SAN environments, the NetApp clustered Data
ONTAP offering allows up to three HA pairs that include six clustered nodes to form a single logical
entity and large resource pool of storage that can be easily managed, logically carved, and efficiently
consumed. For NAS environments, up to 24 nodes can be configured. In both the scenarios, the HA
interconnect allows each HA node pair to assume control of its partner’s storage (disk/shelves) directly.
The local physical high-availability storage failover capability does not extend beyond the HA pair.
Furthermore, a cluster of nodes does not have to include similar hardware. Rather, individual nodes in
an HA pair are configured alike, allowing customers to scale as needed, as they bring additional HA pairs
into the larger cluster.
Network failover is independent of the HA interconnect. Network failover of each node in the cluster is
supported by both the interconnect and switching fabric, permitting cluster, data and management
network interfaces to fail over to different nodes in the cluster, which extends beyond the HA pair.
Figure 3 shows FlexPod distinct uplink design with Data ONTAP operating in 7-Mode. Data ONTAP
operating in 7-Mode is NetApp’s traditional functional model. As depicted, the FAS devices are
configured in an HA pair delivering five nines availability. Scalability is achieved through the addition
of storage capacity (disk/shelves), as well as through additional controllers such as FAS2200, 3200, or
6200 Series. The controllers are only deployed in HA pairs, meaning more HA pairs can be added for
scalability, but each pair is managed separately.

14
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
Figure 3 FlexPod Distinct Uplink Design with Data ONTAP in 7-Mode
*The FAS22xx fully supports IP-based storage, but does not support FCoE.
Figure 4 shows the topology differences between FlexPod model with the clustered Data ONTAP or Data
ONTAP operating in 7-Mode. As shown in Figure 4, the Cisco UCS and Cisco Nexus components do
not require any modifications. These layers of the stack are essentially unaware of the storage
controllers’ mode of operation. The differences occur within the NetApp domain of FlexPod
configuration. Clustered Data ONTAP requires cluster interconnect switches to connect the storage
controllers (nodes) composing the cluster.
Note Data ONTAP 8.1.2 supports up to six nodes (three HA pairs) in a SAN cluster.

15
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
Figure 4 FlexPod Model Comparison
It is a fundamental design decision to leverage clustered Data ONTAP or 7-Mode, because these cannot
be run simultaneously on the same controller, and the choice will influence hardware requirements, the
logical construction of FlexPod stack, and ultimately the operational practices of the enterprise.
Organizations having the following requirements should consider adopting clustered Data ONTAP:
•Large to midsize enterprises that are seeking scalable, shared IT solutions for nondisruptive
operations
•New installations
•Existing clustered Data ONTAP 8.x and Data ONTAP GX organizations that are looking to upgrade
•Organizations deploying an enterprise content repository
Organizations with the following characteristics or needs might want to use the 7-Mode design:
•Existing Data ONTAP 7G and Data ONTAP 8.x 7-Mode customers who are looking to upgrade
•Midsize enterprises; customers who are primarily interested in the FAS2000 Series
•Customers who absolutely require SnapVault®, synchronous SnapMirror®, MetroCluster™,
SnapLock® software, IPv6, or Data ONTAP Edge
Note It is always advisable to seek counsel from experts. Consult your NetApp account team or partner for
further guidance.

16
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
“Logical Build” section on page 29 provides more details regarding the virtual design of the environment
consisting of VMware vSphere, Cisco Nexus 1000v virtual distributed switching, and NetApp storage
controllers.
Integrated System Components
The following components are required to deploy the Distinct Uplink design:
•Cisco Unified Compute System
•Cisco Nexus 7000 Series Switch
•NetApp Unified Storage capable of supporting FCoE storage target adapters
•VMware vSphere
Cisco Unified Computing System
The Cisco Unified Computing System is a next-generation solution for blade and rack server computing.
Cisco UCS is an innovative data center platform that unites compute, network, storage access, and
virtualization into a cohesive system designed to reduce total cost of ownership (TCO) and increase
business agility. The system integrates a low-latency, lossless 10 Gigabit Ethernet unified network fabric
with enterprise-class, x86-architecture servers. The system is an integrated, scalable, multi-chassis
platform in which all resources participate in a unified management domain. Managed as a single system
whether it has one server or 160 servers with thousands of virtual machines, the Cisco UCS decouples
scale from complexity. The Cisco UCS accelerates the delivery of new services simply, reliably, and
securely through end-to-end provisioning and migration support for both virtualized and non-virtualized
systems.
The Cisco Unified Computing System consists of the following components:
•Cisco UCS Manager— Provides unified, embedded management of all software and hardware
components in the Cisco UCS. For more information on Cisco UCS Manager, see:
http://www.cisco.com/en/US/products/ps10281/index.html
•Cisco UCS 6200 Series Fabric Interconnects—Are a family of line-rate, low-latency, lossless,
10-Gbps Ethernet and Fibre Channel over Ethernet interconnect switches providing the management
and communication backbone for the Unified Computing System. Cisco UCS supports VM-FEX
technology, see “Cisco VM-FEX” section on page 19 for details. For more information on Cisco
UCS 6200 Series Fabric Interconnects, see:
http://www.cisco.com/en/US/products/ps11544/index.html
•Cisco UCS 5100 Series Blade Server Chassis—Supports up to eight blade servers and up to two
fabric extenders in a six-rack unit (RU) enclosure. For more information on Cisco UCS 5100 Blade
Chassis, see:
http://www.cisco.com/en/US/products/ps10279/index.html
•Cisco UCS B-Series Blade Servers— Increase performance, efficiency, versatility and productivity
with these Intel based blade servers. For more information on Cisco UCS Blade Servers, see:
http://www.cisco.com/en/US/partner/products/ps10280/index.html
•Cisco UCS C-Series Rack Mount Server—Deliver unified computing in an industry-standard form
factor to reduce total cost of ownership and increase agility. For more information on Cisco UCS
Rack Mount Servers, see:
http://www.cisco.com/en/US/products/ps10493/index.html

17
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
•Cisco UCS Adapters—Wire-once architecture offers a range of options to converge the fabric,
optimize virtualization and simplify management. Cisco adapters support VM-FEX technology, see
“Cisco VM-FEX” section on page 19 for details. For more information Cisco UCS Adapters, see:
http://www.cisco.com/en/US/products/ps10277/prod_module_series_home.html
Cisco Nexus 7000 Series Switch
The modular Cisco Nexus 7000 Series switch offers a comprehensive one-platform solution for data
center networks. It offers aggregation, high density, and end-of-row and top-of-rack server connectivity.
For campus core deployments, it provides a scalable, highly resilient, high-performance solution. The
Cisco Nexus 7000 Series platform runs on Cisco NX-OS Software. It was specifically designed for the
most mission-critical deployments in the data center and on campus.
The Cisco Nexus 7000 Series was designed around four principles:
•Infrastructure scalability
– Design that provides scalability to more than 15Tbps for ongoing investment protection
– Support for consolidated networks with virtual port channel innovations to scale beyond 1500
ports
– Multicore, multithreaded OS to optimize CPU resources and offload tasks to processors
distributed across the modules
– Cisco Trusted Security (Cisco TrustSec®) for scalable security with link-layer encryption,
security group access control lists, and role-based access control
– Flexible NetFlow to optimize the network infrastructure, reducing operating costs and
improving capacity planning capabilities
•Operational continuity
– Lossless nondisruptive upgrades for zero-service downtime through no single point of failure in
the system hardware and a modular operating system
– Innovative stateful process restart for nondisruptive operations in event of process termination
– Comprehensive Extensible Markup Language (XML) API for total platform control
•Transport flexibility
– Foundation for unified fabrics with Cisco DCE unified I/O and FCoE
– Virtualized control plane and data plane forwarding for optimized performance
– Virtual device contexts (VDCs) to maximize software and hardware resource utilization while
providing strong security and software fault isolation
– Built to currently support high-density GbE and 10GbE and the emerging 40Gbps and 100Gbps
Ethernet standards
•Data center switching features
– In-Service Software Upgrade (ISSU) enables hitless upgrades with zero packet loss
– NetFlow provides visibility and flexible monitoring and control over the network
– Multihop FCoE provides director-class FCoE on a modular platform to offer rich LAN and SAN
services
– OTV and LISP enable seamless workload mobility across geographically separated data centers
– MPLS service supports multi-tenant segmentation within and between data centers

18
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
– FabricPath/TRILL allows scalable data center networks to be built without the tree protocol
For more information on Cisco Nexus 7000 Series Switches, see:
http://www.cisco.com/en/US/products/ps9402/index.html
Cisco Nexus 2232PP 10 Gigabit Etherenet Fabric Extender
The Cisco Nexus 2232PP 10Gigabit Fabric Extender provides 32 10 Gigabit Ethernet and Fibre Channel
Over Ethernet (FCoE) Small Form-Factor Pluggable Plus (SFP+) server ports and eight 10 Gigabit
Ethernet and FCoE SFP+ uplink ports in a compact 1 rack unit (1RU) form factor.
The built-in standalone software, Cisco Integrated Management Controller (CIMC), manages Cisco
UCS C-Series Rack Mount Servers. When a C-Series Rack Mount Server is integrated with Cisco UCS
Manager, via the Cisco Nexus 2232 Fabric Extender platform, the CIMC does not manage the server
anymore. Instead it is managed with the Cisco UCS Manager software. The server is managed using the
Cisco UCS Manager GUI or Cisco UCS Manager CLI. The Cisco Nexus 2232 Fabric Extender provides
data and control traffic support for the integrated C-Series servers.
Cisco Nexus 1000v
Cisco Nexus 1000V Series Switches provide a comprehensive and extensible architectural platform for
virtual machine (VM) and cloud networking. These switches are designed to accelerate server
virtualization and multitenant cloud deployments in a secure and operationally transparent manner.
Integrated into the VMware vSphere hypervisor, and fully compatible with VMware vCloud® Director,
the Cisco Nexus 1000V Series provides:
•Advanced virtual machine networking, based on Cisco NX-OS operating system and IEEE 802.1Q
switching technology
•Cisco vPath technology for efficient and optimized integration of virtual network services
•Virtual Extensible Local Area Network (VXLAN), supporting cloud networking
These capabilities help ensure that the virtual machine is a basic building block of the data center, with
full switching capabilities and a variety of Layer 4 through 7 services in both dedicated and multitenant
cloud environments. With the introduction of VXLAN on the Nexus 1000V Series, network isolation
among virtual machines can scale beyond traditional VLANs for cloud-scale networking.
The Cisco Nexus 1000V Series Switches are virtual machine access switches for the VMware vSphere
environments running the Cisco NX-OS operating system. Operating inside the VMware® ESX® or
ESXi™ hypervisors, the Cisco Nexus 1000V Series provides:
•Policy-based virtual machine connectivity
•Mobile virtual machine security and network policy
•Nondisruptive operational model for your server virtualization and networking teams
•Virtualized network services with Cisco vPath providing a single architecture for L4 –L7 network
services such as load balancing, firewalling and WAN acceleration
The Cisco Nexus 1000V distributed virtual switch is an optional component within the solution. The
Cisco Nexus 1000V was used in the validation of this solution; however, customers can also use a
standard VMware vSwitch or a VMware VDS. The VSM in this solution is running from the Cisco
Nexus 1110-X appliance, which is also an optional component.
For more information on Cisco Nexus 1000V Series Switches and Cisco Nexus 1010 Virtual Services
Appliance, see:
http://www.cisco.com/en/US/products/ps9902/index.html

19
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
http://www.cisco.com/en/US/products/ps10785/index.html
Cisco VM-FEX
Cisco VM-FEX technology collapses virtual switching infrastructure and physical switching
infrastructure into a single, easy-to-manage environment. Benefits include:
•Simplified operations—Eliminates the need for a separate, virtual networking infrastructure
•Improved network security—Contains VLAN proliferation
•Optimized network utilization—Reduces broadcast domains
•Enhanced application performance—Off loads virtual machine switching from host CPU to parent
switch Application Specific Integrated Circuits (ASICs)
Cisco VM-FEX is supported on VMware ESX hypervisors and fully supports workload mobility through
VMware vMotion.
The Cisco VM-FEX eliminates the virtual switch within the hypervisor by providing individual Virtual
Machines (VMs) virtual ports on the physical network switch. VM I/O is directly sent to the upstream
physical network switch that takes full responsibility for VM switching and policy enforcement. This
leads to consistent treatment for all network traffic, virtual or physical. The Cisco VM-FEX reduces the
number of network management points by an order of magnitude as the physical and the virtual
switching layers are consolidated into a single switching infrastructure.
The UCS VIC leverages VMware’s Direct Path I/O technology to significantly improve throughput and
latency of VM I/O. Direct Path allows direct assignment of Personal Computer interconnect express
(PCIe) devices to VMs. VM I/O bypasses the hypervisor layer and is placed directly on the PCIe device
associated with the VM. The Cisco VM-FEX unifies the virtual and physical networking infrastructure
by allowing a switch ASIC to perform switching in hardware, not on a software based virtual switch.
The Cisco VM-FEX is off loads the ESXi hypervisor, which may improve the performance of any hosted
VM applications.
NetApp FAS and Data ONTAP
NetApp solutions are user friendly, easy to manage, and quick to deploy and offer increased availability
while consuming fewer IT resources. This means that they dramatically lower the lifetime total cost of
ownership. Whereas others manage complexity, NetApp eliminates it. A NetApp solution includes
hardware in the form of controllers and disk storage and the NetApp Data ONTAP operating system.
NetApp offers the NetApp Unified Storage Architecture. The term “unified” refers to a family of storage
systems that simultaneously support Storage Area Network (SAN), Network Attached Storage (NAS),
and iSCSI across many operating environments such as VMware, Windows®, and UNIX®. This single
architecture provides access to data by using industry-standard protocols, including NFS, CIFS, iSCSI,
FCP, SCSI, FTP, and HTTP. Connectivity options include standard Ethernet (10/100/1000, or 10GbE)
and Fibre Channel (1, 2, 4, or 8Gb/sec). In addition, all systems can be configured with
high-performance Solid State Drives (SSDs) or Serial ATA (SAS) disks for primary storage applications,
low-cost SATA disks for secondary applications (backup, archive, and so on), or a mix of the different
disk types.
A storage system running Data ONTAP has a main unit, also known as the controller or storage engine,
which is the hardware device that receives and sends data. This unit detects and gathers information
about the hardware configuration, the storage system components, the operational status, hardware
failures, and other error conditions.
A storage system uses storage on disk shelves. The disk shelves are the containers or device carriers that
hold disks and associated hardware such as power supplies, connectivity interfaces, and cabling.

20
VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE
FlexPod Design Details
If storage requirements change over time, the NetApp storage offers the flexibility to change quickly, as
needed and without expensive and disruptive “forklift” upgrades. For example, a LUN can be changed
from FC access to iSCSI access without moving or copying the data. Only a simple dismount of the FC
LUN and a mount of the same LUN using iSCSI would be required. In addition, a single copy of data
can be shared between Windows and UNIX systems while allowing each environment to access the data
through native protocols and applications. If a system was originally purchased with all SATA disks for
backup applications, high-performance SAS disks could be added to support primary storage
applications such as Oracle®, Microsoft® Exchange Server, or ClearCase.
NetApp storage solutions provide redundancy and fault tolerance through clustered storage controllers,
hot-swappable redundant components (such as cooling fans, power supplies, disk drives, and shelves),
and multiple network interfaces. This highly available and flexible architecture enables customers to
manage all data under one common infrastructure while achieving mission requirements. The NetApp
Unified Storage Architecture allows data storage with higher availability and performance, easier
dynamic expansion, and more unrivalled ease of management than any other solution.
The storage efficiency built into Data ONTAP provides substantial space savings, allowing more data to
be stored at a lower cost. Data protection provides replication services, making sure that valuable data
is backed up and recoverable. The following features provide storage efficiency and data protection:
•Thin provisioning—Volumes are created using “virtual” sizing. They appear to be provisioned to
their full capacity, but are actually created much smaller and use additional space only when it is
actually needed. Extra unused storage is shared across all volumes, and the volumes can grow and
shrink on demand.
•Snapshot™ copies—Automatically scheduled point-in-time copies that write only changed blocks,
with no performance penalty. The Snapshot copies consume minimal storage space, since only
changes to the active file system are written. Individual files and directories can easily be recovered
from any Snapshot copy, and the entire volume can be restored back to any Snapshot state in
seconds.
•FlexClone® volumes—Near-zero space, instant “virtual” copies of datasets. The clones are
writable, but only changes to the original are stored, so they provide rapid, space-efficient creation
of additional data copies ideally suited for dev/test environments.
•Deduplication—Removes redundant data blocks in primary and secondary storage, with flexible
policies to determine when the deduplication process is run.
•Compression—Compresses data blocks. Compression can be run whether or not deduplication is
enabled and can provide additional space savings, whether run alone or together with deduplication.
•SnapMirror—Volumes can be asynchronously replicated either within the cluster or to another
cluster.
For more information on NetApp Data ONTAP, see:
http://www.netapp.com/us/products/platform-os/data-ontap-8/index.aspx
Data ONTAP Operating in 7-Mode
As previously mentioned customers have a choice of deploying their NetApp storage environment
operating in 7-Mode or clustered Data ONTAP. Data ONTAP operating in 7-Mode provides customers
a broad suite of application integrations, storage efficiencies, and a legacy of customer satisfaction.
As well known and trusted as Data ONTAP operating in 7-Mode is, technology companies must always
look toward new innovations. For this reason NetApp has continually invested in clustered Data ONTAP,
which truly changes the conversation of storage from a cost-center discussion to one in which storage
can add value to the company.