Cisco Nexus 9000 Series User manual

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration
Guide, Release 7.x
First Published: 2015-01-31
Last Modified: 2017-01-31
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH
THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,
CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version
of the UNIX operating system. All rights reserved. Copyright ©1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.
CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network
topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional
and coincidental.
This product includes cryptographic software written by Eric Young ([email protected]).
This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/)
This product includes software written by Tim Hudson ([email protected]).
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://
www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1110R)
©2016-2017 Cisco Systems, Inc. All rights reserved.

CONTENTS
Preface Preface xi
Audience xi
Document Conventions xi
Related Documentation for Cisco Nexus 9000 Series Switches xii
Documentation Feedback xii
Obtaining Documentation and Submitting a Service Request xiii
CHAPTER 1 New and Changed Information 1
New and Changed Information 1
CHAPTER 2 Overview 5
About QoS Features 5
Using QoS 6
Classification 6
Marking 6
Policing 7
Queuing and Scheduling 7
Sequencing of QoS Actions 7
Sequencing of Ingress Traffic Actions 7
Sequencing of Egress Traffic Actions 8
High Availability Requirements for QoS Features 8
QoS Feature Configuration with MQC 8
QoS Statistics 9
Default QoS Behavior 9
Virtual Device Contexts 9
CHAPTER 3 Using Modular QoS CLI 11
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
iii

About MQC 11
System Classes 12
Default System Classes 12
Licensing Requirements for Using MQC Objects 13
Using an MQC Object 13
Type qos Policies 13
Type Queuing Policies 14
System-Defined MQC Objects 15
System-Defined MQC Objects for 4q Mode 16
System-Defined MQC Objects for 8q Mode 18
Changing to 8q Mode 21
Notes About 8q Mode 22
Example of Changing to 8q Mode 22
Example of set qos-groups 25
Changing from 8q Mode to 4q Mode 26
Configuring an MQC Object 26
Configuring or Modifying a Class Map 26
Configuring or Modifying a Policy Map 28
Applying Descriptions to MQC Objects 29
Verifying an MQC Object 31
Attaching and Detaching a QoS Policy Action 32
Configuring a Service Policy for a Layer 2 Interface 33
Configuring a Service Policy for a Layer 3 Interface 34
Attaching the System Service Policy 35
Attaching a QoS Policy Action to a VLAN 36
Session Manager Support for QoS 38
CHAPTER 4 Configuring QoS TCAM Carving 39
About QoS TCAM Carving 39
About QoS TCAM Lite Regions 41
Guidelines and Limitations 42
Configuring QoS TCAM Carving 44
Enabling Layer 3 QoS (IPv6) 44
Enabling VLAN QoS (IPv4) 47
Enabling FEX QoS (IPv4) 48
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
iv
Contents

Enabling Egress QoS (IPv4) 49
Using Templates to Configure TCAM Region Sizes 50
Verifying QoS TCAM Carving 52
CHAPTER 5 Configuring Classification 55
About Classification 55
Licensing Requirements for Classification 56
Prerequisites for Classification 56
Guidelines and Limitations 56
Configuring Traffic Classes 57
Configuring ACL Classification 57
Configuring DSCP Classification 58
Configuring IP Precedence Classification 60
Configuring Protocol Classification 62
Configuring Layer 3 Packet Length Classification 63
Configuring CoS Classification 64
Configuring CoS Classification for FEX 66
Configuring IP RTP Classification 67
Verifying the Classification Configuration 68
Configuration Examples for Classification 68
CHAPTER 6 Configuring Marking 69
About Marking 69
Trust Boundaries 70
Class of Behavior 70
Class of Behavior 71
Licensing Requirements for Marking 71
Prerequisites for Marking 72
Guidelines and Limitations 72
Configuring Marking 73
Configuring DSCP Marking 73
Configuring IP Precedence Marking 75
Configuring CoS Marking 77
Configuring CoS Marking for FEX 78
Configuring DSCP Port Marking 79
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
v
Contents

Verifying the Marking Configuration 81
Configuration Examples for Marking 81
CHAPTER 7 Configuring Policing 83
About Policing 83
Licensing Requirements for Policing 83
Prerequisites for Policing 84
Guidelines and Limitations 84
Configuring Policing 85
Configuring 1-Rate and 2-Rate, 2-Color and 3-Color Policing 85
Configuring Ingress and Egress Policing 90
Configuring Markdown Policing 90
Verifying the Policing Configuration 92
Configuration Examples for Policing 92
CHAPTER 8 Configuring Queuing and Scheduling 93
About Queuing and Scheduling 93
Modifying Class Maps 94
Congestion Avoidance 94
Congestion Management 94
Explicit Congestion Notification 94
Approximate Fair Drop 95
Traffic Shaping 97
Licensing Requirements for Queuing and Scheduling 97
Prerequisites for Queuing and Scheduling 97
Guidelines and Limitations 98
Configuring Queuing and Scheduling 100
Configuring Type Queuing Policies 101
Configuring Congestion Avoidance 102
Configuring Tail Drop on Egress Queues 102
Configuring WRED on Egress Queues 105
Configuring AFD on Egress Queues 107
Configuring Congestion Management 108
Configuring Bandwidth and Bandwidth Remaining 109
Configuring Bandwidth and Bandwidth Remaining for FEX 111
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
vi
Contents

Configuring Priority 113
Configuring Priority for FEX 115
Configuring Traffic Shaping 117
Applying a Queuing Policy on a System 119
Verifying the Queuing and Scheduling Configuration 120
Controlling the QoS Shared Buffer 120
Monitoring the QoS Packet Buffer 121
Configuration Examples for Queuing and Scheduling 122
Example: Configuring WRED on Egress Queues 122
Example: Configuring Traffic Shaping 122
CHAPTER 9 Configuring Network QoS 125
About Network QoS 125
Licensing Requirements for Network QoS 125
Prerequisites for Network QoS 126
Guidelines and Limitations 126
Dynamic Packet Prioritization 126
Configuring Network QoS Policies 127
Copying a Predefined Network QoS Policy 127
Configuring a User-Defined Network QoS Policy 128
Applying a Network QoS Policy on a System 129
Verifying the Network QoS 130
CHAPTER 10 Configuring Link Level Flow Control 131
Link Level Flow Control 131
Guidelines and Restrictions for Link Level Flow Control 131
Information About Link Level Flow Control 132
Link Level Flow Control on Interfaces 132
Link Level Flow Control on Ports 132
Mismatched Link Level Flow Control Configurations 133
How to Configure Link Level Flow Control 133
Configuring Link Level Flow Control Receive 133
Configuring Link Level Flow Control Transmit 134
Configuration Examples for Link Level Flow Control 135
Example: Configuring a No-Drop Policy 135
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
vii
Contents

Example: Configuring Link Level Flow Control Receive and Send 136
CHAPTER 11 Configuring Priority Flow Control 137
About Priority Flow Control 137
Licensing Requirements for Priority Flow Control 138
Prerequisites for Priority Flow Control 138
Guidelines and Limitations for Priority Flow Control 138
Default Settings for Priority Flow Control 141
Configuring Priority Flow Control 141
Enabling Priority Flow Control on a Traffic Class 142
Configuring a Priority Flow Control Watchdog Interval 146
Configuring Pause Buffer Thresholds and Queue Limit Using Ingress Queuing Policy 146
Verifying the Priority Flow Control Configuration 148
Configuration Examples for Priority Flow Control 148
CHAPTER 12 Monitoring QoS Statistics 151
About QoS Statistics 151
Licensing Requirements for Monitoring QoS Statistics 151
Prerequisites for Monitoring QoS Statistics 152
Guidelines and Limitations 152
Enabling Statistics 154
Monitoring the Statistics 155
Clearing Statistics 155
Configuration Examples For Monitoring QoS Statistics 156
CHAPTER 13 Micro-Burst Monitoring 159
Micro-Burst Monitoring 159
Guidelines and Limitations for Micro-Burst Monitoring 159
Configuring Micro-Burst Detection 161
Clearing Micro-Burst Detection 163
Verifying Micro-Burst Detection 163
Example of Micro-Burst Detection Output 164
APPENDIX A FEX QoS Configuration 165
FEX QoS Configuration Information 165
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
viii
Contents

TCAM Carving for FEX QoS 167
FEX QoS Configuration Example 168
Verifying the FEX QoS Configuration 182
APPENDIX B Additional References 183
RFCs 183
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
ix
Contents

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
x
Contents

Preface
This preface includes the following sections:
•Audience, page xi
•Document Conventions, page xi
•Related Documentation for Cisco Nexus 9000 Series Switches, page xii
•Documentation Feedback, page xii
•Obtaining Documentation and Submitting a Service Request, page xiii
Audience
This publication is for network administrators who install, configure, and maintain Cisco Nexus switches.
Document Conventions
Command descriptions use the following conventions:
DescriptionConvention
Bold text indicates the commands and keywords that you enter literally
as shown.
bold
Italic text indicates arguments for which the user supplies the values.
Italic
Square brackets enclose an optional element (keyword or argument).[x]
Square brackets enclosing keywords or arguments separated by a vertical
bar indicate an optional choice.
[x | y]
Braces enclosing keywords or arguments separated by a vertical bar
indicate a required choice.
{x | y}
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xi

DescriptionConvention
Nested set of square brackets or braces indicate optional or required
choices within optional or required elements. Braces and a vertical bar
within square brackets indicate a required choice within an optional
element.
[x {y | z}]
Indicates a variable for which you supply values, in context where italics
cannot be used.
variable
A nonquoted set of characters. Do not use quotation marks around the
string or the string will include the quotation marks.
string
Examples use the following conventions:
DescriptionConvention
Terminal sessions and information the switch displays are in screen font.screen font
Information you must enter is in boldface screen font.boldface screen font
Arguments for which you supply values are in italic screen font.
italic screen font
Nonprinting characters, such as passwords, are in angle brackets.< >
Default responses to system prompts are in square brackets.[ ]
An exclamation point (!) or a pound sign (#) at the beginning of a line
of code indicates a comment line.
!, #
Related Documentation for Cisco Nexus 9000 Series Switches
The entire Cisco Nexus 9000 Series switch documentation set is available at the following URL:
http://www.cisco.com/en/US/products/ps13386/tsd_products_support_series_home.html
Documentation Feedback
To provide technical feedback on this document, or to report an error or omission, please send your comments
to [email protected]. We appreciate your feedback.
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xii
Preface
Related Documentation for Cisco Nexus 9000 Series Switches

Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service
request, and gathering additional information, see What's New in Cisco Product Documentation at: http://
www.cisco.com/c/en/us/td/docs/general/whatsnew/whatsnew.html.
Subscribe to What's New in Cisco Product Documentation, which lists all new and revised Cisco technical
documentation as an RSS feed and delivers content directly to your desktop using a reader application. The
RSS feeds are a free service.
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xiii
Preface
Obtaining Documentation and Submitting a Service Request

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xiv
Preface
Obtaining Documentation and Submitting a Service Request

CHAPTER 1
New and Changed Information
This chapter provides release-specific information for each new and changed feature in the Cisco Nexus
9000 Series NX-OS QoS Configuration Guide, Release.
•New and Changed Information, page 1
New and Changed Information
This table summarizes the new and changed features for the Cisco Nexus 9000 Series NX-OS Quality of Service
Configuration Guide and where they are documented.
Table 1: New and Changed Features
Where DocumentedChanged
in
Release
DescriptionFeature
Guidelines and Limitations, on
page 98
Configuring Pause Buffer
Thresholds and Queue Limit
Using Ingress Queuing Policy,
on page 146
7.0(3)F1(1)Added support for the
X9636C-R and X9636Q-R line
cards and the C9508-FM-R
fabric module.
Ingress queuing
Micro-Burst Monitoring7.0(3)I5(1)Added support for micro-burst
monitoring on Application
Spine Engines (ASE2, ASE3)
and Leaf Spine Engines (LSE).
Micro-Burst Monitoring and
Detection
Approximate Fair Drop7.0(3)I5(1)Added support for AFD, an
Active Queue Management
(AQM) algorithm to avoid
traffic congestion.
AFD (Approximate Fair
Dropping)
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
1

Where DocumentedChanged
in
Release
DescriptionFeature
About Priority Flow Control,
on page 137
7.0(3)I5(1)Added support for the Cisco
Nexus 9200 and 9300-EX
Series switches.
DCBXP
Dynamic Packet Prioritization7.0(3)I5(1)Added support for DPP. DPP
prioritizes a configured number
of packets of every flow in a
particular class of traffic to be
sent out of another class of
traffic that DPP is mapped to.
DPP (Dynamic Packet
Prioritization)
Guidelines and Limitations, on
page 98
Configuring WRED on Egress
Queues, on page 105
7.0(3)I4(5)Added support for the feature
on the Cisco Nexus 9508
switches and Cisco Nexus
3164Q switches.
Configuring WRED on Egress
Queues
Guidelines and Limitations for
Priority Flow Control, on page
138
Configuring a Priority Flow
Control Watchdog Interval, on
page 146
7.0(3)I4(5)Added support for the feature
on the Cisco Nexus 9508
switches and Cisco Nexus
3164Q switches.
Configuring a Priority Flow
Control Watchdog Interval
Guidelines and Limitations7.0(3)I4(4)Added maximum queue
occupancy support for Leaf
Spine Engine (LSE) enabled
switches.
Maximum queue occupancy
support for Leaf Spine Engines
(LSE)
Configuring Tail Drop on
Egress Queues
7.0(3)I4(4)Added updated alpha values for
dynamic queue size.
Updated alpha values for
dynamic queue size
Configuring Tail Drop on
Egress Queues
7.0(3)I4(4)Added this command to tune
the random noise parameter on
Cisco Nexus 9300-EX Series
switches.
hardware qos q-noise percent
<value> command
Configuring WRED on Egress
Queues
7.0(3)I4(2)Added a command to allow
non-ECN-capable traffic to
bypass WRED thresholds.
Congestion avoidance
Guidelines and Limitations for
Priority Flow Control
7.0(3)I4(2)Changed the behavior of the no
lldp tlv-select dcbxp command
such that PFC mode will be
disabled for interfaces on both
sides of back-to-back switches.
DCBX support
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
2
New and Changed Information
New and Changed Information

Where DocumentedChanged
in
Release
DescriptionFeature
Configuring Priority Flow
Control
7.0(3)I4(2)Added commands to globally
disable PFC on all interfaces
and to configure a watchdog
interval to detect whether
packets in a no-drop queue are
being drained within a specified
time period.
Priority flow control
Using Templates to Configure
TCAM Region Sizes
7.0(3)I3(1)Added support to create and
apply custom TCAM templates.
Custom TCAM template
support
Micro-Burst Monitoring7.0(3)I3(1)Added support for micro-burst
detection
Micro-Burst Detection support
Configuring Priority Flow
Control
7.0(3)I3(1)Added support for DCBX
protocol.
DCBX support
FEX QoS Configuration
Example
7.0(3)I2(1)Added support for FEX QoS
configuration.
FEX QoS configuration support
Attaching a QoS Policy Action
to a VLAN, on page 36
7.0(3)I1(2)Enables label sharing when the
same QoS policy is applied on
multiple VLANs.
Label sharing support for QoS
policies under VLANs
About QoS TCAM Carving7.0(3)I1(2)Added support for 100G
devices.
100G device support
Enabling Egress QoS (IPv4)7.0(3)I1(2)Added support for egress QoS
TCAM carving.
Enabling egress QoS
Configuring Ingress and Egress
Policing
7.0(3)I1(1)Added support to apply the
policing instructions in a QoS
policy map to egress packets.
Egress policing support
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
3
New and Changed Information
New and Changed Information

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
4
New and Changed Information
New and Changed Information

CHAPTER 2
Overview
•About QoS Features, page 5
•Using QoS, page 6
•Classification, page 6
•Marking, page 6
•Policing, page 7
•Queuing and Scheduling, page 7
•Sequencing of QoS Actions, page 7
•High Availability Requirements for QoS Features, page 8
•QoS Feature Configuration with MQC, page 8
•QoS Statistics, page 9
•Default QoS Behavior, page 9
•Virtual Device Contexts, page 9
About QoS Features
You use the QoS features to provide the most desirable flow of traffic through a network. QoS allows you to
classify the network traffic, police and prioritize the traffic flow, and help avoid traffic congestion in a network.
The control of traffic is based on the fields in the packets that flow through the system. You use the Modular
QoS (MQC) CLI to create the traffic classes and policies of the QoS features.
QoS features are applied using QoS and queuing policies as follows:
•QoS policies include classification and marking features.
•QoS policies include policing features.
•QoS policies include shaping, weighted random early detection (WRED), and explicit congestion
notification (ECN) features.
•Queuing policies use the queuing and scheduling features.
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
5

The system-defined QoS features and values that are discussed in the “Using Modular QoS CLI”section
apply globally to the entire device and cannot be modified.
Note
Using QoS
Traffic is processed based on how you classify it and the policies that you create and apply to traffic classes.
To configure QoS features, you use the following steps:
1Create traffic classes by classifying the incoming packets that match criteria such as IP address or QoS
fields.
2Create policies by specifying actions to take on the traffic classes, such as policing, marking, or dropping
packets.
3Apply policies to a port, port channel, or subinterface.
You use MQC to create the traffic classes and policies of the QoS features.
The queuing and scheduling operations of the overall QoS feature are applicable to both IPv4 and IPv6.Note
IP tunnels do not support access control lists (ACLs) or QoS policies.Note
Classification
You use classification to partition traffic into classes. You classify the traffic based on the port characteristics
or the packet header fields that include IP precedence, differentiated services code point (DSCP), Layer 3 to
Layer 4 parameters, and the packet length.
The values used to classify traffic are called match criteria. When you define a traffic class, you can specify
multiple match criteria, you can choose to not match on a particular criterion, or you can determine the traffic
class by matching any or all criteria.
Traffic that fails to match any class is assigned to a default class of traffic called class-default.
Marking
Marking is the setting of QoS information that is related to a packet. You can set the value of a standard QoS
field for COS, IP precedence and DSCP, and internal labels (such as QoS groups) that can be used in subsequent
actions. Marking QoS groups is used to identify the traffic type for queuing and scheduling traffic.
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
6
Overview
Using QoS