Cisco Nexus 9000 Series User manual

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide,
Release 7.x
First Published: 2015-01-31
Last Modified: 2020-08-31
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS REFERENCED IN THIS DOCUMENTATION ARE SUBJECT TO CHANGE WITHOUT NOTICE.
EXCEPT AS MAY OTHERWISE BE AGREED BY CISCO IN WRITING, ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS DOCUMENTATION ARE
PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
The Cisco End User License Agreement and any supplemental license terms govern your use of any Cisco software, including this product documentation, and are located at:
http://www.cisco.com/go/softwareterms.Cisco product warranty information is available at http://www.cisco.com/go/warranty. US Federal Communications Commission Notices are found
here http://www.cisco.com/c/en/us/products/us-fcc-notice.html.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any products and features described herein as in development or available at a future date remain in varying stages of development and will be offered on a when-and if-available basis. Any
such product or feature roadmaps are subject to change at the sole discretion of Cisco and Cisco will have no liability for delay in the delivery or failure to deliver any products or feature
roadmap items that may be set forth in this document.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network
topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional
and coincidental.
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based
on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language
that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com
go trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any
other company. (1721R)
©2016–2020 Cisco Systems, Inc. All rights reserved.

CONTENTS
Preface xi
PREFACE
Audience xi
Document Conventions xi
Related Documentation for Cisco Nexus 9000 Series Switches xii
Documentation Feedback xii
Communications, Services, and Additional Information xii
New and Changed Information 1
CHAPTER 1
New and Changed Information 1
Overview 7
CHAPTER 2
Licensing Requirements 7
About QoS Features 7
Using QoS 8
Classification 8
Marking 8
Policing 9
Queuing and Scheduling 9
Sequencing of QoS Actions 9
Sequencing of Ingress Traffic Actions 9
Sequencing of Egress Traffic Actions 10
High Availability Requirements for QoS Features 10
QoS Feature Configuration with MQC 10
QoS Statistics 10
Default QoS Behavior 11
Virtual Device Contexts 11
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
iii

Using Modular QoS CLI 13
CHAPTER 3
About MQC 13
System Classes 14
Default System Classes 14
Using an MQC Object 14
Type qos Policies 15
Type Queuing Policies 15
System-Defined MQC Objects 16
System-Defined MQC Objects for 4q Mode 17
System-Defined MQC Objects for 8q Mode 19
Changing to 8q Mode 22
Changing from 8q Mode to 4q Mode 28
Guidelines and Limitations 28
Configuring an MQC Object 28
Configuring or Modifying a Class Map 28
Configuring or Modifying a Policy Map 30
Applying Descriptions to MQC Objects 31
Verifying an MQC Object 32
Attaching and Detaching a QoS Policy Action 33
Configuring a Service Policy for a Layer 2 Interface 34
Configuring a Service Policy for a Layer 3 Interface 35
Attaching the System Service Policy 36
Attaching a QoS Policy Action to a VLAN 37
Session Manager Support for QoS 38
Configuring QoS TCAM Carving 39
CHAPTER 4
About QoS TCAM Carving 39
About QoS TCAM Lite Regions 42
Guidelines and Limitations 43
Configuring QoS TCAM Carving 44
Enabling Layer 3 QoS (IPv6) 44
Enabling VLAN QoS (IPv4) 47
Notes for Enabling VLAN QoS 49
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
iv
Contents

Enabling FEX QoS (IPv4) 49
Enabling Egress QoS (IPv4) 50
Using Templates to Configure TCAM Region Sizes 52
Verifying QoS TCAM Carving 53
Configuring Classification 55
CHAPTER 5
About Classification 55
Prerequisites for Classification 56
Guidelines and Limitations 56
Configuring Traffic Classes 58
Configuring ACL Classification 58
Examples: Configuring ACL Classification 59
Configuring a DSCP Wildcard Mask 60
Configuring DSCP Classification 62
Configuring IP Precedence Classification 63
Configuring Protocol Classification 65
Configuring Layer 3 Packet Length Classification 66
Configuring CoS Classification 67
Configuring CoS Classification for FEX 68
Configuring IP RTP Classification 70
Verifying the Classification Configuration 71
Configuration Examples for Classification 71
Configuring Marking 73
CHAPTER 6
About Marking 73
Trust Boundaries 74
Class of Behavior 74
Prerequisites for Marking 75
Guidelines and Limitations 75
Configuring Marking 76
Configuring DSCP Marking 76
Configuring IP Precedence Marking 78
Configuring CoS Marking 80
Configuring CoS Marking for FEX 81
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
v
Contents

Configuring DSCP Port Marking 82
Verifying the Marking Configuration 84
Configuration Examples for Marking 84
Configuring Policing 85
CHAPTER 7
About Policing 85
Shared Policers 85
Prerequisites for Policing 86
Guidelines and Limitations 86
Configuring Policing 88
Configuring Ingress Policing 88
Configuring Egress Policing 88
Configuring 1-Rate and 2-Rate, 2-Color and 3-Color Policing 90
Configuring Markdown Policing 95
Configuring Shared Policers 97
Verifying the Policing Configuration 99
Configuration Examples for Policing 99
Configuring Queuing and Scheduling 101
CHAPTER 8
About Queuing and Scheduling 101
Modifying Class Maps 101
Congestion Avoidance 102
Congestion Management 102
Explicit Congestion Notification 102
Approximate Fair Drop 103
Traffic Shaping 105
Prerequisites for Queuing and Scheduling 106
Guidelines and Limitations 106
Configuring Queuing and Scheduling 109
Configuring Type Queuing Policies 110
Configuring Congestion Avoidance 111
Configuring Tail Drop on Egress Queues 111
Configuring WRED on Egress Queues 114
Configuring AFD on Egress Queues 116
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
vi
Contents

Configuring Congestion Management 118
Configuring Bandwidth and Bandwidth Remaining 118
Configuring Bandwidth and Bandwidth Remaining for FEX 120
Configuring Priority 122
Configuring Priority for FEX 124
Configuring Traffic Shaping 126
Applying a Queuing Policy on a System 128
Verifying the Queuing and Scheduling Configuration 129
Controlling the QoS Shared Buffer 129
Managing Dynamic Buffer Sharing 130
Monitoring the QoS Packet Buffer 130
Configuration Examples for Queuing and Scheduling 132
Example: Configuring WRED on Egress Queues 132
Example: Configuring Traffic Shaping 132
Configuring Network QoS 133
CHAPTER 9
About Network QoS 133
Prerequisites for Network QoS 133
Guidelines and Limitations 133
Dynamic Packet Prioritization 134
Configuring Network QoS Policies 135
Copying a Predefined Network QoS Policy 135
Configuring a User-Defined Network QoS Policy 135
Applying a Network QoS Policy on a System 136
Verifying the Network QoS 137
Configuring Link Level Flow Control 139
CHAPTER 10
Link Level Flow Control 139
Guidelines and Restrictions for Link Level Flow Control 139
Information About Link Level Flow Control 140
Link Level Flow Control on Interfaces 140
Link Level Flow Control on Ports 141
Mismatched Link Level Flow Control Configurations 141
How to Configure Link Level Flow Control 141
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
vii
Contents

Configuring Link Level Flow Control Receive 141
Configuring Link Level Flow Control Transmit 142
Configuring a Link Level Flow Control Watchdog Interval 143
Configuration Examples for Link Level Flow Control 146
Example: Configuring a No-Drop Policy 146
Example: Configuring Link Level Flow Control Receive and Send 147
Configuring Priority Flow Control 149
CHAPTER 11
About Priority Flow Control 149
Prerequisites for Priority Flow Control 150
Guidelines and Limitations for Priority Flow Control 150
Default Settings for Priority Flow Control 153
Configuring Priority Flow Control 153
Enabling Priority Flow Control on a Traffic Class 155
Configuring a Priority Flow Control Watchdog Interval 158
Configuring Pause Buffer Thresholds and Queue Limit Using Ingress Queuing Policy 161
Configuring QoS Pause Buffer and Dynamic Queue Limits 163
Verifying the Priority Flow Control Configuration 164
Configuration Examples for Priority Flow Control 164
Monitoring QoS Statistics 167
CHAPTER 12
About QoS Statistics 167
Prerequisites for Monitoring QoS Statistics 167
Guidelines and Limitations 167
Enabling Statistics 169
Monitoring the Statistics 170
Clearing Statistics 171
Configuration Examples For Monitoring QoS Statistics 171
Micro-Burst Monitoring 173
CHAPTER 13
Micro-Burst Monitoring 173
Guidelines and Limitations for Micro-Burst Monitoring 173
Configuring Micro-Burst Detection Per-Queue 175
Clearing Micro-Burst Detection 177
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
viii
Contents

Verifying Micro-Burst Detection 178
Example of Micro-Burst Detection Output 178
FEX QoS Configuration 179
APPENDIX A
FEX QoS Configuration Information 179
TCAM Carving for FEX QoS 181
FEX QoS Configuration Example 182
Verifying the FEX QoS Configuration 198
Additional References 199
APPENDIX B
RFCs 199
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
ix
Contents

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
x
Contents

Preface
This preface includes the following sections:
•Audience, on page xi
•Document Conventions, on page xi
•Related Documentation for Cisco Nexus 9000 Series Switches, on page xii
•Documentation Feedback, on page xii
•Communications, Services, and Additional Information, on page xii
Audience
This publication is for network administrators who install, configure, and maintain Cisco Nexus switches.
Document Conventions
Command descriptions use the following conventions:
DescriptionConvention
Bold text indicates the commands and keywords that you enter literally
as shown.
bold
Italic text indicates arguments for which you supply the values.Italic
Square brackets enclose an optional element (keyword or argument).[x]
Square brackets enclosing keywords or arguments that are separated by
a vertical bar indicate an optional choice.
[x | y]
Braces enclosing keywords or arguments that are separated by a vertical
bar indicate a required choice.
{x | y}
Nested set of square brackets or braces indicate optional or required
choices within optional or required elements. Braces and a vertical bar
within square brackets indicate a required choice within an optional
element.
[x {y | z}]
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xi

DescriptionConvention
Indicates a variable for which you supply values, in context where italics
cannot be used.
variable
A nonquoted set of characters. Do not use quotation marks around the
string or the string includes the quotation marks.
string
Examples use the following conventions:
DescriptionConvention
Terminal sessions and information the switch displays are in screen font.screen font
Information that you must enter is in boldface screen font.boldface screen font
Arguments for which you supply values are in italic screen font.italic screen font
Nonprinting characters, such as passwords, are in angle brackets.< >
Default responses to system prompts are in square brackets.[ ]
An exclamation point (!) or a pound sign (#) at the beginning of a line
of code indicates a comment line.
!, #
Related Documentation for Cisco Nexus 9000 Series Switches
The entire Cisco Nexus 9000 Series switch documentation set is available at the following URL:
http://www.cisco.com/en/US/products/ps13386/tsd_products_support_series_home.html
Documentation Feedback
To provide technical feedback on this document, or to report an error or omission, please send your comments
to [email protected]. We appreciate your feedback.
Communications, Services, and Additional Information
• To receive timely, relevant information from Cisco, sign up at Cisco Profile Manager.
• To get the business impact you’re looking for with the technologies that matter, visit Cisco Services.
• To submit a service request, visit Cisco Support.
• To discover and browse secure, validated enterprise-class apps, products, solutions and services, visit
Cisco Marketplace.
• To obtain general networking, training, and certification titles, visit Cisco Press.
• To find warranty information for a specific product or product family, access Cisco Warranty Finder.
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xii
Preface
Related Documentation for Cisco Nexus 9000 Series Switches

Cisco Bug Search Tool
Cisco Bug Search Tool (BST) is a web-based tool that acts as a gateway to the Cisco bug tracking system
that maintains a comprehensive list of defects and vulnerabilities in Cisco products and software. BST provides
you with detailed defect information about your products and software.
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xiii
Preface
Preface

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
xiv
Preface
Preface

CHAPTER 1
New and Changed Information
This chapter provides release-specific information for each new and changed feature in the Cisco Nexus 9000
Series NX-OS QoS Configuration Guide, Release.
•New and Changed Information, on page 1
New and Changed Information
This table summarizes the new and changed features for the Cisco Nexus 9000 Series NX-OS Quality of Service
Configuration Guide and where they are documented.
Table 1: New and Changed Features
Where DocumentedChanged
in
Release
DescriptionFeature
Configuring a DSCP Wildcard
Mask, on page 60
7.0(3)I7(9)Support for creating an ACL
that matches or filters traffic
based on a DSCP bit mask.
DSCP Wildcard Mask
Configuring a Link Level Flow
Control Watchdog Interval, on
page 143
7.0(3)I7(8)Support for reacting to LLFC
packets on a PFC-enabled
interface by shutting the
no-drop queue until a timer
resets it.
Link Layer Flow Control
(LLFC) Watchdog
Guidelines and Restrictions for
Link Level Flow Control, on
page 139
7.0(3)I7(8)Support to enable syslog
messages to account all the
incoming global and link level
pause frames.
Enabling syslog messages to
account link level pause frames.
Configuring QoS Pause Buffer
and Dynamic Queue Limits, on
page 163
7.0(3)I7(8)New commands to modify QoS
pause buffers and dynamic
queue limits across all the
interfaces on Cisco Nexus 3164
switches
Configuring QoS Pause Buffer
and Dynamic Queue Limits
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
1

Where DocumentedChanged
in
Release
DescriptionFeature
Guidelines and Limitations for
Priority Flow Control, on page
150
7.0(3)I7(8)Support to enable syslog
messages to account packet
drops on multicast queues for
no-drop class.
Enabling syslog messages to
account packet drops
Guidelines and Limitations for
Priority Flow Control, on page
150
7.0(3)I7(4)Support to write message to
syslog when queue is stuck.
Write message to syslog when
queue is stuck
Guidelines and Limitations for
Priority Flow Control, on page
150
7.0(3)I7(4)Support to drop PFC traffic and
drop all PAUSE frames.
Drop PFC traffic and all
PAUSE frames
Configuring a Priority Flow
Control Watchdog Interval, on
page 158
7.0(3)I7(4)Support to configure PFC
watchdog timers and multiplier
per interface.
Configure PFC watchdog timers
and multiplier per interface
Guidelines and Limitations for
Priority Flow Control, on page
150
7.0(3)I7(4)Support to drop
multicast/broadcast traffic on
no-drop configured class.
Drop multicast/broadcast traffic
on no-drop configured class
Guidelines and Limitations for
Priority Flow Control, on page
150
7.0(3)I7(4)Support to create log entry
when PFC is received on lossy
(non-configured) group.
Create log entry when PFC is
received on lossy group.
Configuring a Priority Flow
Control Watchdog Interval, on
page 158
7.0(3)I7(4)Support for Enable log-only
option for PFC.
Enable log-only option for PFC
Managing Dynamic Buffer
Sharing, on page 130
7.0(3)I7(4)Support for dynamic buffer
sharing across slices.
(Supported only on Cisco
Nexus 93360YC-FX2,
9336C-FX2 and
93240YC-FX2)
Dynamic Buffer Sharing
Enabling Priority Flow Control
on a Traffic Class, on page 155
7.0(3)I7(4)Support to enable NetQoS
pause configuration per qos
class with the pause pfc-cos
cos-list receive command.
Asymmetric PFC
Guidelines and Limitations for
Micro-Burst Monitoring, on
page 173
7.0(3)I7(3)Added support on Cisco Nexus
9300-FX, Cisco Nexus
9300-FX2, and Cisco Nexus
N9K-C9364C switches.
Micro-burst Monitoring and
Detection
Notes for Enabling VLAN QoS,
on page 49
7.0(3)F3(3)Added support for VLAN QoS
for Cisco Nexus 9508 switch.
VLAN QoS
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
2
New and Changed Information
New and Changed Information

Where DocumentedChanged
in
Release
DescriptionFeature
Shared Policers, on page 857.0(3)F3(3)Added support for shared
policer for Cisco Nexus 9508
switch.
Shared Policer
Guidelines and Limitations, on
page 106
Configuring Pause Buffer
Thresholds and Queue Limit
Using Ingress Queuing Policy,
on page 161
7.0(3)F1(1)Added support for the Cisco
Nexus 9636C-R and 9636Q-R
line cards and the 9508-FM-R
fabric module.
Ingress queuing
Configuring Egress Policing,
on page 88
7.0(3)I6(1)Added support for egress
policing per-VLAN.
Egress policing
Configuring WRED on Egress
Queues, on page 114
7.0(3)I6(1)Added the ability to configure
drop thresholds for non-ecn
flows.
WRED
Configuring a Priority Flow
Control Watchdog Interval, on
page 158
7.0(3)I6(1)Added support for PFC
watchdog interval to detect
whether packets in a no-drop
queue are being drained within
a specific time period on Cisco
Nexus 93180YC-FX,
93108TC-FX, and
93180LC-FX, ToR switches
with Cisco Nexus 9736C-EX,
97160YC-EX, 9732C-EX, and
9732C-EXM line cards.
Priority Flow Control Watchdog
Interval
Micro-Burst Monitoring7.0(3)I5(1)Added support for micro-burst
monitoring on Application
Spine Engines (ASE2, ASE3)
and Leaf Spine Engines (LSE).
Micro-Burst Monitoring and
Detection
Approximate Fair Drop7.0(3)I5(1)Added support for AFD, an
Active Queue Management
(AQM) algorithm to avoid
traffic congestion.
AFD (Approximate Fair
Dropping)
About Priority Flow Control,
on page 149
7.0(3)I5(1)Added support for the Cisco
Nexus 9200 and 9300-EX
Series switches.
DCBXP
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
3
New and Changed Information
New and Changed Information

Where DocumentedChanged
in
Release
DescriptionFeature
Dynamic Packet Prioritization7.0(3)I5(1)Added support for DPP. DPP
prioritizes a configured number
of packets of every flow in a
particular class of traffic to be
sent out of another class of
traffic that DPP is mapped to.
DPP (Dynamic Packet
Prioritization)
Guidelines and Limitations, on
page 106
Configuring WRED on Egress
Queues, on page 114
7.0(3)I4(5)Added support for the feature
on the Cisco Nexus 9508
switches and Cisco Nexus
3164Q switches.
Configuring WRED on Egress
Queues
Guidelines and Limitations for
Priority Flow Control, on page
150
Configuring a Priority Flow
Control Watchdog Interval, on
page 158
7.0(3)I4(5)Added support for the feature
on the Cisco Nexus 9508
switches and Cisco Nexus
3164Q switches.
Configuring a Priority Flow
Control Watchdog Interval
Guidelines and Limitations7.0(3)I4(4)Added maximum queue
occupancy support for Leaf
Spine Engine (LSE) enabled
switches.
Maximum queue occupancy
support for Leaf Spine Engines
(LSE)
Configuring Tail Drop on
Egress Queues
7.0(3)I4(4)Added updated alpha values for
dynamic queue size.
Updated alpha values for
dynamic queue size
Configuring Tail Drop on
Egress Queues
7.0(3)I4(4)Added this command to tune
the random noise parameter on
Cisco Nexus 9200 and
9300-EX Series switches.
hardware qos q-noise percent
<value> command
Configuring WRED on Egress
Queues
7.0(3)I4(2)Added a command to allow
non-ECN-capable traffic to
bypass WRED thresholds.
Congestion avoidance
Guidelines and Limitations for
Priority Flow Control
7.0(3)I4(2)Changed the behavior of the no
lldptlv-selectdcbxpcommand
such that PFC mode will be
disabled for interfaces on both
sides of back-to-back switches.
DCBX support
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
4
New and Changed Information
New and Changed Information

Where DocumentedChanged
in
Release
DescriptionFeature
Configuring Priority Flow
Control
7.0(3)I4(2)Added commands to globally
disable PFC on all interfaces
and to configure a watchdog
interval to detect whether
packets in a no-drop queue are
being drained within a specified
time period.
Priority flow control
Using Templates to Configure
TCAM Region Sizes
7.0(3)I3(1)Added support to create and
apply custom TCAM templates.
Custom TCAM template
support
Micro-Burst Monitoring7.0(3)I3(1)Added support for micro-burst
detection
Micro-Burst Detection support
Configuring Priority Flow
Control
7.0(3)I3(1)Added support for DCBX
protocol.
DCBX support
FEX QoS Configuration
Example
7.0(3)I2(1)Added support for FEX QoS
configuration.
FEX QoS configuration support
Attaching a QoS Policy Action
to a VLAN, on page 37
7.0(3)I1(2)Enables label sharing when the
same QoS policy is applied on
multiple VLANs.
Label sharing support for QoS
policies under VLANs
About QoS TCAM Carving7.0(3)I1(2)Added support for 100G
devices.
100G device support
Enabling Egress QoS (IPv4)7.0(3)I1(2)Added support for egress QoS
TCAM carving.
Enabling egress QoS
Configuring Egress Policing7.0(3)I1(1)Added support to apply the
policing instructions in a QoS
policy map to egress packets.
Egress policing support
Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
5
New and Changed Information
New and Changed Information

Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 7.x
6
New and Changed Information
New and Changed Information