Brocade Communications Systems ServerIron ADX 12.4.00a User manual

53-1002440-03
June 2012
®
ServerIron ADX
Security Guide
Supporting Brocade ServerIron ADX version 12.4.00a

© 2012 Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, MLX, SAN Health, VCS, and VDX are registered trademarks, and
AnyIO, Brocade One, CloudPlex, Effortless Networking, ICX, NET Health, OpenScript, and The Effortless Network are trademarks of
Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names
mentioned may be trademarks of their respective owners.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning
any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to
this document at any time, without notice, and assumes no responsibility for its use. This informational document describes
features that may not be currently available. Contact a Brocade sales office for information on feature and product availability.
Export of technical data contained in this document may require an export license from the United States government.
The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any person or entity with
respect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs that
accompany it.
The product described by this document may contain "open source" software covered by the GNU General Public License or other
open source license agreements. To find out which open source software is included in Brocade products, view the licensing
terms applicable to the open source software, and obtain a copy of the programming source code, please visit
http://www.brocade.com/support/oscd.
Brocade Communications Systems, Incorporated
Document History
Corporate and Latin American Headquarters
Brocade Communications Systems, Inc.
130 Holger Way
San Jose, CA 95134
E-mail: [e-mail]
Asia-Pacific Headquarters
Brocade Communications Systems China HK, Ltd.
No. 1 Guanghua Road
Chao Yang District
Units 2718 and 2818
Beijing 100020, China
Tel: [phone]
Fax: [phone]
E-mail: [email protected]
European Headquarters
Brocade Communications Switzerland Sàrl
Centre Swissair
Tour B - 4ème étage
29, Route de l'Aéroport
Case Postale 105
CH-1215 Genève 15
Switzerland
Tel: [phone]
Fax: [phone]
E-mail: emea-i[email protected]
Asia-Pacific Headquarters
Brocade Communications Systems Co., Ltd. (Shenzhen WFOE)
Citic Plaza
No. 233 Tian He Road North
Unit 1308 – 13th Floor
Guangzhou, China
Tel: [phone]
Fax: [phone]
E-mail: [email protected]
Title Publication number Summary of changes Date
ServerIron ADX Security Guide 53-1002440-01 New document January, 2012
ServerIron ADX Security Guide 53-1002440-02 Corrections made to ACL
chapter
April, 2012
ServerIron ADX Security Guide 53-1002440-03 Updates made to
documentation.
June, 2012

ServerIron ADX Security Guide v
53-1002440-03
Contents
About This Document
Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Supported hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Document conventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Text formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Notes, cautions, and danger notices . . . . . . . . . . . . . . . . . . . . . xiv
Notice to the reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Getting technical help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Chapter 1 Network Security
TCP SYN attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
IP TCP syn-proxy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Granular application of syn-proxy feature . . . . . . . . . . . . . . . . . . . . . . 2
Syn-def . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
show server traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
SYN-def-dont-send-ack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
show server debug. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
No response to non-SYN first packet of a TCP flow . . . . . . . . . . . . . . 4
Prioritizing management traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Protection against attack in hardware . . . . . . . . . . . . . . . . . . . . . 6
Peak BP utilization with TRAP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Show CPU-utilization command enhancement . . . . . . . . . . . . . . 6
BP utilization threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
MP utilization threshold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

vi ServerIron ADX Security Guide
53-1002440-03
Transaction Rate Limit (TRL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Understanding transaction rate limit . . . . . . . . . . . . . . . . . . . . . . 7
Configuring transaction rate limit . . . . . . . . . . . . . . . . . . . . . . . . . 8
Configuring the maximum number of rules . . . . . . . . . . . . . . . . 12
Saving a TRL configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Transaction rate limit command reference . . . . . . . . . . . . . . . . 13
Global TRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
TRL plus security ACL-ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
security acl-id . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Transaction rate limit hold-down value. . . . . . . . . . . . . . . . . . . . 15
Displaying TRL rules statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Displaying TRL rules in a policy. . . . . . . . . . . . . . . . . . . . . . . . . . 15
Displaying IP address with held down traffic . . . . . . . . . . . . . . . 16
Refusing new connections from a specified IP address . . . . . . 16
HTTP TRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Overview of HTTP TRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
HTTP TRL features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Configuring HTTP TRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Configuring HTTP TRL client . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Configuring HTTP TRL defaults . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Sample HTTP TRL configuration . . . . . . . . . . . . . . . . . . . . . . . . . 20
Displaying HTTP TRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Display all HTTP TRL policies. . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Display HTTP TRL policy from index . . . . . . . . . . . . . . . . . . . . . .22
Display HTTP TRL policy client. . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Display HTTP TRL policy starting from index . . . . . . . . . . . . . . .23
Display HTTP TRL policy matching a regular expression. . . . . . 24
Display HTTP TRL policy client index (MP) . . . . . . . . . . . . . . . . . 24
Display HTTP TRL policy client index (BP). . . . . . . . . . . . . . . . . .25
Display HTTP TRL policy for all client entries (BP) . . . . . . . . . . .26
Downloading an HTTP TRL policy through TFTP . . . . . . . . . . . . . . . . 26
HTTP TRL policy commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Client-name <client-name> monitor-interval . . . . . . . . . . . . . . . 27
Client-name <client-name> max-conn . . . . . . . . . . . . . . . . . . . . 27
Client-name <client-name> exceed-action . . . . . . . . . . . . . . . . 28
Default monitor-interval. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Default max-conn. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Default exceed-action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Logging for DoS Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Configuration commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
show server conn-rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Maximum connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
clear statistics dos-attack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Maximum concurrent connection limit per client . . . . . . . . . . . . . . . 32
Limiting the number of concurrent connections per client. . . .32

ServerIron ADX Security Guide vii
53-1002440-03
Firewall load balancing enhancements. . . . . . . . . . . . . . . . . . . . . . . 34
Enabling firewall strict forwarding. . . . . . . . . . . . . . . . . . . . . . . . 34
Enabling firewall VRRPE priority . . . . . . . . . . . . . . . . . . . . . . . . . 34
Enabling track firewall group. . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Enabling firewall session sync delay. . . . . . . . . . . . . . . . . . . . . . 35
Syn-cookie threshhold trap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Service port attack protection in hardware. . . . . . . . . . . . . . . . . . . . 35
Traffic segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
VLAN bridging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Considerations when configuring VLAN bridging. . . . . . . . . . . . 38
Configuring VLAN bridging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Displaying VLAN bridge information . . . . . . . . . . . . . . . . . . . . . . 39
Traffic segmentation using the use-session-for-vip-mac command41
DNS attack protection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Notes: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Configuring DNS attack protection . . . . . . . . . . . . . . . . . . . . . . . 43
Displaying DNS attack protection information. . . . . . . . . . . . . . 46
Chapter 2 Access Control List
How ServerIron processes ACLs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Prior to release 12.3.01 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Beginning with release 12.3.01 and later . . . . . . . . . . . . . . . . . 49
Rule-based ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
How fragmented packets are processed . . . . . . . . . . . . . . . . . . 51
Default ACL action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Types of IP ACLs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
ACL IDs and entries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
ACL entries and the Layer 4 CAM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Aging out of entries in the Layer 4 CAM . . . . . . . . . . . . . . . . . . .53
Displaying the number of Layer 4 CAM entries . . . . . . . . . . . . .53
Specifying the maximum number of CAM entries for rule-based ACLs
54
Configuring numbered and named ACLs. . . . . . . . . . . . . . . . . . . . . . 54
Configuring standard numbered ACLs . . . . . . . . . . . . . . . . . . . . 55
Configuring extended numbered ACLs . . . . . . . . . . . . . . . . . . . . 56
Extended ACL syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Configuring standard or extended named ACLs . . . . . . . . . . . . 62
Displaying ACL definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Displaying ACLs using keywords . . . . . . . . . . . . . . . . . . . . . . . . . 64
Modifying ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Displaying a list of ACL entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Applying an ACLs to interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Reapplying modified ACLs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

viii ServerIron ADX Security Guide
53-1002440-03
ACL logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Displaying ACL log entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Displaying ACL statistics for flow-based ACLs . . . . . . . . . . . . . . 72
Clearing flow-based ACL statistics . . . . . . . . . . . . . . . . . . . . . . . 72
Dropping all fragments that exactly match a flow-based ACL . . . . .72
Clearing the ACL statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Enabling ACL filtering of fragmented packets . . . . . . . . . . . . . . . . . . 73
Filtering fragmented packets for rule-based ACLs. . . . . . . . . . . 73
Enabling hardware filtering for packets denied by flow-based ACLs75
Enabling strict TCP or UDP mode for flow-based ACLs . . . . . . . . . . . 76
Enabling strict TCP mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Enabling strict UDP mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Configuring ACL packet and flow counters. . . . . . . . . . . . . . . . . 78
ACLs and ICMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Using flow-based ACLs to filter ICMP packets based on the IP packet
length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
ICMP filtering with flow-based ACLs . . . . . . . . . . . . . . . . . . . . . . 79
Using ACLs and NAT on the same interface (flow-based ACLs) . . . .82
Displaying ACL bindings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Troubleshooting rule-based ACLs. . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Chapter 3 IPv6 Access Control Lists
IACL overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Configuration Notes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Processing of IPv6 ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Configuring an IPv6 ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Applying an IPv6 ACL to an interface . . . . . . . . . . . . . . . . . . . . . 93
Displaying ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Displaying ACLs bound to an interface. . . . . . . . . . . . . . . . . . . . 94
Using an ACL to Restrict SSH Access. . . . . . . . . . . . . . . . . . . . . . . . .94
Using an ACL to Restrict Telnet Access . . . . . . . . . . . . . . . . . . . . . . . 95
Logging IPv6 ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Chapter 4 Network Address Translation
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Configuring NAT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Configuring static NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Configuring dynamic NAT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
NAT configuration examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
PAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
Forwarding packets without NAT translation. . . . . . . . . . . . . . . . . .103

ServerIron ADX Security Guide ix
53-1002440-03
Translation timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104
Configuring the NAT translation aging timer . . . . . . . . . . . . . .104
Stateless static IP NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105
Redundancy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105
Enabling IP NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .106
Enabling static NAT redundancy . . . . . . . . . . . . . . . . . . . . . . . .106
Enabling dynamic NAT redundancy . . . . . . . . . . . . . . . . . . . . . 107
Displaying NAT information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Displaying NAT statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108
Displaying NAT translation. . . . . . . . . . . . . . . . . . . . . . . . . . . . .110
Displaying NAT redundancy information. . . . . . . . . . . . . . . . . .111
Displaying VRRPE information . . . . . . . . . . . . . . . . . . . . . . . . .112
Clearing NAT entries from the table . . . . . . . . . . . . . . . . . . . . . . . . .112
Chapter 5 Syn-Proxy and DoS Protection
Understanding Syn-Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113
Syn-Proxy auto control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113
Difference between ServerIron ADX and JetCore Syn-Proxy Behavior
113
Configuring Syn-Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114
Setting a minimum MSS value for SYN-ACK packets . . . . . . . 117
Configuring Syn-Proxy auto control . . . . . . . . . . . . . . . . . . . . . .120
Displaying Syn-Proxy Commands . . . . . . . . . . . . . . . . . . . . . . .121
DDoS protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124
Configuring a security filter . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
Configuring a Generic Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
Configuring a rule for common attack types. . . . . . . . . . . . . .127
Configuring a rule for ip-option attack types . . . . . . . . . . . . . .129
Configuring a rule for icmp-type options . . . . . . . . . . . . . . . . .130
Configuring a rule for IPv6 ICMP types. . . . . . . . . . . . . . . . . . .131
Configuring a rule for IPv6 ext header types . . . . . . . . . . . . . .132
Binding the filter to an interface . . . . . . . . . . . . . . . . . . . . . . . .133
Clearing DOS attack statistics. . . . . . . . . . . . . . . . . . . . . . . . . .133
Clearing all DDOS Filter & Attack Counters . . . . . . . . . . . . . . .133
Logging for DoS attacks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133
Displaying security filter statistics . . . . . . . . . . . . . . . . . . . . . .134
Address-sweep and port-scan logging . . . . . . . . . . . . . . . . . . .134

xServerIron ADX Security Guide
53-1002440-03
Chapter 6 Secure Socket Layer (SSL) Acceleration
SSL overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .135
Public Key Infrastructure (PKI) . . . . . . . . . . . . . . . . . . . . . . . . .135
Asymmetric cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Certificate Authority (CA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Certificate Revocation List (CRL) . . . . . . . . . . . . . . . . . . . . . . .136
Cipher suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Digital certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Digital signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Key pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Private key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Public key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137
SSL acceleration on the ServerIron ADX . . . . . . . . . . . . . . . . . . . . .137
SSL Termination Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137
SSL Proxy Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138
ServerIron ADX SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138
Configuring SSL on a ServerIron ADX . . . . . . . . . . . . . . . . . . . . . . .140
Obtaining a ServerIron ADX keypair file . . . . . . . . . . . . . . . . . .140
Certificate management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Converting certificate formats. . . . . . . . . . . . . . . . . . . . . . . . . . 147
Importing keys and certificates. . . . . . . . . . . . . . . . . . . . . . . . .148
Support for SSL renegotiation. . . . . . . . . . . . . . . . . . . . . . . . . .164
Basic SSL profile configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . .164
Specifying a keypair file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165
Specifying a cipher suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165
Specifying a certificate file . . . . . . . . . . . . . . . . . . . . . . . . . . . .166
Advanced SSL profile configuration. . . . . . . . . . . . . . . . . . . . . . . . .166
Configuring client authentication . . . . . . . . . . . . . . . . . . . . . . .166
Enabling session caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . .170
Configuring session cache size. . . . . . . . . . . . . . . . . . . . . . . . .170
Configuring a session cache timeout . . . . . . . . . . . . . . . . . . . . 171
Enabling SSL Version 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Enabling close notify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Disabling certificate verification . . . . . . . . . . . . . . . . . . . . . . . . 171
Enabling a ServerIron ADX SSL to respond with renegotiation
headers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172
Configuring Real and Virtual Servers for SSL Termination and Proxy Mode
172
Configuring Real and Virtual Servers for SSL Termination Mode173
Configuring Real and Virtual Servers for SSL Proxy Mode . . . 174
Configuration Examples for SSL Termination and Proxy Modes . . 176
Configuring SSL Termination Mode . . . . . . . . . . . . . . . . . . . . . 176
Configuring SSL Proxy Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
TCP configuration issues with SSL Terminate and SSL Proxy .178
Other protocols supported for SSL . . . . . . . . . . . . . . . . . . . . . .184
Configuring the system max values . . . . . . . . . . . . . . . . . . . . .185

ServerIron ADX Security Guide xi
53-1002440-03
SSL debug and troubleshooting commands . . . . . . . . . . . . . . . . . .187
Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187
Displaying SSL information . . . . . . . . . . . . . . . . . . . . . . . . . . . .188
Displaying the status of a CRL record . . . . . . . . . . . . . . . . . . .191
Displaying socket information . . . . . . . . . . . . . . . . . . . . . . . . . . . . .199
Displaying SSL Statistics information . . . . . . . . . . . . . . . . . . . .201
Displaying TCP IP information . . . . . . . . . . . . . . . . . . . . . . . . . .205
ASM SSL dump commands. . . . . . . . . . . . . . . . . . . . . . . . . . . .209

xii ServerIron ADX Security Guide
53-1002440-03

ServerIron ADX Security Guide xiii
53-1002440-03
About This Document
Audience
This document is designed for system administrators with a working knowledge of Layer 2 and
Layer 3 switching and routing.
If you are using a Brocade Layer 3 Switch, you should be familiar with the following protocols if
applicable to your network – IP, RIP, OSPF, BGP, ISIS, IGMP, PIM, DVMRP, and VRRP.
Supported hardware and software
Although many different software and hardware configurations are tested and supported by
Brocade Communications Systems, Inc. for 12.3 documenting all possible configurations and
scenarios is beyond the scope of this document.
The following hardware platforms are supported by this release of this guide:
•ServerIron ADX 1000
•ServerIron ADX 4000
•ServerIron ADX 8000
•ServerIron ADX 10000
Document conventions
This section describes text formatting conventions and important notice formats used in this
document.
Text formatting
The narrative-text formatting conventions that are used are as follows:

xiv ServerIron ADX Security Guide
53-1002440-03
For readability, command names in the narrative portions of this guide are presented in bold: for
example, show version.
Notes, cautions, and danger notices
The following notices and statements are used in this manual. They are listed below in order of
increasing severity of potential hazards.
NOTE
A note provides a tip, guidance or advice, emphasizes important information, or provides a reference
to related information.
CAUTION
A Caution statement alerts you to situations that can be potentially hazardous to you or cause
damage to hardware, firmware, software, or data.
DANGER
A Danger statement indicates conditions or situations that can be potentially lethal or extremely
hazardous to you. Safety labels are also attached directly to products to warn of these conditions
or situations.
Notice to the reader
This document may contain references to the trademarks of the following corporations. These
trademarks are the properties of their respective companies and corporations.
These references are made for informational purposes only.
bold text Identifies command names
Identifies the names of user-manipulated GUI elements
Identifies keywords
Identifies text to enter at the GUI or CLI
italic text Provides emphasis
Identifies variables
Identifies document titles
code text Identifies CLI output
Corporation Referenced Trademarks and Products
Sun Microsystems Solaris

ServerIron ADX Security Guide xv
53-1002440-03
Related publications
The following Brocade documents supplement the information in this guide:
•Release Notes for ServerIron Switch and Router Software TrafficWorks 12.2.00
•ServerIron ADX Graphical User Interface
•ServerIron ADX Server Load Balancing Guide
•ServerIron ADX Advanced Server Load Balancing Guide
•ServerIron ADX Global Server Load Balancing Guide
•ServerIron ADX Security Guide
•ServerIron ADX Administration Guide
•ServerIron ADX Switching and Routing Guide
•ServerIron ADX Firewall Load Balancing Guide
•ServerIron ADX Chassis Hardware Installation Guide
•Ironware MIB Reference Manual
Getting technical help
To contact Technical Support, got to http://www.brocade.com/services-support/index.page for the
latest e-mail and telephone contact information..
Microsoft Corporation Windows NT, Windows 2000
The Open Group Linux
Corporation Referenced Trademarks and Products

xvi ServerIron ADX Security Guide
53-1002440-03

ServerIron ADX Security Guide 1
53-1002440-03
Chapter
1
Network Security
TCP SYN attacks
ServerIron software contains many intrusion detection and prevention capabilities. The ServerIron
can be configured to defend against a variety of TCP SYN attacks, Denial of Service (DoS) attacks,
and Smurf attacks.
TCP SYN attacks disrupt normal traffic flow by exploiting the way TCP connections are established.
When a normal TCP connection occurs, the connecting host first sends a TCP SYN packet to the
destination host. The destination host (actually the ServerIron, acting as an intermediary between
the source and destination hosts) responds with a SYN ACK packet. The connecting host then
returns an ACK packet. This process, known as a “TCP three-way handshake”, establishes the TCP
connection.
A TCP SYN attack floods a host with TCP SYN packets. For each of these TCP SYN packets, the
ServerIron responds with a SYN ACK packet and adds an entry to its session table. However, no
ACK packet is actually sent back, so the connection is incomplete. If the attacker sends enough
TCP SYN packets, the session table fills up with incomplete connections, and service can be denied
to legitimate TCP connections.
syn-proxy
IP TCP syn-proxy
Configure the ip tcp syn-proxy command as shown in the following.
1. Configure syn-proxy in the global mode.
ServerIronADX(config)# ip tcp syn-proxy
Syntax: ip tcp syn-proxy
NOTE
You must configure ip tcp syn-proxy command only at the global level, to turn on and off the
global syn-proxy flag.
2. Enable syn-proxy on each interface handling inbound SYN requests (no change here).
ServerIronADX(config)#interface e 3/1
ServerIronADX(config-if-3/1)# ip tcp syn-proxy in
Usage guidelines:
•The default value for a valid ACK time is 32 seconds and is not user configurable.
•If you enter a value, it is ignored. The command remains in the config file the way you enter it,
in case you need to downgrade to the previous release.

2ServerIron ADX Security Guide
53-1002440-03
Granular application of syn-proxy feature
1
•ServerIron may accept the ACK during 33 seconds to 64 seconds due to the syn-proxy
algorithm, but it does not accept the ACK after 64 seconds.
•If you enter a value for the ip tcp syn-proxy <value> command from the CLI or upgrade from an
older release such as 09.4.x to 09.5.2a with the ip tcp syn-proxy <value> command in the
config file, you receive the following warning message.
Warning: The value 10 is being ignored.
Default ACK validate time of 32 seconds will be used.
To change the MSL value, issue 'server msl <value>'.
Granular application of syn-proxy feature
This feature applies to ServerIron ADX Syn-Proxy. When this feature is enabled, traffic destined to a
virtual server IP is denied if the destination port is not defined under any of the virtual server
definitions.
This feature prevents ServerIron ADX from responding with TCP SYN-ACK to TCP SYN for ports not
defined under VIP.
Use the following command to validate traffic against a configured virtual port.
ServerIronADX(config)# server syn-cookie-check-vport
Syntax: [no] server syn-cookie-check-vport
Syn-def
Introduction
Use SYN-def (also known as SYN-Defense) to protect the hosts behind the ServerIron (not the
ServerIron itself) by the ServerIron to complete the TCP three-way handshake on behalf of a
connecting client. There is no SYN-cookie functionality with SYN-def.
NOTE
SYN-Defense is recommened for only where Direct Server Return (DSR) is used. DSR is not
supported with SYN-proxy and is supported with SYN-def. For non DSR scenarios, use Syn-Proxy only.
show server traffic
Use the show server traffic command to display information about the number of times the
incomplete connection threshold was reached.

ServerIron ADX Security Guide 3
53-1002440-03
Syn-def 1
The last line contains information relevant to the incomplete connection threshold. The TCP
SYN-DEF RST field displays the number of times the incomplete connection threshold was reached.
The Server Resets field displays the number of times the ServerIron sent a TCP RESET packet to
the destination real server.
SYN-def-dont-send-ack
The SYN-def feature allows the ServerIron to complete the TCP three-way handshake on behalf of a
connecting client. When a connecting client sends a TCP SYN to a server, the ServerIron forwards
the SYN to the real server, then forwards the SYN ACK from the server to the client. Next, the
ServerIron sends an ACK to the real server, completing the three-way handshake on behalf of the
connecting client. This action allows the real server to move the connection from its pending
connection queue to its established (and much larger) connection queue.
Use the server syn-def-dont-send-ack command to prevent the ServerIron from sending the ACK to
the real server to complete the three-way handshake.
Example
ServerIronADX(config)#server syn-def-dont-send-ack
show server debug
Use the show server debug command to display information about the configuration, as shown in
the following example.
ServerIronADX# show server traffic
Client->Server = 0 Server->Client = 0
Drops = 0 Aged = 0
Fw_drops = 0 Rev_drops = 0
FIN_or_RST = 0 old-conn = 0
Disable_drop = 0 Exceed_drop = 0
Stale_drop = 0 Unsuccessful = 0
TCP SYN-DEF RST = 0 Server Resets = 0
Out of Memory = 0 Out of Memory = 0

4ServerIron ADX Security Guide
53-1002440-03
No response to non-SYN first packet of a TCP flow
1
No response to non-SYN first packet of a TCP flow
ServerIron can remain passive for non-SYN packet in the beginning of the flow. The default
behavior is to send TCP RESET to client when a non-SYN packet is received in the beginning.
SLB-chassis1/1#show server debug
Generic Deug Info
BP Distribution = Enabled JetCore = No
No of BPs = 3 No of Partner BPs = 0
Partner Chassis MAC = 0000.0000.0000
Partner BP1 MAC = 0000.0000.0000 Partner BP2 MAC = 0000.0000.0000
Partner BP3 MAC = 0000.0000.0000 Partner BP4 MAC = 0000.0000.0000
Partner BP5 MAC = 0000.0000.0000 Partner BP6 MAC = 0000.0000.0000
Server Load Balancing Debug Info
Total Get = 3 Total Free = 0
Get Fails = 0 Get Buffer failure = 0
Forward Sp = 0 Reverse Sp = 0
Bad creates = 0 TCP Resets = 0
Fw resets = 0 Rev Resets = 0
Double Free = 0 Error = 0
Free inv Sess Idx = 0 Free list Idx inv = 0
Cache-Reassigns = 0 Trans-Denied = 0
Multi Path Fwd Use = 0 Multi Path Rev Use = 0
Bad non-owner = 0 Select Fwall = 0
FTP-trans-error = 0 Cache track-error = 0
Fw tcp inside move = 0 Fw udp inside move = 0
Fw SYNC delayed = 0 ownership contention = 0
FW stale to conns = 0 FW stale to delq con = 0
FW stale from conns = 0 FW stale from delq c = 0
FW stale from nuke c = 0 Sac frwds = 0
Unxpectd udata = 0 Unxpectd udata(def) = 0
Client->Server = 0 Server->Client = 0
Drops = 0 Aged = 0
Fw_drops = 0 Rev_drops = 0
FIN_or_RST = 0 old-conn = 0
Disable_drop = 0 Exceed_drop = 0
Stale_drop = 0 Unsuccessful = 0
SYN def/proxy RST = 0 Server Resets = 0
Out of Memory = 0 Out of Memory = 0
last conn rate = 0 max conn rate = 0
last TCP attack rate = 0 max TCP attack rate = 0
fast vport found = 0 fast vport n found = 0
Fwd to non-static FI = 0 Dup stale SYN = 0
TCP forward FIN = 0 TCP reverse FIN = 0
Fast path FWD FIN = 0 Fast path REV FIN = 0
Fast path SLB SYN = 0 Dup SYN after FIN = 0
Duplicate SYN = 0 Duplicate sessions = 0
TCP ttl FIN recvd = 0 TCP ttl reset recvd = 0
Sessions in DEL_Q = 0 Sess force deleted = 0
Fwd sess not found = 0 sess already in delQ = 0
Sess rmvd from delQ = 0
Fragment buf full er = 0 Incoming TCP cksum e = 0
New sess sync sent = 0 New sess sync recvd = 0
L4 msg sent = 0 L4 msg recvd = 0
foundry packet sent = 0 ipc packet sent = 2818942
TCP SYN received = 0 TCP SYN dropped = 0
TCP SYN to MP = 0 TCP SYN ACK to MP = 0
TCP SYN ACK received = 0 TCP SYN ACK dropped = 0
TCP pkt received = 0 TCP pkt dropped = 0
TCP pkt to MP = 0 PBSLB tftp status = In progres
Avail. Sessions = 1999996 Total Sessions = 2000000
Hash size = 200001
Total C->S Conn = 0 Total S->C Conn = 0
Total Reassign = 0 Unsuccessful Conn = 0
Server State - 0: diasbled, 1:enabled, 2:failed, 3:test, 4:suspect, 5:grace_dn, 6:active
Real Server St CurrConn TotConn TotRevConn CurrSess PeakConn
R1 1 0/0/0 0 0 0 0
rs1 1 0/0/0 0 0 0 0

ServerIron ADX Security Guide 5
53-1002440-03
Prioritizing management traffic 1
By default, when ServerIron ADX receives TCP packet that is destined to VIP and there is no session
match then it sends TCP reset to the sender. However, if one desires to remain passive then the
above feature can be enabled.
To not send the reset packet, use the following command.
ServerIronADX(config)# server reset-on-syn-only
To remove the configuration, use the following command.
ServerIronADX(config)# no server reset-on-syn-only
Syntax: [no] server reset-on-syn-only
Prioritizing management traffic
ServerIron ADX software allows the system to prioritize traffic destined to the management IP
address in order to facilitate uninterrupted access to the ServerIron switch even under heavy load
conditions. This feature allows you to prioritize management traffic based on the following.
1. Client IP address/subnet
2. Protocol (TCP/UDP/IP) and
3. TCP or UDP port number
With this feature turned on, the specified traffic is directly forwarded to the Management Module in
hardware. In the following example, traffic from the source subnet 1.1.1.1 and destined to
management IP 10.45.16.104 for TCP port 22 (SSH) is prioritized.
ServerIronADX(config)# server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0
10.45.16.104 6 22
Syntax: server prioritize-mgmt-traffic <source ip> <mask> <destination ip> [<protocol>] [<port>]
The <source ip> variable specifies the Source IP address.
The <mask> variable specifies the Mask for the source IP address.
The <destination ip> variable specifies the Destination management IP address. The destination IP
address must already be configured on the ServerIron ADX. If the IP address is not configured, the
command is rejected.
The <protocol> variable specifies any protocol.
The <port> variable specifies a TCP or UDP port.
It is also possible to prioritize management traffic from any source ip as shown in the example
below.
ServerIronADX(config)# server prioritize-mgmt-traffic any 10.45.16.104 6 22
Syntax: [no] server prioritize-mgmt-traffic any <destination ip> [<protocol>] [<port>]
NOTE
The prioritizing management traffic feature should not be enabled for a ServerIron ADX router VE
address if this interface is used for source-NAT as that would break the SLB traffic flow.
Refer to the following examples.
Prioritization of TCP port 80 traffic to management IP 200.1.1.1

6ServerIron ADX Security Guide
53-1002440-03
Peak BP utilization with TRAP
1
ServerIronADX# server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0 200.1.1.1 6
80
Prioritization of TCP port 80 traffic to management IP 200.1.1.1 from any source IP address
ServerIronADX# server prioritize-mgmt-traffic any 200.1.1.1 6 80
Prioritization of UDP port 2222 traffic to management IP 200.1.1.1
ServerIronADX# server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0 200.1.1.1 17
2222
Prioritization of IP protocol 89 (OSPF) traffic to management IP 200.1.1.1
ServerIronADX# server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0 200.1.1.1 89
Protection against attack in hardware
ServerIron ADX allows for protection against attack in hardware without impacting MP or BP CPU
utilization. Configure the server the drop-all-mgmt-access command to drop all traffic destined to a
specified management IP address.
The following command drops all traffic destined to the management IP address 10.45.16.104.
ServerIronADX(config)# server drop-all-mgmt-access 10.45.16.104
Syntax: [no] server drop-all-mgmt-access <destination ip>
NOTE
For a router, the destination IP address is the physical or ve interface IP address For a switch, the
destination IP address is the management IP address.
The server drop-all-mgmt-access feature when used in combination with the server
prioritize-mgmt-traffic feature allows you to prioritize valid traffic while blocking unwanted traffic
destined to the management IP address.
For example, with the following configuration, only ssh, telnet and http traffic destined to
management IP address 10.45.16.104 will be prioritized and all other traffic destined to
10.45.16.104 will be dropped.
ServerIronADX(config)#server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0
10.45.16.104 6 22
ServerIronADX(config)#server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0
10.45.16.104 6 23
ServerIronADX(config)#server prioritize-mgmt-traffic 1.1.1.1 255.255.255.0
10.45.16.104 6 80
ServerIronADX(config)#server drop-all-mgmt-access 10.45.16.104
Peak BP utilization with TRAP
Show CPU-utilization command enhancement
The show cpu-utilization command displays CPU utilization peaks since the system boot or the last
reset of counters (using the clear cpu utilization command).
The command, clear cpu-utilization, on both the MP and the BP is used to reset the counter.