Nortel NN46120-104 User manual

Nortel VPN Gateway
User Guide
Release: 7.1
Document Revision: 02.01
www.nortel.com
NN46120-104 216368-G
.

Nortel VPN Gateway
Release: 7.1
Publication: NN46120-104
Document status: Standard
Document release date: 14 April 2008
Copyright © 2007-2008 Nortel Networks
All Rights Reserved.
The information in this document is subject to change without notice. The statements, configurations, technical
data, and recommendations in this document are believed to be accurate and reliable, but are presented without
express or implied warranty. Users must take full responsibility for their applications of any products specified in this
document. The information in this document is proprietary to Nortel Networks.
*Nortel, Nortel Networks, the Nortel logo and the Globemark are trademarks of Nortel Networks.
Export
This product, software and related technology is subject to U.S. export control and may be subject to export or
import regulations in other countries. Purchaser must strictly comply with all such laws and regulations. A license to
export or reexport may be required by the U.S. Department of Commerce.
Licensing
This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit
(https://www.openssl.org/).
This product includes cryptographic software written by Eric Young ([email protected]).
This product includes software written by Tim Hudson ([email protected]).
This product includes software developed by the Apache Software Foundation http://www.apache.org/
This product includes a TAP-Win32 driver derived from the CIPE-Win32 kernel driver, Copyright©Damion K.
Wilson, and is licensed under the GPL.
.

3
.
Contents
Preface 7
Who Should Use This Book 8
Related documentation 9
Product Names 10
How This Book Is Organized 11
Typographic Conventions 13
How to Get Help 14
Getting help from the Nortel Web site 14
Getting help over the phone from a Nortel Solutions Center 14
Getting help from a specialist by using an Express Routing Code 14
Getting help through a Nortel distributor or reseller 14
Introducing the VPN Gateway 15
SSL Acceleration 16
VPN 17
Hardware Platforms 18
Feature List 19
Introducing the ASA 310-FIPS 27
HSM Overview 28
Extended Mode vs. FIPS Mode 29
The Concept of iKey Authentication 30
Additional HSM Information 33
Initial Setup 35
Clusters 36
IP Address Types 37
Ports 38
Interfaces 39
Configuration at Boot Up 41
Installing an NVG in a New Cluster 42
Joining a VPN Gateway to an Existing Cluster 52
Installing an ASA 310-FIPS 58
Reinstalling the Software 70
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

4
Upgrading the NVG Software 73
Performing Minor/Major Release Upgrades 74
Managing Users and Groups 79
User Rights and Group Membership 80
Adding a New User 81
Changing a Users Group Assignment 86
Changing a Users Password 88
Deleting a User 91
Certificates and Client Authentication 93
Generating and Submitting a CSR Using the CLI 94
Adding Certificates to the NVG 99
Update Existing Certificate 107
Configure a Virtual SSL Server to Require a Client Certificate 108
Generating client certificates 110
Managing Revocation of Client Certificates 116
Client certificate support 123
Signing CSRs 124
Generate Test Certificate 126
General Commands 128
Virtual Desktop 131
Running the Virtual Desktop on Client Computers 132
Licensing vdesktop 132
Launch Vdesktop from Portal 133
Virtual Desktop Operations 134
The Command Line Interface 135
Connecting to the VPN Gateway 136
Accessing the NVG Cluster 140
CLI vs. Setup 142
Command Line History and Editing 143
Idle Timeout 144
Troubleshooting the NVG 145
Cannot Connect to VPN Gateway through Telnet or SSH 146
Cannot Add an NVG to a Cluster 148
Cannot Contact the MIP 149
The NVG Stops Responding 151
A User Password is Lost 152
An ASA 310-FIPS Stops Processing Traffic 153
Resetting HSM Cards on the ASA 310-FIPS 155
AnASA 310-FIPS Cluster Must be Reconstructed onto New Devices 158
A User Fails to Connect to the VPN 163
User Unable to Connect to the VPN Gateway through the Net Direct Client 168
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

Licensing 5
Cannot download the NetDirect Zipped file from client PC 171
System Diagnostics 172
Unable to download NetDirect from VPN server 175
Supported Ciphers 177
Cipher List Formats 179
Modifying a Cipher List 180
Supported Cipher Strings and Meanings 181
The SNMP Agent 183
Supported MIBs 184
Supported Traps 189
Syslog Messages 191
List of Syslog Messages 192
Syslog Messages in Alphabetical Order 209
222
License Information 223
HSM Security Policy 233
Definition of Key Codes 253
Syntax Description 254
SSH host keys 257
Methods for Protection 258
The VPN Gateway 259
Adding User Preferences Attribute to Active Directory 261
262
Using the Port Forwarder API 271
General 272
Creating a Port Forwarder 273
Demo Application 274
Creating a Port Forwarder Authenticator 276
Adding a Port Forwarder Logger 279
Connecting Through a Proxy 282
Monitoring the Port Forwarder 283
Glossary 285
Index 295
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

6
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

7
.
Preface
This User’s Guide describes how to perform basic configuration and
maintenance of the Nortel VPN Gateway (NVG).
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

8Preface
Who Should Use This Book
This User’s Guide is intended for network installers and system
administrators engaged in configuring and maintaining a network. It
assumes that you are familiar with Ethernet concepts and IP addressing.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

Licensing 9
Related documentation
For full documentation on installing and using the many features available
in the VPN Gateway software, see the following manuals:
•VPN Gateway 7.1 Command Reference
(part number 216369-F, April 2008)
Describes each command in detail. The commands are listed per
menu, according to the order they appear in the Command Line
Interface (CLI).
•VPN Gateway 6.0 Application Guide for SSL Acceleration
(part number 216370-D, April 2008)
Provides examples on how to configure SSL Acceleration through the
CLI.
•VPN Gateway 7.1 CLI Application Guide for VPN
(part number 216371-F, April 2008)
Provides examples on how to configure VPN deployment through the
CLI.
•VPN Gateway 7.1 BBI Application Guide for VPN
(part number 217239-E, April 2008)
Provides examples on how to configure VPN deployment through the
BBI (Browser-Based Management Interface).
•VPN Gateway 7.1 VPN Administrator Guide
(part number 217238-E, April 2008)
VPN management guide intended for end-customers in a Secure
Service Partitioning configuration.
•VPN Gateway 3050/3070 Hardware Installation Guide
(part number 216213-B, March 2005)
Describes installation of the VPN Gateway 3050 and 3070 hardware
models.
•VPN Gateway 7.1 Configuration-Secure Portable Office Client
(part number 324637-A, April 2008
•VPN Gateway 7.1 Troubleshooting Guide
(part number 324371-B, April 2008)
Describes the prerequisites and various tools used to troubleshoot the
Nortel VPN Gateway (NVG).
•VPN Gateway 7.1 Release Notes
(part number 216372-W, April 2008)
Lists new features available in version 7.1 and provides up-to-date
product information.
The preceding manuals are available for download (see “How to Get Help”
(page 14).
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

10 Preface
Product Names
The software described in this manual runs on several different hardware
models. Whenever the generic terms Nortel VPN Gateway, VPN gateway
or NVG are used in the documentation, the following hardware models
are implied:
•Nortel VPN Gateway 3050 (NVG 3050)
•Nortel VPN Gateway 3070 (NVG 3070)
•Nortel SSL VPN Module 1000 (SVM 1000)
•Nortel SSL Accelerator 310-FIPS (ASA 310-FIPS)
•The integrated SSL Accelerator (SSL processor) on the Nortel
2424-SSL switch
•Nortel VPN Gateway Universal Serial Bus
Similarly, all references to the old product name – iSD-SSL or iSD – in
commands or screen outputs should be interpreted as applying to the
preceding hardware models.
Note: Manufacturing of the Nortel SSL Accelerator (formerly Alteon
SSL Accelerator) has been discontinued.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

11
How This Book Is Organized
The chapters in this book are organized as follows:
Users Guide
“Introducing the VPN Gateway” (page 15) provides an overview of the
major features of the VPN Gateway, including its physical layout and the
basic concepts of its operation.
“Introducing the ASA 310-FIPS” (page 27) provides information about
the ASA 310 equipped with HSM cards, as well as information about the
available security modes and the concept of iKey authentication.
“Initial Setup” (page 35) describes how to install the NVG in a new cluster,
and how to add an NVG to an existing cluster. The chapter also provides
information about the concept of NVG clusters, as well as the usage and
configuration of ports and networks within a cluster. A section describing
how to reinstall the software is also included.
“Upgrading the NVG Software” (page 73) describes how to upgrade the
NVG software for a minor release upgrade, and a major release upgrade,
as well as upgrading from software versions earlier than 2.0.11.16 to
version 3.0.7.
“Managing Users and Groups” (page 79) describes the management
of users, groups, and passwords. The chapter also explains how the
Administrator user role can be fully separated from the Certificate
Administrator user role.
“Certificates and Client Authentication” (page 93) describes how to
generate and prepare keys and certificates for use with the NVG.
“The Command Line Interface” (page 135) describes how to connect to the
NVG and access the information and configuration menus.
“Troubleshooting the NVG” (page 145) provides suggestions for
troubleshooting basic problems. Information about performing system
diagnostics on the NVG is also included, as well as some operations
related to the ASA 310-FIPS model.
Appendices
provides a list of ciphers supported in this product.
“The SNMP Agent” (page 183) provides information about the SNMP
agent on the NVG, and which MIBs (Management Information Bases) are
supported.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

12 Preface
“Syslog Messages” (page 191), contains a list of all syslog messages
that can be sent to a syslog server that is added to the NVG system
configuration.
“License Information” (page 223) provides licensing information for the
software used in this product.
“HSM Security Policy” (page 233) provides detailed information about the
security policy of the CryptoSwift®HSM card that comes installed in the
ASA 310-FIPS.
“Definition of Key Codes” (page 253) provides information about how
to compile a keycode definition file to be used with the Terminal applet
available on the Telnet/SSH tab (located under the Portal’s Advanced tab).
“SSH host keys” (page 257) provides information about the purpose of
SSH host keys and how they are used to protect the connection between
the SSH client and the VPN Gateway.
“Adding User Preferences Attribute to Active Directory” (page
261) provides step-by-step instructions on how to add the User
Preferences attribute to Active Directory. This is required to support
storage of Portal bookmarks in Active Directory.
“Using the Port Forwarder API” (page 271) provides instructions on how to
perform the tasks needed when using the Port Forwarder API. The Port
Forwarder API is used to provide tunnels through the Nortel VPN Gateway
(NVG) without the user having to start any applets from the Portal.
“Glossary” (page 285) includes definitions of terminology used throughout
this document.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

13
Typographic Conventions
The following table describes the typographic styles used in this book.
Table 1
Typographic Conventions
Typeface or
Symbol Meaning Example
AaBbCc123 This type is used for names of
commands, files, and directories used
within the text.
View the readme.tx
tfile.
It also depicts on-screen computer
output and prompts. Main#
AaBbCc123 This bold type appears in command
examples. It shows text that must be
typed in exactly as shown. Main# sys
<AaBbCc123
>This italicized type appears in
command examples as a parameter
placeholder. Replace the indicated
text with the appropriate real name or
value when using the command. Do
not type the brackets.
To establish a Telnet
session, enter:
host# telnet <IP
address>
This also shows book titles, special
terms, or words to be emphasized. Read your User’s Guide thoroughly.
[ ] Command items shown inside
brackets are optional and can be used
or excluded as the situation demands.
Do not type the brackets.
host# ls[-a]
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

14 Preface
How to Get Help
This section explains how to get help for Nortel products and services.
Getting help from the Nortel Web site
The best way to get technical support for Nortel products is from the Nortel
Technical Support Web site: https://www.nortel.com/support/
This site provides quick access to software, documentation, bulletins, and
tools to address issues with Nortel products. From this site, you can:
•download software, documentation, and product bulletins for answers
to technical issues
•sign up for automatic notification of new software and documentation
for Nortel equipment
•open and manage technical support cases
Getting help over the phone from a Nortel Solutions Center
If you do not find the information you require on the Nortel Technical
Support web site, and have a Nortel support contract, you can also get
help over the phone from a Nortel Solutions Center. In North America,
call 1-800-4NORTEL (1-800-466-7835). Outside North America, go
to the following web site to obtain the phone number for your region:
www.nortel.com/callus
Getting help from a specialist by using an Express Routing Code
An Express Routing Code (ERC) is available for many Nortel products and
services. When you use an ERC, your call is routed to a technical support
person who specializes in supporting that product or service. To locate the
ERC for your product or service, go to: http://www.nortel.com/erc/
Getting help through a Nortel distributor or reseller
If you purchased a service contract for your Nortel product from a
distributor or authorized reseller, contact the technical support staff for that
distributor or reseller.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

15
.
Introducing the VPN Gateway
The Nortel VPN Gateway (NVG) software includes two major functionality
groups:
•SSL Acceleration
•VPN
These features can be used separately or be combined. This User’s Guide
covers the basic tasks that need to be completed irrespective of which
feature you wish to deploy.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

16 Introducing the VPN Gateway
SSL Acceleration
The VPN Gateway can function as a peripheral Secure Sockets Layer
(SSL) offload platform that attaches to a Nortel Application Switch or a
comparable switch from another vendor. (The VPN Gateway can also
operate in standalone mode, i.e. without being connected to a switch.)
The VPN Gateway performs a TCP three-way handshake with the client
through the Nortel Application Switch and performs all the SSL encryption
and decryption for the session. Combined with the load balancing
features of the Nortel Application Switch, the VPN Gateway offloads SSL
encryption/decryption functions from back-end servers.
For examples on how to configure the VPN Gateway for SSL Acceleration,
see the Application Guide for SSL Acceleration.
For more information about the basic operations of the VPN Gateway, see
the "Public Key Infrastructure and SSL" chapter in the Application Guide
for SSL Acceleration.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

Getting help through a Nortel distributor or reseller 17
VPN
The VPN feature supports remote access to intranet or extranet resources
(applications, mail, files, intranet web pages) through a secure connection.
What information should be accessible to the remote user after login is
determined by access rules (ACLs).
The intranet’s resources can be accessed in clientless mode, transparent
mode or both:
•From any computer connected to the Internet (clientless mode). The
remote user connects to the VPN Gateway through a secure SSL
connection through the web browser. When successfully authenticated,
the user can access services and resources on the intranet from a
Web Portal provided by the VPN Gateway. Clientless mode also
enables download of the Net Direct client, a simple and secure method
for accessing intranet resources through the remote user’s native
applications.
•From a computer with the Nortel IPsec VPN client (formerly Contivity
VPN client) or the Nortel SSL VPN client installed (transparent mode).
For examples on how to configure the VPN Gateway for VPN deployment,
see the Application Guide for VPN.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

18 Introducing the VPN Gateway
Hardware Platforms
The VPN Gateway software is supported on the following hardware
platforms:
•Nortel VPN Gateway 3050 and 3070
•Nortel SSL VPN Module 1000
•Nortel SSL Accelerator 310 and 410
•Nortel SSL Accelerator 310-FIPS, with FIPS-compliant Hardware
Security Module (HSM). See “Introducing the ASA 310-FIPS” (page
27).
•Nortel 2424-SSL Application Switch
For a detailed technical specification of the hardware platforms, see the
"Specifications" appendix in the VPN Gateway 3050/3070 Hardware
Installation Guide and the Alteon SSL Accelerator Hardware Installation
Guide respectively.
No hardware installation is required for the Nortel Application Switch
2424-SSL. The VPN Gateway software resides on the SSL Processor
which is mounted inside the switch chassis.
Note: Manufacturing of the Nortel SSL Accelerator (formerly Alteon
SSL Accelerator) has been discontinued.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

Software Features 19
Feature List
Software Features
Web Portal
•Web Portal interface for remote users accessing the VPN Gateway in
clientless mode, that is, through the browser.
•Corporate resources available to users as preconfigured group links or
accessible through the Portal tabs.
•Support for native Telnet and SSH (including X11 forwarding) access
to intranet servers through terminal Java applet (available on the
Portal’s Advanced tab).
•Support for handling plugins, Flash and Java applets using HTTP
proxy Java applet (available on the Portal’s Advanced tab).
•Support for application tunneling (port forwarding) through SOCKS
encapsulated in SSL (available on the Portal’s Advanced tab).
•API provided for developing a custom application that automatically
logs in the user to the desired VPN and executes a previously
configured port forwarder link
•Support for customizing the Web Portal, for example, color, logo,
language and company name.
•Three user views available (novice, medium and advanced) to limit
access to Portal tabs.
•Support for automatic redirection of requests to another URL (Portal
pass-through).
•Support for Portal bookmarks.
•Ability to specify domains for which single sign-on is allowed.
•Net Direct client (SSL). VPN client temporarily downloaded from the
Portal and removed when the user exits the session. On Windows, Net
Direct is also available as an installable client (setup.exe file).
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.

20 Introducing the VPN Gateway
Transparent Mode Access
Access to intranet resources in transparent mode, that is, without going
through the Web Portal, is accomplished using Windows VPN clients
installed on the client PCs. In this mode, remote users will experience
network access as if sitting within the local area network. The following
VPN clients are available:
•Nortel SSL VPN client (TDI and LSP version).
•Nortel IPsec VPN client (formerly the Contivity VPN client). Not
supported on the ASA 310, ASA 310-FIPS and ASA 410 hardware
models.
•Net Direct installable client.
User Authentication
User authentication is supported using the following methods:
•RADIUS (including Challenge/Response)
•LDAP (including Microsoft Active Directory)
•NTLM (Windows NT Domain, including Microsoft Active Directory)
•Secure Computing SafeWord (RADIUS)
•Netegrity SiteMinder
•RSA SecurID (native or through RADIUS)
•RSA ClearTrust
•ActivCard (RADIUS)
•Novell NDS/eDirectory (LDAP)
•Client certificate authentication
•Local database authentication
User Authorization
User authorization is controlled through the user’s group membership. Two
different authorization profile types are supported:
•The base profile defines a group member’s access rights to networks,
services and paths.
•The extended profile (optional) also defines a group member’s access
rights depending on conditions related to the user’s connection, for
example, source network, authentication method, access method,
client certificate installed and/or Tunnel Guard checks passed.
Nortel VPN Gateway
User Guide
NN46120-104 02.01 Standard
14 April 2008
Copyright © 2007-2008 Nortel Networks
.