Symantec Security 5600 Series, Security 5400 Series,Clientless VPN 4400... Instruction Manual

Symantec™ Gateway Security 5000
Series v3.0
Administration Guide
Supported hardware platforms:
Symantec Gateway Security 5600 Series, Symantec Gateway Security 5400 Series,
Symantec Clientless VPN Gateway 4400 Series

Symantec™ Gateway Security 5000 Series v3.0
Administration Guide
Documentation version 8.0
September 16, 2005
Copyright © 2005 Symantec Corporation. All rights reserved.
Federal Acquisitions: Commercial Software - Government Users Subject to Standard License Terms
and Conditions.
Symantec, the Symantec Logo, Bloodhound, NAVEX, LiveUpdate, Striker, Symantec Client Firewall,
Symantec Security Response, and Symantec DeepSight Analyzer are trademarks or registered
trademarks of Symantec Corporation in the United States and certain other countries. Additional
company and product names may be trademarks or registered trademarks of the individual companies
and are respectfully acknowledged.
The product described in this document is distributed under licenses restricting its use, copying,
distribution, and decompilation/reverse engineering. No part of this document may be reproduced in
any form by any means without prior written authorization of Symantec Corporation and its licensors,
if any.
THE DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS,
REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE
DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY
INVALID.
Printed in the United States of America.
10987654321

Technical support
As part of Symantec Security Response, the Symantec global Technical Support group maintains
support centers throughout the world. The Technical Support group’s primary role is to respond to
specific questions on product feature/function, installation, and configuration, as well as to author
content for our Web-accessible Knowledge Base. The Technical Support group works collaboratively
with the other functional areas within Symantec to answer your questions in a timely fashion. For
example, the Technical Support group works with Product Engineering as well as Symantec Security
Response to provide Alerting Services and Virus Definition Updates for virus outbreaks and security
alerts.
Symantec technical support offerings include:
■A range of support options that give you the flexibility to select the right amount of service for any
size organization
■Telephone and Web support components that provide rapid response and up-to-the-minute
information
■Upgrade insurance that delivers automatic software upgrade protection
■Content Updates for virus definitions and security signatures that ensure the highest level of
protection
■Global support from Symantec Security Response experts, which is available 24 hours a day, 7 days
a week worldwide in a variety of languages for those customers enrolled in the Platinum Support
program
■Advanced features, such as the Symantec Alerting Service and Technical Account Manager role,
offer enhanced response and proactive security support
Please visit our Web site for current information on Support Programs. The specific features available
may vary based on the level of support purchased and the specific product that you are using.
Licensing and registration
This product requires a license file. The fastest and easiest way to register your service is to access the
Symantec licensing and registration site at https://licensing.symantec.com.
Contacting Technical Support
Customers with a current maintenance agreement may contact the Technical Support group by phone
or online at www.symantec.com/techsupp.
Customers with Platinum support agreements may contact Platinum Technical Support by the
Platinum Web site at https://www-secure.symantec.com/platinum. When contacting the Technical
Support group, please have the following:
■Product release level
■Hardware information
■Available memory, disk space, NIC information
■Operating system
■Version and patch level
■Network topology
■Router, gateway, and IP address information
■Problem description
■Error messages/log files
■Troubleshooting performed prior to contacting Symantec
■Recent software configuration changes and/or network changes

Customer Service
To contact Enterprise Customer Service online, go to www.symantec.com/techsupp, select the
appropriate Global Site for your country, then select the enterprise Continue link. Customer Service is
available to assist with the following types of issues:
■Questions regarding product licensing or serialization
■Product registration updates such as address or name changes
■General product information (features, language availability, local dealers)
■Latest information on product updates and upgrades
■Information on upgrade insurance and maintenance contracts
■Information on Symantec Value License Program
■Advice on Symantec’s technical support options
■Nontechnical presales questions
■Missing or defective CD-ROMs or manuals

Contents
Chapter 1 Introducing the security gateway
About Symantec Gateway Security 5000 Series v3.0 ...........................................................................................15
Key components of the security gateway ..............................................................................................................15
Firewall technology ...........................................................................................................................................16
Virtual Private Network (VPN) server technology .......................................................................................16
Antispam scanning ............................................................................................................................................17
Antivirus scanning ............................................................................................................................................18
Intrusion detection and prevention ................................................................................................................18
Content filtering ................................................................................................................................................19
High availability/load balancing .....................................................................................................................19
LiveUpdate support ...........................................................................................................................................19
Security Gateway Management Interface ......................................................................................................20
Network security best practices ..............................................................................................................................20
Chapter 2 Becoming familiar with the SGMI
About the SGMI ..........................................................................................................................................................21
Logging on to the SGMI ............................................................................................................................................21
Logging on to the SGMI for the first time ......................................................................................................22
Integrating the SGMI to the desktop ..............................................................................................................24
Logging on to the SGMI from the desktop .....................................................................................................25
Logging on to the SGMI from a browser ........................................................................................................26
Avoiding hostname mismatches .....................................................................................................................27
Using the SGMI home page ......................................................................................................................................29
Viewing Quick Status ........................................................................................................................................29
Accessing commonly used configuration wizards .......................................................................................30
Viewing DeepSight’s ThreatCon status ..........................................................................................................31
Leaving the SGMI .......................................................................................................................................................31
Navigating in the SGMI ............................................................................................................................................33
Using the SGMI menus .....................................................................................................................................35
Using the SGMI toolbar ....................................................................................................................................37
Navigating from the left pane ..........................................................................................................................38
Navigating the right pane .................................................................................................................................43
Using online Help ......................................................................................................................................................45
Displaying Help ..................................................................................................................................................45
Searching Help ...................................................................................................................................................46
Printing Help ......................................................................................................................................................47
Working with configurations of objects .................................................................................................................47
Changing the display of objects in a table .....................................................................................................48
Viewing and modifying object properties ......................................................................................................50
Adding configuration objects ...........................................................................................................................52
Configuring objects that reference other objects .........................................................................................55
Saving and activating configuration changes ...............................................................................................59
Deleting configuration objects ........................................................................................................................61
Using the lower pane when you change configurations ..............................................................................62
Viewing system information ....................................................................................................................................63
Using wizards to simplify configuration ...............................................................................................................64

6Contents
Chapter 3 Managing administrative access
Providing access to the security gateway ..............................................................................................................67
Creating administrator accounts ............................................................................................................................67
Creating machine accounts for security gateway access from remote computers .........................................69
Changing passwords .................................................................................................................................................70
Changing administrator passwords ................................................................................................................71
Changing the root password ............................................................................................................................73
Changing a machine account password .........................................................................................................74
Enabling SSH for command-line access to the appliance ...................................................................................74
Chapter 4 Maintaining your security gateway
Performing maintenance tasks ...............................................................................................................................77
Installing and uninstalling hotfixes .......................................................................................................................77
Configuring and running LiveUpdate ....................................................................................................................79
Defining a LiveUpdate server ..........................................................................................................................79
LiveUpdating components ...............................................................................................................................81
Starting and stopping the security gateway .........................................................................................................84
Rebooting the security gateway appliance ............................................................................................................85
Shutting down the security gateway appliance ....................................................................................................86
Understanding and using licenses ..........................................................................................................................86
Viewing the license status of security gateway components .....................................................................87
Viewing license usage .......................................................................................................................................88
Viewing installed licenses ................................................................................................................................88
Obtaining licenses .............................................................................................................................................89
Installing licenses ..............................................................................................................................................94
Removing all license files .................................................................................................................................95
Enabling and disabling security gateway features .......................................................................................96
Backing up and restoring configurations ..............................................................................................................98
Backing up configuration files from the SGMI .............................................................................................98
Restoring security gateway configuration files from the SGMI .................................................................99
Using command-line utilities to perform a local or remote backup ........................................................101
Making system changes with the System Setup Wizard ...................................................................................104
Adding a network interface ............................................................................................................................104
Modifying a network interface ......................................................................................................................108
Modifying system information ......................................................................................................................109
Configuring network interface properties ...................................................................................................112
Maintaining traffic flow .........................................................................................................................................113
Chapter 5 Establishing your network
Understanding security gateway networking components ..............................................................................115
Deployment scenarios .............................................................................................................................................115
Basic deployment .............................................................................................................................................116
Fault tolerant deployment .............................................................................................................................117
Advanced deployment ....................................................................................................................................118
Enclave deployment ........................................................................................................................................119
Advanced enclave deployment ......................................................................................................................120
Defining security gateway routing .......................................................................................................................125
Understanding static routing ........................................................................................................................125
Understanding dynamic routing ...................................................................................................................126
How the security gateway routes traffic ......................................................................................................127
Configuring static routes ................................................................................................................................128
Configuring dynamic routing ........................................................................................................................129

7Contents
Allowing DHCP traffic .............................................................................................................................................131
How the security gateway handles DHCP traffic ........................................................................................131
Configuring the security gateway to allow DHCP traffic ..........................................................................132
Allowing multicast traffic ......................................................................................................................................135
How the security gateway handles multicast traffic ..................................................................................136
Configuring the security gateway to allow multicast traffic ....................................................................136
About the security gateway’s implementation of DNS ......................................................................................138
Configuring a caching name server with no internal name server .........................................................138
Configuring a caching name server with an internal name server ..........................................................140
Configuring an authoritative name server for a domain ..........................................................................140
Configuring an authoritative name server with delegation .....................................................................142
Configuring enclave DNS ...............................................................................................................................143
Understanding the security gateway’s DNS resource records .................................................................143
Configuring resource records for the security gateway ............................................................................145
DNS alternatives ..............................................................................................................................................153
Enabling reverse lookups ...............................................................................................................................157
Solving DNS problems ....................................................................................................................................157
Chapter 6 Defining your security environment
Identifying the objects used to pass traffic .........................................................................................................159
Defining traffic endpoints with network entities ...............................................................................................160
Configuring a single computer with a host network entity ......................................................................160
Defining a network or subnet with a subnet entity ....................................................................................162
Defining a registered domain with a domain name network entity ........................................................163
Creating security gateway network entities for use in tunnels ................................................................164
Creating a network entity group for rules that apply to multiple entities .............................................166
Defining an entity and security gateway pair with a VPN security entity .............................................167
Understanding how protocols affect traffic ........................................................................................................168
Using protocols that are paired with proxies ..............................................................................................168
Using protocols that are not paired with proxies .......................................................................................169
Viewing reserved ports ...................................................................................................................................173
Configuring custom protocols to handle data from special applications ...............................................178
About service groups ...............................................................................................................................................183
Creating service groups ..................................................................................................................................183
Using service groups to customize protocols for rules ..............................................................................185
Understanding proxies ...........................................................................................................................................187
Configuring a GSP for protocols without proxies .......................................................................................188
Configuring the Oracle Net9 Connection Manager proxy .........................................................................189
Controlling full application inspection of traffic ...............................................................................................192
Defining file control and access ....................................................................................................................192
Sending and receiving files ............................................................................................................................198
Controlling Internet-based data communications .....................................................................................203
Controlling Web traffic ...................................................................................................................................208
Controlling news feeds ....................................................................................................................................216
Synchronizing security gateway time ..........................................................................................................222
Supporting UNIX services ..............................................................................................................................223
Handling streaming audio and video ............................................................................................................225
Managing electronic mail ...............................................................................................................................227
Enabling remote logon ....................................................................................................................................236
Allowing ICMP traffic ......................................................................................................................................238

8Contents
Chapter 7 Limiting user access
Understanding authentication ..............................................................................................................................243
Configuring users for internal authentication ...................................................................................................243
Creating a user account on the internal server ...........................................................................................244
Creating an IKE-enabled user ........................................................................................................................245
Ensuring that the internal server is enabled ...............................................................................................247
Configuring user groups for internal and external authentication .................................................................247
Configuring user groups to authenticate with the internal authentication server ..............................248
Creating an IKE user group ............................................................................................................................250
Importing users and user groups ..................................................................................................................251
Authenticating with an external authentication server ...................................................................................253
Creating authentication server records .......................................................................................................253
Configuring an authentication scheme ........................................................................................................260
Adding an authentication scheme to a rule .................................................................................................261
Authenticating users on external servers ...................................................................................................262
Authenticating using Out-Of-Band Authentication (OOBA) .............................................................................264
Configuring the OOBA service .......................................................................................................................265
Adding OOBA authentication to a rule .........................................................................................................266
Chapter 8 Controlling traffic at the security gateway
How the security gateway controls traffic ..........................................................................................................269
Creating rules and filters to control traffic through the firewall ....................................................................270
Understanding and using rules .............................................................................................................................271
How rules are applied ......................................................................................................................................271
Planning to create rules ..................................................................................................................................272
Configuring rules .............................................................................................................................................272
Rule examples ..................................................................................................................................................280
Configuring HTTP, FTP, and mail (SMTP and POP3) rules with the Firewall Rule Wizard .................284
Using protocols and proxies for specific rules ............................................................................................286
Controlling traffic by date and time .....................................................................................................................287
Configuring a time period range ...................................................................................................................287
Configuring a time period group ...................................................................................................................288
Using packet filters to allow or deny traffic ........................................................................................................289
When to use packet filters ..............................................................................................................................289
Creating a packet filter ...................................................................................................................................290
Understanding packet filter groups .............................................................................................................291
Applying filters and filter groups .................................................................................................................292
Blocking inappropriate content with content filtering .....................................................................................295
Content filtering processing order ...............................................................................................................296
Filtering content by allowing or denying access to defined settings ......................................................296
Filtering by subject matter .............................................................................................................................306
Understanding content filtering and newsgroups .....................................................................................314
Adding content filtering protection to a rule ..............................................................................................316
Chapter 9 Preventing attacks
About preventing attacks .......................................................................................................................................319
Blocking suspicious or malicious traffic with IDS ..............................................................................................319
About intrusion detection and prevention ..................................................................................................320
About IDS/IPS policies ....................................................................................................................................320
Creating IDS/IPS policies ...............................................................................................................................321
Applying IDS/IPS policies ..............................................................................................................................322
Managing intrusion events ............................................................................................................................326
Managing portmap settings ...........................................................................................................................331

9Contents
Protecting your network resources from virus infections ................................................................................333
About antivirus scanning ...............................................................................................................................333
Preventing denial of service attacks .............................................................................................................335
Blocking files that cannot be scanned ..........................................................................................................336
Optimizing scanning resources .....................................................................................................................337
Avoiding potential session time-out errors .................................................................................................340
Blocking mail attachments that are known threats ...................................................................................342
Responding to virus detections .....................................................................................................................344
Adding antivirus protection to a rule ...........................................................................................................347
Troubleshooting antivirus protection ..........................................................................................................349
Increasing productivity by identifying spam email ...........................................................................................349
About the antispam scanning process ..........................................................................................................350
Identifying spam email ...................................................................................................................................351
Reducing false positives .................................................................................................................................356
Adding antispam protection to a rule ...........................................................................................................358
Making your network more secure by hiding addresses ...................................................................................359
Controlling IP addresses with address transforms ....................................................................................359
Mapping addresses with NAT pools ..............................................................................................................361
Redirecting connections to unpublished addresses with service redirections ......................................364
Creating virtual clients by using NAT pools and address transforms ....................................................366
Enabling protection for logical network interfaces ...........................................................................................368
Enabling port scan detection .........................................................................................................................368
Enabling SYN flood protection ......................................................................................................................369
Enabling spoof protection ..............................................................................................................................370
Chapter 10 Providing remote access using VPN tunnels
About VPN tunnels ..................................................................................................................................................373
Understanding gateway-to-gateway tunnels ..............................................................................................374
Understanding Client VPN tunnels ..............................................................................................................374
Tunnel endpoints .............................................................................................................................................375
Tunnel indexes .................................................................................................................................................376
Tunnel communication ...................................................................................................................................376
Tunnel security ................................................................................................................................................377
Types of tunnels ...............................................................................................................................................377
Understanding VPN policies ..................................................................................................................................378
Understanding tunnel negotiation ...............................................................................................................379
Using pre-configured VPN policies ...............................................................................................................379
Creating custom VPN policies ........................................................................................................................380
Creating a VPN policy for IPsec with static key ..........................................................................................382
Viewing or modifying the global IKE policy ................................................................................................384
Configuring tunnels ................................................................................................................................................385
Running the Gateway-to-Gateway Tunnel Wizard ....................................................................................385
Using the Remote Access Tunnel Wizard to create Client VPN tunnels ................................................389
Creating tunnels manually .............................................................................................................................392
Ensuring compliance of remote Client VPN computers ....................................................................................397
Applying client compliance to user groups .................................................................................................398
Simplifying multiple Client VPN computer configuration ...............................................................................399
Delivering Client VPN packages to users .....................................................................................................400
How the Client VPN package is processed on the Symantec Client VPN ................................................400
Importing Client VPN information .......................................................................................................................401
Creating the pkimpvpn file ............................................................................................................................401
Authenticating tunnels using Entrust certificates .............................................................................................402
Multicast traffic through gateway-to-gateway IPsec tunnels ..........................................................................403
How multicast traffic passes through a gateway-to-gateway IPsec tunnel ...........................................404
Configuring multicast support for a gateway-to-gateway IPsec tunnel .................................................406

10 Contents
Chapter 11 Enabling remote access with clientless VPN
About clientless VPN ...............................................................................................................................................409
Clientless VPN concepts .................................................................................................................................410
How clientless VPN controls authentication and remote access .............................................................410
Managing clientless VPN users .............................................................................................................................411
Controlling remote access ......................................................................................................................................412
Defining VPN profiles to allow communication between the security gateway and clientless users ........413
Using rules to allow or deny clientless VPN access ...........................................................................................415
Rule components .............................................................................................................................................415
About simple rules ...........................................................................................................................................415
About advanced rules ......................................................................................................................................419
Using rule sets to group clientless VPN access rules .........................................................................................423
Creating and populating rule sets .................................................................................................................423
Using roles to assign rules to users ......................................................................................................................424
Role structure and inheritance ......................................................................................................................425
Role attributes ..................................................................................................................................................426
Creating and assigning roles ..........................................................................................................................426
Assigning a rule or rule set to a role .............................................................................................................430
Configuring clientless VPN logon policy .....................................................................................................430
Using portal pages to customize the user experience .......................................................................................432
Creating a user portal page ............................................................................................................................433
Creating resource QuickLinks ........................................................................................................................434
Grouping resources .........................................................................................................................................435
Adding resource links to portal pages ..........................................................................................................436
Adding a corporate name and logo ...............................................................................................................437
Adding news items to a portal page ..............................................................................................................437
Removing news items from a portal page ....................................................................................................438
Assign the portal page to a role .....................................................................................................................438
Enabling single sign-on for remote users ............................................................................................................439
Collecting resource logon information ........................................................................................................439
Creating a single sign-on rule ........................................................................................................................439
Deleting user sign-on data .............................................................................................................................440
Using reverse proxy translation ...........................................................................................................................441
Using the Remote Access Tunnel Wizard to set up clientless VPN connections ..........................................442
Clientless VPN connection types ...................................................................................................................442
Advanced mail actions ............................................................................................................................................446
Ensuring client compliance for clientless VPN users ........................................................................................449
Applying client compliance to clientless VPN roles ...................................................................................450
Specifying the SSL cipher suite for data encryption .........................................................................................451
Configuring access to common applications .......................................................................................................452
Identifying resources with URLs ...........................................................................................................................454
Resource URL syntax ......................................................................................................................................455
Chapter 12 Monitoring the security gateway
About monitoring ....................................................................................................................................................461
Viewing system health ............................................................................................................................................462
Monitoring network throughput ...................................................................................................................462
Monitoring system usage and connections .................................................................................................463
Monitoring disk usage .....................................................................................................................................463
Monitoring appliance temperature and fan status ....................................................................................464
Changing the health check poll interval ......................................................................................................465
Viewing quick status .......................................................................................................................................465