Fortinet Network Device IPS User manual

www.fortinet.com
FortiGate
IPS User Guide
Version 3.0 MR7
USER GUIDE

FortiGate IPS User Guide
Version 3.0 MR7
September 16, 2008
01-30007-0080-20080916
© Copyright 2008 Fortinet, Inc. All rights reserved. No part of this
publication including text, examples, diagrams or illustrations may be
reproduced, transmitted, or translated in any form or by any means,
electronic, mechanical, manual, optical or otherwise, for any purpose,
without prior written permission of Fortinet, Inc.
Trademarks
Dynamic Threat Prevention System (DTPS), APSecure, FortiASIC,
FortiBIOS, FortiBridge, FortiClient, FortiGate, FortiGate Unified Threat
Management System, FortiGuard, FortiGuard-Antispam, FortiGuard-
Antivirus, FortiGuard-Intrusion, FortiGuard-Web, FortiLog, FortiAnalyzer,
FortiManager, Fortinet, FortiOS, FortiPartner, FortiProtect, FortiReporter,
FortiResponse, FortiShield, FortiVoIP, and FortiWiFi are trademarks of
Fortinet, Inc. in the United States and/or other countries. The names of
actual companies and products mentioned herein may be the trademarks
of their respective owners.

Contents
FortiGate IPS User Guide Version 3.0 MR7
01-30007-0080-20080916 3
Contents
Introduction ........................................................................................ 5
The FortiGate IPS............................................................................................... 5
About this document......................................................................................... 6
Document conventions.................................................................................. 6
Fortinet documentation .................................................................................... 6
Fortinet Knowledge Center .......................................................................... 8
Comments on Fortinet technical documentation .......................................... 8
Customer service and technical support ........................................................ 8
IPS overview and general configuration.......................................... 9
The FortiGate IPS............................................................................................... 9
IPS settings and controls .............................................................................. 9
When to use IPS ......................................................................................... 10
Network performance...................................................................................... 10
Default signature and anomaly settings ...................................................... 10
Default fail open setting............................................................................... 10
Controlling sessions .................................................................................... 11
Setting the buffer size ................................................................................. 11
Monitoring the network and dealing with attacks ........................................ 11
Configuring logging and alert email............................................................. 11
Attack log messages ................................................................................... 12
The FortiGuard Center ................................................................................ 13
Using IPS sensors in a protection profile ..................................................... 14
Creating a protection profile that uses IPS sensors .................................... 14
Adding protection profiles to firewall policies .............................................. 14
Adding protection profiles to user groups.................................................... 15
Predefined signatures ..................................................................... 17
IPS predefined signatures .............................................................................. 17
Viewing the predefined signature list ............................................................ 17
Fine tuning IPS predefined signatures for enhanced system performance 18
Custom signatures........................................................................... 21
IPS custom signatures.................................................................................... 21
Viewing the custom signature list.................................................................. 21
Custom signature configuration .................................................................... 22
Adding custom signatures using the web-based manager ......................... 22
Adding custom signatures using the CLI..................................................... 22

FortiGate IPS User Guide Version 3.0 MR7
401-30007-0080-20080916
Creating custom signatures........................................................................... 23
Custom signature fields .............................................................................. 23
Custom signature syntax ............................................................................ 24
Example custom signatures........................................................................ 33
Protocol decoders ........................................................................... 37
Protocol decoders........................................................................................... 37
Upgrading the IPS protocol decoder list....................................................... 37
Viewing the protocol decoder list.................................................................. 38
IPS sensors ...................................................................................... 39
Viewing the IPS sensor list............................................................................. 39
Adding an IPS sensor ................................................................................. 40
Configuring IPS sensors................................................................................. 40
Configuring filters ........................................................................................ 42
Configuring pre-defined and custom overrides ........................................... 43
DoS sensors..................................................................................... 45
Viewing the DoS sensor list ........................................................................... 46
Configuring DoS sensors ............................................................................... 46
Understanding the anomalies ........................................................................ 48
SYN flood attacks ............................................................................ 51
What is a SYN flood attack? ........................................................................... 51
How SYN floods work ..................................................................................... 51
The FortiGate IPS Response to SYN flood attacks ...................................... 52
What is SYN threshold?.............................................................................. 52
What is SYN proxy? ................................................................................... 52
How IPS works to prevent SYN floods........................................................ 52
Configuring SYN flood protection ................................................................. 54
Suggested settings for different network conditions .................................. 54
ICMP sweep attacks......................................................................... 55
What is an ICMP sweep? ................................................................................ 55
How ICMP sweep attacks work ...................................................................... 55
The FortiGate IPS response to ICMP sweep attacks.................................... 55
Predefined ICMP signatures ....................................................................... 56
ICMP sweep anomalies .............................................................................. 57
Configuring ICMP sweep protection.............................................................. 58
Suggested settings for different network conditions .................................. 58
Index.................................................................................................. 59

Introduction The FortiGate IPS
FortiGate IPS User Guide Version 3.0 MR7
01-30007-0080-20080916 5
Introduction
This section introduces you to the FortiGate Intrusion Prevention System (IPS)
and the following topics:
•The FortiGate IPS
•About this document
•Fortinet documentation
•Customer service and technical support
The FortiGate IPS
Spam and viruses are not the only threats facing enterprises and small
businesses. Sophisticated, automated attack tools are prevalent on the Internet
today, making intrusion detection and prevention vital to securing corporate
networks. An attack or intrusion can be launched to steal confidential information,
force a costly web site crash, or use network resources to launch other attacks.
The FortiGate IPS detects intrusions by using attack signatures for known
intrusion methods, and detects anomalies in network traffic to identify new or
unknown intrusions. Not only can the IPS detect and log attacks, but users can
choose actions to take on the session when an attack is detected. This guide
describes how to configure and use the IPS and the IPS response to some
common attacks.
This guide describes:
•IPS overview and general configuration
•Predefined signatures
•Custom signatures
•Protocol decoders
•IPS sensors
•DoS sensors
•SYN flood attacks
•ICMP sweep attacks

FortiGate IPS User Guide Version 3.0 MR7
601-30007-0080-20080916
About this document Introduction
About this document
Document conventions
The following document conventions are used in this guide:
• In the examples, private IP addresses are used for both private and public IP
addresses.
• Notes and Cautions are used to provide important information:
Typographic conventions
FortiGate documentation uses the following typographical conventions:
Fortinet documentation
The most up-to-date publications and previous releases of Fortinet™ product
documentation are available from the Fortinet Technical Documentation web site
at http://docs.forticare.com.
The following FortiGate product documentation is available:
• FortiGate QuickStart Guide
Provides basic information about connecting and installing a FortiGate unit.
Note: Highlights useful additional information.
!
Caution: Warns you about commands or procedures that could have unexpected or
undesirable results including loss of data or damage to equipment.
Convention Example
Keyboard input In the Gateway Name field, type a name for the remote VPN
peer or client (for example, Central_Office_1).
Code examples F-SBID (--protocol tcp; --flow
established; --content "content here";
--no_case)
CLI command syntax config firewall policy
edit id_integer
set http_retry_count <retry_integer>
set natip <address_ipv4mask>
end
Document names FortiGate Administration Guide
File content <HTML><HEAD><TITLE>Firewall
Authentication</TITLE></HEAD>
<BODY><H4>You must authenticate to use this
service.</H4>
Menu commands Go to VPN > IPSEC > Phase 1 and select Create New.
Program output Welcome!
Variables <address_ipv4>

Introduction Fortinet documentation
FortiGate IPS User Guide Version 3.0 MR7
01-30007-0080-20080916 7
• FortiGate Installation Guide
Describes how to install a FortiGate unit. Includes a hardware reference,
default configuration information, installation procedures, connection
procedures, and basic configuration procedures. Choose the guide for your
product model number.
• FortiGate Administration Guide
Provides basic information about how to configure a FortiGate unit, including
how to define FortiGate protection profiles and firewall policies; how to apply
intrusion prevention, antivirus protection, web content filtering, and spam
filtering; and how to configure a VPN.
• FortiGate online help
Provides a context-sensitive and searchable version of the Administration
Guide in HTML format. You can access online help from the web-based
manager as you work.
• FortiGate CLI Reference
Describes how to use the FortiGate CLI and contains a reference to all
FortiGate CLI commands.
• FortiGate Log Message Reference
Describes the structure of FortiGate log messages and provides information
about the log messages that are generated by FortiGate units.
• FortiGate High Availability User Guide
Contains in-depth information about the FortiGate high availability feature and
the FortiGate clustering protocol.
• FortiGate IPS User Guide
Describes how to configure the FortiGate Intrusion Prevention System settings
and how the FortiGate IPS deals with some common attacks.
• FortiGate IPSec VPN User Guide
Provides step-by-step instructions for configuring IPSec VPNs using the web-
based manager.
• FortiGate SSL VPN User Guide
Compares FortiGate IPSec VPN and FortiGate SSL VPN technology, and
describes how to configure web-only mode and tunnel-mode SSL VPN access
for remote users through the web-based manager.
• FortiGate PPTP VPN User Guide
Explains how to configure a PPTP VPN using the web-based manager.
• FortiGate Certificate Management Guide
Contains procedures for managing digital certificates including generating
certificate requests, installing signed certificates, importing CA root certificates
and certificate revocation lists, and backing up and restoring installed
certificates and private keys.
• FortiGate VLANs and VDOMs User Guide
Describes how to configure VLANs and VDOMS in both NAT/Route and
Transparent mode. Includes detailed examples.

FortiGate IPS User Guide Version 3.0 MR7
801-30007-0080-20080916
Customer service and technical support Introduction
Fortinet Knowledge Center
Additional Fortinet technical documentation is available from the Fortinet
Knowledge Center. The knowledge center contains troubleshooting and how-to
articles, FAQs, technical notes, and more. Visit the Fortinet Knowledge Center at
http://kc.forticare.com.
Comments on Fortinet technical documentation
Please send information about any errors or omissions in this document, or any
Fortinet technical documentation, to [email protected].
Customer service and technical support
Fortinet Technical Support provides services designed to make sure that your
Fortinet systems install quickly, configure easily, and operate reliably in your
network.
Please visit the Fortinet Technical Support web site at http://support.fortinet.com
to learn about the technical support services that Fortinet provides.

IPS overview and general configuration The FortiGate IPS
FortiGate IPS User Guide Version 3.0 MR7
01-30007-0080-20080916 9
IPS overview and general
configuration
This section contains the following topics:
•The FortiGate IPS
•Network performance
•Monitoring the network and dealing with attacks
•Using IPS sensors in a protection profile
The FortiGate IPS
An IPS is an Intrusion Prevention System for networks. While early systems
focused on intrusion detection, the continuing rapid growth of the Internet, and the
potential for the theft of sensitive data, has resulted in the need for not only
detection, but prevention.
The FortiGate IPS detects intrusions by using attack signatures for known
intrusion methods, and detects anomalies in network traffic to identify new or
unknown intrusions. Not only can the IPS detect and log attacks, but users can
choose actions to take on the session when an attack is detected. This guide
describes how to configure and use the IPS and the IPS response to some
common attacks.
Both the IPS predefined signatures and the IPS engine are upgraded through the
FortiGuard Distribution Network (FDN). These upgrades provide the latest
protection against IM/P2P and other threats. Firmware upgrades will update
anomaly options. The FortiGate IPS default settings implement the recommended
settings for all signatures and anomalies. Signature settings and some anomaly
thresholds are pre-set to work best with the normal traffic on the protected
networks. You can create custom signatures for the FortiGate IPS in diverse
network environments.
Administrators are notified of intrusions and possible intrusions through log
messages and alert email.
Packet logging provides administrators with the ability to analyze packets for
forensics and false positive detection.
IPS settings and controls
Configure the Intrusion Protection system using either the web-based manager or
the CLI, then select IPS sensors in individual firewall protection profiles.
Note: If virtual domains are enabled on the FortiGate unit, the Intrusion Protection settings
are configured separately in each VDOM. All sensors and custom signatures will appear
only in the VDOM in which they were created.

FortiGate IPS User Guide Version 3.0 MR7
10 01-30007-0080-20080916
Network performance IPS overview and general configuration
To create an IPS sensor, go to Intrusion Protection > IPS Sensor. See “IPS
sensors” on page 39 for details. To access the protection profile IPS sensor
selection, go to Firewall > Protection Profile, select Edit or Create New, and
select IPS.
To create a DoS Sensor, go to Intrusion Protection > DoS Sensor. See “DoS
sensors” on page 45 for details.
When to use IPS
IPS is best for large networks or for networks protecting highly sensitive
information. Using IPS effectively requires monitoring and analysis of the attack
logs to determine the nature and threat level of an attack. An administrator can
adjust the threshold levels to ensure a balance between performance and
intrusion prevention. Small businesses and home offices without network
administrators may be overrun with attack log messages and not have the
networking background required to configure the thresholds and other IPS
settings. In addition, the other protection features in the FortiGate unit, such as
antivirus (including grayware), spam filters, and web filters offer excellent
protection for all networks.
Network performance
The FortiGate IPS is extremely accurate and reliable as an in-line network device.
Independent testing shows that the FortiGate IPS successfully detects and blocks
attacks even under high traffic loads, while keeping latency within expected limits.
This section describes:
•Default signature and anomaly settings
•Default fail open setting
•Controlling sessions
•Setting the buffer size
Default signature and anomaly settings
You can use IPS sensors to apply appropriate IPS signatures to different
protection profiles, then different firewall policies.
Default fail open setting
If for any reason the IPS should cease to function, it will fail open by default. This
means that crucial network traffic will not be blocked and the Firewall will continue
to operate while the problem is resolved.
Change the default fail open setting using the CLI:
config ips global
set fail-open [enable | disable]
end