Juniper NETWORKS STRM - TECHNICAL NOTE REV 6-2008 User manual

Release 2008.2
JUNIPER NETWORKS STRM
TECHNICAL NOTE
USING EXTENSION DOCUMENTS
JUNE 2008
Device extensions allow you to modify how a DSM parses logs, which is useful for
resolving parsing issues. However, before you define a device extension, you must
build an extension document. This document provides information on defining an
extension document including:
•About Extension Documents
•Understanding Extension Document Elements
•Creating Extension Documents
•Device Type IDs
This document assumes an advanced knowledge of XML coding.
About Extension
Documents The extension document is specified in Extensible Markup Language (XML)
format. You can create and edit the document using any common word processing
application. You can create multiple extension documents and associate an
extension document to various device types. Using an extension document, you
can resolve parsing issues, such as:
•Fixing an event that has missing or incorrect fields (for example, if the
username is not being parsed).
•Completing the parsing of an event when the DSM to which it is attached fails to
produce a result. Any new events produced by the device extension are
associated to the device that failed to parse the original payload. This action
prevents these events from appearing as un-parsed in the STRM interface.
XML format allows for a simplification of the extension document’s organization,
and verifies the extension document’s contents. Using this format requires that all
regular expressions be contained in character data (CDATA) sections to prevent
the special characters that are needed by regular expressions from interfering with
the markup format. For example:
<pattern id="Protocol" case-insensitive="true" xmlns="">
<![CDATA[(tcp|udp|icmp|gre)]]></pattern>
Where (tcp|udp|icmp|gre) is the actual regular expression pattern.

Release 2008.2
2
The extension document allows you to parse a device’s payload. Within the
extension document you can include statements of varying degrees of complexity,
as required to parse the desired information.
Understanding
Extension
Document
Elements
This section explains the two main divisions of the extension document:
•Patterns
•Match Groups
Patterns Rather than associating a regular expression directly with a particular field name,
patterns (patterns) are declared separately at the top of the extension document
and can be subsequently referenced multiple times within the file.
Match Groups A match group (match-group) is a set of patterns used for parsing or modifying
one or more types of events. A matcher is an entity within a match group that is
parsed (for example, EventName) and is paired with the appropriate pattern and
group for parsing. Any number of match groups may appear in the extension
document.
Table 1 Pattern Parameters
Parameter Description
id
(Required) Specify a regular string that is unique within the extension document.
case-insensitive
(Optional) Specify if you wish the pattern to ignore character case when doing a match, for example abc is the same as ABC.
If not specified, this parameter defaults to false.
trim-whitespace
(Optional) Specify if you wish the pattern to ignore white space and carriage returns. If the CDATA sections are split onto different
lines, this parameter ensures that any extra spaces and
carriage returns are not interpreted as part of the pattern.
If not specified, this parameter defaults to false.
Table 2 Match Group Parameters
Parameter Description
order
(Required) Specify an integer greater than zero to specify the order in which the match groups are executed. It must be unique within the
extension document.
description
(Optional) Specify a description for the match group, which can be any string. This information may appear in the logs.
If not specified, this parameter defaults to empty.

Release 2008.2
Understanding Extension Document Elements 3
Match groups can have up to three different types of entities:
•Matcher (matcher)
•Single-Event Modifier (event-match-single)
•Multi-Event Modifier (event-match-multiple)
Matcher (matcher)
A matcher entity is a field that is parsed (for example, EventName) and is paired
with the appropriate pattern and group for parsing. Matchers have an associated
order, so if multiple matchers are specified for the same field name, the matchers
are executed in the order presented until a successful parse is found or a failure
occurs.
device-type-
id-override
(Optional)
Specify a different device’s QID. Allows the particular match group
to search in the specified device for the event type. It must be a
valid device type ID, represented as an integer. A list of device type
IDs is presented in Table 6.
If not specified, this parameter defaults to the device type of the
device to which the extension is attached.
Table 3 Matcher Entity Parameters
Parameter Description
field
(Required) Specify the field to which you wish the pattern to apply, for example, EventName, or SourceIp. See Table 4 for a
list of valid field names.
pattern-id
(Required) Specify the pattern you wish to use when parsing the field out of the payload. This value must match (including
case) the ID parameter of the pattern previously defined
in a pattern ID parameter (Table 1).
order
(Required) Specify the order that you wish this pattern to attempt among matchers assigned to the same field. If there are
two matchers assigned to the EventName field, the one
with the lowest order is attempted first.
Table 2 Match Group Parameters (continued)
Parameter Description

Release 2008.2
4
capture-group
(Optional) Specify capture group(s), as denoted in the regular expression inside parenthesis ( ). These captures are
indexed starting at one and processed from left to right
in the pattern. The capture-group field must be a
positive integer less than or equal to the number of
capture groups contained in the pattern. The default
value is zero, which will be the entire match.
For example, you can define a single pattern for a
source IP address andport; wherethe SourceIpmatcher
may use a capture group of 1, and the SourcePort
matcher may use a capture group of 2, but only one
pattern needs to be defined.
This field has a dual purpose when combined with the
enable-substitutions parameter.
enable-substitutions
(Optional) Specify this Boolean parameter as true when a field cannot be adequately represented with a straight group
capture.
You can combine multiple groups together with extra
text to form a value. This parameter enables that
behavior.
This parameter changes the meaning of the
capture-group parameter. The capture-group
parameter creates the new value, and group
substitutions are specified using \x where xis a group
number from 1 to 9. You may use groups multiple times,
and any free-form text can also be inserted into the
value. For example, if you need to form a value out of
group 1, followedby an underscore, followedby group 2,
an @, and then group 1 again, the appropriate
capture-group syntax is:
capture-group=”\1_\2@\1”
In another example, a MAC address is separated by
colons, but STRM assumes that MAC addresses are
hyphen separated. The syntax to parse and capture the
individual portions is:
capture-group=”\1:\2:\3:\4:\5:\6”
If no groups are specified in the capture-group when
substitutions are enabled, a direct text replacement
occurs.
Default is false.
Table 3 Matcher Entity Parameters (continued)
Parameter Description

Release 2008.2
Understanding Extension Document Elements 5
Table 4 provides a list of valid field names for use in the matcher field parameter (see
Table 3).
Table 4 Matcher Field Names
Field Name Description
EventName
(Required) Specify the event name to be retrieved from the QID to identify the event.
EventCategory Specify an event category for any event with a category
not handled by an event-match-single entity or an
event-match-multiple entity.
Combined with EventName, EventCategory is used to
search for the event in the QID.
SourceIp Specify the source IP address for the message.
SourcePort Specify the source port for the message.
SourceIpPreNAT Specify the source IP address for the message before
Network Address Translation (NAT) occurs.
SourceIpPostNAT Specify the source IP address for the message after NAT
occurs.
SourceMAC Specify the source MAC address for the message.
SourcePortPreNAT Specify the source port for the message before NAT
occurs.
SourcePortPostNAT Specify the source port for the message after NAT occurs.
DestinationIp Specify the destination IP address for the message.
DestinationPort Specify the destination port for the message.
DestinationIpPreNAT Specify the destination IP address for the message before
NAT occurs.
DestinationIpPostNAT Specify the destination IP address for the message after
NAT occurs.
DestinationPortPreNAT Specify the destination port for the message before NAT
occurs.
DestinationPortPostNAT Specify the destination port for the message after NAT
occurs.
DestinationMAC Specify the destination MAC address for the message.
DeviceTime Specify the time that the event was sent, according to the
device (this is NOT the time that the event arrived).
STRM detects timestamps in the following formats:
•Valid syslog timestamp in the form of mm dd hh:mm:ss,
for example:
Jan 13 12:33:10
•Current locale timestamp
Any other formats will not properly convert.

Release 2008.2
6
Single-Event Modifier (event-match-single)
Single-event modifier (event-match-single) matches (and subsequently
modifies) exactly one type of event, as specified by the required, case-sensitive
EventName parameter. This entity allows mutation of successful events by
changing the device event category, severity, or the method for sending identity
events.
When events matching this event name are parsed, the device category, severity,
and identity properties are imposed upon the resulting event. An
event-match-single entity consists of three optional properties:
Protocol Specify the protocol associated with the event; for
example, TCP, UDP, or ICMP.
If a protocol is not properly parsed out of a message, ports
that were parsed may not appear in STRM (it only displays
ports for port-based protocols).
UserName Specify the user name associated with the event.
HostName Specify the host name associated with the event. This field
is usually only associated with identity events.
GroupName Specify the group name associated with the event. This
field is usually only associated with identity events.
NetBIOSName Specify the NetBIOS name associated with the event. This
field is usually only associated with identity events.
Table 5 Single-Event Modifier Parameters
Parameter Description
device-event-category Specify a new category for searching in the QIDfor the
event. This is an optimizing parameter, since some
devices have the same category for all events.
severity Specify the severity of the event. This parameter must
be an integer value between 1 and 10.
If a severity of less than 1 or greater than 10 is
specified, the system defaults to 5.
If not specified, the default is whatever is found in the
QID.
Table 4 Matcher Field Names (continued)
Field Name Description

Release 2008.2
Creating Extension Documents 7
Multi-Event Modifier (event-match-multiple)
The multi-event modifier (event-match-multiple) matches a range of event
types (and subsequently modifies) as specified by the pattern-id parameter
and the capture-group-index parameter.
Note: This match is not run against the payload, but is run against the results of
the EventName matcher previously parsed out of the payload.
This entity allows mutation of successful events by changing the device event
category, severity, or the method the event uses to send identity events. The
capture-group-index must be an integer value (substitutions are not
supported) and pattern-ID must reference an existing pattern entity. All other
properties are identical to their counterparts in the single-event modifier
Creating Extension
Documents This section provides you with information on creating extension documents
including:
•Writing a Complete Extension Document
•Uploading Extension Documents
•Solving Specific Parsing Issues
send-identity Specifies the sending of identity change information
from the event. Choose one of the following options:
•UseDSMResults – If the DSM returns an identity
event, the event is passed on. If the DSM does not
return an identity event, the DSM does not create or
modify the identity information.
This is the default value if no value is specified.
•SendIfAbsent – If the DSM creates identity
information, the identity event is passed through
unaffected. If no identity event is produced by the
DSM, but there is enough information in the event
to create an identity event, an event is generated
with all the relevant fields set.
•OverrideAndAlwaysSend – Ignores any identity
event returned by the DSM and creates a new
identity event, if there is enough information.
•OverrideAndNeverSend – Suppress any identity
information returned by the DSM.
Table 5 Single-Event Modifier Parameters (continued)
Parameter Description

Release 2008.2
8
Writing a Complete
Extension Document The example of an extension document included in this section provides information on how to parse one particular type of Cisco FWSM so that events are
not sent with an incorrect event name. For example, if you wish to resolve the word
session, which is embedded in the middle of the event name:
Nov 17 09:28:26 129.15.126.6 %FWSM-session-0-302015: Built UDP
connection for faddr 38.116.157.195/80 gaddr
129.15.127.254/31696 laddr 10.194.2.196/2157 duration 0:00:00
bytes 57498 (TCP FINs)
This condition causes the DSM to not recognize any events and all the events are
un-parsed and associated with the generic logger.
Although only a portion of the text string (302015) is used for the QID search, the
entire text string (%FWSM-session-0-302015) identifies the event as coming from
a Cisco FWSM. Since the entire text string is invalid, the DSM assumes that the
event is invalid.
A FWSM device has a large number of event types, many with unique formats.
The following extension document example, indicates how to parse one event
type.
The pattern IDs do not have to match the field names that they are parsing. Even
though the following example duplicates the pattern, the SourceIp field and the
SourceIpPreNAT field could use the exact same pattern in this case (this may not
be true in all FWSM events).
<?xml version="1.0" encoding="UTF-8"?>
<device-extension xmlns="event_parsing/device_extension">
<pattern id="EventNameFWSM" xmlns=""><![CDATA[%FWSM[a-zA-Z\-]*\d-(\d{1,6})]]></pattern>
<pattern id="SourceIp" xmlns=""><![CDATA[gaddr
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
<pattern id="SourceIpPreNAT" xmlns=""><![CDATA[gaddr
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
<pattern id="SourceIpPostNAT" xmlns=""><![CDATA[laddr
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
<pattern id="DestinationIp" xmlns=""><![CDATA[faddr
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
<pattern id="Protocol" case-insensitive="true"
xmlns=""><![CDATA[(tcp|udp|icmp|gre)]]></pattern>
<pattern id="Protocol_6" case-insensitive="true" xmlns=""><![CDATA[ protocol=6]]></pattern>
<pattern id="EventNameId" xmlns=""><![CDATA[(\d{1,6})]]></pattern>
<match-group order="1" description="FWSM Test" device-type-id-override="6" xmlns="">
<matcher field="EventName" order="1" pattern-id="EventNameFWSM" capture-group="1"/>
<matcher field="SourceIp" order="1" pattern-id="SourceIp" capture-group="1" />
<matcher field="SourcePort" order="1" pattern-id="SourceIp" capture-group="2" />
<matcher field="SourceIpPreNAT" order="1" pattern-id="SourceIpPreNAT" capture-group="1" />
<matcher field="SourceIpPostNAT" order="1" pattern-id="SourceIpPostNAT" capture-group="1" />
<matcher field="SourcePortPreNAT" order="1" pattern-id="SourceIpPreNAT" capture-group="2" />
<matcher field="SourcePortPostNAT" order="1" pattern-id="SourceIpPostNAT" capture-group="2" />
<matcher field="DestinationIp" order="1" pattern-id="DestinationIp" capture-group="1" />

Release 2008.2
Creating Extension Documents 9
<matcher field="DestinationPort" order="1" pattern-id="DestinationIp" capture-group="2" />
<matcher field="Protocol" order="1" pattern-id="Protocol" capture-group="1" />
<matcher field="Protocol" order="2" pattern-id="Protocol_6" capture-group="TCP"
enable-substitutions=”true”/>
<event-match-multiple pattern-id="EventNameId" capture-group-index="1"
device-event-category="Cisco Firewall"/>
</match-group>
</device-extension>
The above extension document example demonstrates some of the basic aspects
of parsing:
•IP addresses
•Ports
•Protocol
•Multiple fields using the same pattern with different groups
This example parses all FWSM events that follow the specified pattern, although
the fields that are parsed may not be present in those events (if the events include
different content).
The information that was necessary to create this configuration that was not
available from the event:
•The event name is only the last six digits (302015) of the
%FWSM-session-0-302015 portion of the event.
•The FWSM has a hard-coded device type category of Cisco Firewall.
•The FWSM uses the Cisco Pix QID and therefore includes the
device-type-id-override="6" parameter in the match group (the Pix
firewall’s device type ID is 6, see Table 6).
If the QID information is not specified or is unavailable, you can modify the event
mapping using the Event Viewer. For more information, see the Modifying Event
Mapping section in the STRM Users Guide.
An event name and a device event category is required when looking for the event
in the QID. This device event category is a grouping parameter within the database
that helps define like events within a device. The event-match-multiple at the
end of the match group includes hard-coding of the category. The
event-match-multiple uses the EventNameId pattern on the parsed event
name to match up to six digits. This pattern is not run against the full payload, just
that portion parsed as the EventName field.
The EventName pattern references the %FWSM portion of the events; all Cisco
FWSM events contain the %FWSM portion. The pattern in the example matches
%FWSM followed by any number (zero or more) of letters and dashes. This pattern
match resolves the word session that is embedded in the middle of the event
name that needs to removed. The event severity (according to Cisco), followed by

Release 2008.2
10
a dash and then the true event name as expected by STRM. The only string with a
capture group (that is, bounded by parenthesis) is this pattern of digits
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).
The IP addresses and ports for the event all follow the same basic pattern: an IP
address followed by a slash followed by the numeric port number. This pattern
parses two pieces of data (the IP address and the port), and specifies different
capture groups in the matcher section.
<matcher field="SourceIp" order="1" pattern-id="SourceIp" capture-group="1" />
<matcher field="SourcePort" order="1" pattern-id="SourceIp" capture-group="2" />
Thus, the IP address/port patterns are four sets of one to three digits, separated by
periods followed by a slash and the port number. The IP address section is in a
group, as is the port number (but not the slash). As was previously mentioned, the
matcher sections for these fields reference the same pattern name, but a different
capture group (the IP address is group 1 and the port is group 2).
The protocol is a common pattern that searches the payload for the first instance of
TCP, UDP, ICMP, or GRE (the pattern is marked with the case-insensitive
parameter so that any occurrence matches).
Note: You must search for the protocol when writing extension documents, as
STRM may not display port numbers if the event is not based on a port-based
protocol. See Converting a Protocol for an example of how to search for a protocol.
Although a second protocol pattern does not occur in the event being used as an
example, there is a second protocol pattern defined with an order of two. If the
lowest-ordered protocol pattern does not match, the next one is attempted (and so
on). The second protocol pattern also demonstrates the concept of a direct
substitution; there are no match groups in the pattern, but with the
enable-substitutions parameter enabled, the text TCP can be used in place of
protocol=6.
Uploading Extension
Documents Multiple extension documentscan be created, uploaded, and associated to various device types. Extension documents can be stored anywhere prior to uploading to
STRM. When you select an extension document for uploading, STRM validates
the document against the internal XSD. STRM also verifies the validity of the
document before uploading to the system. For more information about device
extensions, see Managing Sensor Devices Guide.