Juniper STRM User manual

Juniper Networks, Inc.
1194 North Mathilda Avenue
Sunnyvale, CA 94089
USA
408-745-2000
www.juniper.net
Published: 2013-07-19
Security Threat Response Manager
STRM Troubleshooting Guide
Release 2013.2

2
Copyright Notice
Copyright © 2013 Juniper Networks, Inc. All rights reserved.
Juniper Networks, Junos, Steel-Belted Radius, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and
other countries. The Juniper Networks Logo, the Junos logo, and JunosE are trademarks of Juniper Networks, Inc.
The following terms are trademarks or registered trademarks of other companies:
JavaTM and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
All other trademarks, service marks, registered trademarks, or registered service marks are the property of their respective owners.
All specifications are subject to change without notice. Juniper Networks assumes no responsibility for any inaccuracies in this document or for any
obligation to update information in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication
without notice.
FCC Statement
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A
digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the
equipment is operated in a commercial environment. The equipment generates, uses, and can radiate radio-frequency energy and, if not installed and
used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential
area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense. The following
information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it
is not installed in accordance with Juniper Networks’ installation instructions, it may cause interference with radio and television reception. This
equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC
rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no
guarantee that interference will not occur in a particular installation. If this equipment does cause harmful interference to radio or television reception,
which can be determined by turning the equipment off and on, the user is encouraged to try to correct the interference by one or more of the following
measures: Reorient or relocate the receiving antenna. Increase the separation between the equipment and receiver. Consult the dealer or an
experienced radio/TV technician for help. Connect the equipment to an outlet on a circuit different from that to which the receiver is connected.
Caution: Changes or modifications to this product could void the user's warranty and authority to operate this device.
Disclaimer
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT, SUBJECT TO THE MODIFICTAIONS SET FORTH
BELOW ON THIS PAGE, ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED
HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR
JUNIPER NETWORKS REPRESENTATIVE FOR A COPY.
STRM Troubleshooting Guide
Release 2013.2
Copyright © 2013, Juniper Networks, Inc.
All rights reserved. Printed in USA.
Revision History
July 2013—STRM Troubleshooting Guide
The information in this document is current as of the date listed in the revision history.
END USER LICENSE AGREEMENT
The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with) Juniper Networks software. Use
of such software is subject to the terms and conditions of the End User License Agreement (“EULA”) posted at http://www.juniper.net/support/eula.html,
as modified by the following text, which shall be treated under the EULA as an Entitlement Document taking precedence over any conflicting provisions
of such EULA as regards such software:
As regards software accompanying the STRM products (the “Program”), such software contains software licensed by Q1 Labs and is further
accompanied by third-party software that is described in the applicable documentation or materials provided by Juniper Networks.

3
For the convenience of Licensee, the Program may be accompanied by a third party operating system. The operating system is not part of the Program,
and is licensed directly by the operating system provider (e.g., Red Hat Inc., Novell Inc., etc.) to Licensee. Neither Juniper Networks nor Q1 Labs is a
party to the license between Licensee and the third party operating system provider, and the Program includes the third party operating system “AS IS”,
without representation or warranty, express or implied, including any implied warranty of merchantability, fitness for a particular purpose or
non-infringement. For an installed Red Hat operating system, see the license file: /usr/share/doc/redhat-release-server-6Server/EULA.
By downloading, installing or using such software, you agree to the terms and conditions of that EULA as so modified.

4

CONTENTS
ABOUT THIS GUIDE
Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Technical Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Requesting Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1STRM SYSTEM NOTIFICATIONS
Performance Degradation of Disk Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Verifying the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Increasing the Partition Test Timeout Period . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Application Error after Protocol Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Purging STRM files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Disk Usage System Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Verifying Disk Usage Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Resolving Disk Usage Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
User Configurations that Impact Event Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 13
DSM Extensions and Optimized Custom Properties . . . . . . . . . . . . . . . . . . . . . . 14
Identifying DSM and Optimized Custom Property Issues. . . . . . . . . . . . . . . . . . . 14
Non-Optimized Custom Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Rule Tests that Impact Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Global Views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Incomplete Report Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Resolving Missing Report Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Limited Disk Space to Perform Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Verifying the Backup Partition Disk Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Resolving Backup Partition Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

STRM Troubleshooting Guide
ABOUT THIS GUIDE
The STRM Troubleshooting Guide provides diagnostic and resolution information
for common system notifications and errors that can be displayed when using
STRM.
Audience This guide is intended for all STRM users responsible for investigating and
managing network security. This guide assumes that you have STRM access and
a knowledge of your corporate network and networking technologies.
Documentation
Conventions Table 1 lists conventions that are used throughout this guide.
Technical
Documentation You can access technical documentation, technical notes, and release notes
directly from the Juniper customer support website at
https://www.juniper.net/support/. Once you access the Juniper customer support
website, locate the product and software release for which you require
documentation.
Your comments are important to us. Please send your e-mail comments about this
guide or any of the Juniper Networks documentation to:
[e-mail].
Include the following information with your comments:
•Document title
Table 1 Icons
Icon Type Description
Information note Information that describes important features or
instructions.
Caution Information that alerts you to potential loss of
data or potential damage to an application,
system, device, or network.
Warning Information that alerts you to potential personal
injury.

STRM Troubleshooting Guide
8ABOUT THIS GUIDE
•Page number
Requesting
Technical Support Technical product support is available through the Juniper Networks Technical
Assistance Center (JTAC). If you are a customer with an active J-Care or JNASC
support contract, or are covered under warranty, and need postsales technical
support, you can access our tools and resources online or open a case with JTAC.
•JTAC policies—For a complete understanding of our JTAC procedures and
policies, review the JTAC User Guide located at
http://www.juniper.net/us/en/local/pdf/resource-guides/7100059-en.pdf .
•Product warranties—For product warranty information, visit
http://www.juniper.net/support/warranty/ .
•JTAC Hours of Operation —The JTAC centers have resources available 24
hours a day, 7 days a week, 365 days a year.
Self-Help Online Tools and Resources
For quick and easy problem resolution, Juniper Networks has designed an online
self-service portal called the Customer Support Center (CSC) that provides you
with the following features:
•Find CSC offerings: http://www.juniper.net/customers/support/
•Search for known bugs: http://www2.juniper.net/kb/
•Find product documentation: http://www.juniper.net/techpubs/
•Find solutions and answer questions using our Knowledge Base:
http://kb.juniper.net/
•Download the latest versions of software and review release notes:
http://www.juniper.net/customers/csc/software/
•Search technical bulletins for relevant hardware and software notifications:
https://www.juniper.net/alerts/
•Join and participate in the Juniper Networks Community Forum:
http://www.juniper.net/company/communities/
•Open a case online in the CSC Case Management tool:
http://www.juniper.net/cm/
To verify service entitlement by product serial number, use our Serial Number
Entitlement (SNE) Tool: https://tools.juniper.net/SerialNumberEntitlementSearch/
Opening a Case with JTAC
You can open a case with JTAC on the Web or by telephone.
•Use the Case Management tool in the CSC at http://www.juniper.net/cm/ .
•Call 1-888-314-JTAC (1-888-314-5822 toll-free in the USA, Canada, and
Mexico).
For international or direct-dial options in countries without toll-free numbers, visit
us at http://www.juniper.net/support/requesting-support.html.

STRM Troubleshooting Guide
1STRM SYSTEM NOTIFICATIONS
System notifications are displayed on the STRM dashboard or in the notification
window when unexpected system behavior occurs. You can troubleshoot the most
common STRM notifications.
Error messages can occur for a variety of reasons. After consulting this guide, if
you are unable to resolve a STRM error or system notification message, gather
diagnostic information and contact Juniper Customer Support.
Performance
DegradationofDisk
Storage
Each host in your STRM deployment monitors the availability of partitions using
hostcontext. Disk availability is tested every minute by opening, writing to, and
deleting a file.
If this process takes longer than the default time period of five seconds, then the
hostcontext process reports an error in the STRM logs.
The error might resemble the following output:
Jun 24 07:22:41 127.0.0.1 [hostcontext.hostcontext]
[5b3acf9a-aa8a-437a-b059-01da87333f43/SequentialEventDispatcher
] com.q1labs.hostcontext.ds.DiskSpaceSentinel: [ERROR]
[NOT:0150062100][172.16.77.116/- -] [-/- -]The storage
partition(s) /store/backup on qradarfc (172.16.77.116) are not
currently accessible. Manual intervention may be required to
restore normal operation.
NOTE
If your system is experiencing high loading and large volumes of data are being
written, searched, purged, or copied to another system, an error might be
displayed when your file system is still operational.
You must identify the frequency of the error message, by choosing one of the
following options:
•If the message is displayed repeatedly, then verify the problem, see Verifying
the Problem.

STRM Troubleshooting Guide
10 STRM SYSTEM NOTIFICATIONS
•If the message is only displayed during peak times, then increase your partition
test timeout period, see Increasing the Partition Test Timeout Period.
Verifying the Problem You can verify a partition storage problem by creating a temporary file on your
STRM Console or Managed Host.
About this task
Partition storage problems can occur on the Console or any Managed Host in your
STRM deployment.
Procedure
Step 1 Using SSH, log in to the STRM Console or Managed Host as the root user:
Username: root
Password: <password>
Step 2 Type the following commands:
touch /store/backup/testfile
ls -la /store/backup/testfile
Step 3 If either of the following messages are displayed, then go to Step 4.
touch: cannot touch `/store/backup/testfile': Read-only file
system
nfs server time out
Step 4 Choose from one of the following options:
•If you are using a network file system, such as iSCSI, Fibre Channel or NFS,
then contact your storage administrator to verify that the file servers are
accessible and operational.
•If you are using a local file system on your STRM appliance, you might have a
file system issue or your disk might have failed. contact Juniper Customer
Support.
•If you are unable to identify the cause of your problem, contact Juniper
Customer Support.
Increasing the
Partition Test
Timeout Period
You can modify the partition test timeout period.
About this task
The partition test timeout period must be increased to a level at which STRM does
not generate false positives, but remains operational. Do not increase the timeout
period to a level that is excessive.
Procedure
Step 1 Click the Admin tab.
Step 2 On the navigation menu, click System Configuration.

STRM Troubleshooting Guide
Application Error after Protocol Update 11
Step 3 Click the System Settings icon.
Step 4 In the Partition Tester Timeout (seconds) list box, select or type 20 seconds.
Step 5 Click Save.
Application Error
after Protocol
Update
You might receive an error message when you attempt to edit a log source if you
recently upgraded STRM or updated Device Service Module (DSM), Protocol, or
Vulnerability Information Services (VIS) components.
The message indicates that the web server might not have started after STRM was
updated. The web server might be storing old files in memory. To remove these
files you must purge your STRM files. See Purging STRM files.
An error has occurred. Refresh your browser (press F5) and
attempt the action again. If the problem persists, please
contact Juniper Customer Support for assistance.
Purging STRM files You can clear STRM files from your browser cache.
Before you begin
Ensure that you only have one instance of your web browser open, otherwise the
cache cannot be cleared. If you are using Mozilla Firefox, you must clear the cache
in Internet Explorer and Mozilla Firefox.
Procedure
Step 1 Using SSH, log in to the STRM Console as the root user:
Username: root
Password: <password>
Step 2 Stop tomcat by typing the following command:
service tomcat stop
Step 3 Clear your browser’s cache.
Step 4 Restart tomcat by typing the following command:
service tomcat start
Step 5 If the problem persists then contact Juniper Customer Support.
Disk Usage System
Notifications The STRM disksentinel process monitors the /root, /store, and /store/tmp partitions
in your deployment to determine if these partitions have reached a pre-defined
usage threshold.

STRM Troubleshooting Guide
12 STRM SYSTEM NOTIFICATIONS
Depending on the disk usage of each monitored partition, the hostcontext process
might display the following system notifications:
NOTE
The /var/log partition can continue to operate when disk usage reaches 100%.
However, log data will not be written to disk and this can affect STRM startup
processes and components. For more information, see Resolving Disk Usage
Issues.
Verifying Disk Usage
Levels You can verify the usage levels of the partitions on your STRM Console or Managed Host.
Procedure
Step 1 Using SSH, log in to the STRM Console or Managed Host as the root user:
Username: root
Password: <password>
Step 2 Type the following command:
df -h
Step 3 Review the partitions to check their disk usage levels.
If any of the monitored partitions have reached 95%, review the recommended
solutions to this problem. For more information, see Resolving Disk Usage
Issues.
Table 3-1 Disk Usage Notifications
Notification Description
Disk Sentry:
Disk Usage
exceeded
warning
threshold.
This message is displayed when disk usage reaches 90% on
any of the monitored partitions. The operation of your STRM
system is not affected when the partition reaches this
threshold. Continue to monitor your partition levels. For more
information, see Verifying Disk Usage Levels.
Disk Sentry:
Disk Usage
exceeded max
threshold.
This message is displayed when disk usage reaches 95% on
any of the monitored partitions. STRM data collection (ecs)
and search processes (ariel) are shut down in order to protect
the file system from reaching 100%. For more information,
see Resolving Disk Usage Issues.
Disk sentry:
System disk
usage back to
normal
levels.
After disk usage has reached a threshold of 95%, disk usage
must return to 92% before STRM automatically restarts data
collection and search processes. To lower the disk usage
threshold, manually remove data from the affected partitions.
For more information, see Resolving Disk Usage Issues.

STRM Troubleshooting Guide
User Configurations that Impact Event Processing 13
Resolving Disk
Usage Issues You can resolve disk usage issues.
About this task
Disk usage warnings might occur on the Console or any Managed Host in your
STRM deployment. Your file system partitions can reach 95% when your data
retention period settings are too high or you have insufficient storage available for
the rate at which STRM receives data.
NOTE
If you reconfigure your retention bucket storage settings, this will have a global
effect on the storage across your entire STRM deployment.
Procedure
1 In the /root file system, identify and remove older debug or patch files.
2 Reduce disk usage on the /store file system. Choose one of the following options:
- Remove the oldest data from the /store/ariel/events file system. If you are
not familiar with UNIX commands or performing large scale data removal,
then contact Juniper Customer Support.
- Reduce your data retention period by adjusting the default retention bucket
storage settings. For more information, see the STRM Administration Guide.
- Identify which log sources you can retain for shorter periods and use the
retention buckets feature to manage this. For more information, see the
STRM Administration Guide.
- Consider an offboard storage solution. For example, iSCSI or Fibre
Channel. For more information, see the STRM Offboard Storage Guide.
3 In the /store/tmp file system, if you identify that a large Log Activity or Network
Activity export has occurred, contact Juniper Customer Support for assistance with
removing data from your system.
4 If the /var/log file system reaches 100% capacity, STRM will not shut down.
However, there might be other issues which will cause your log files to grow faster
than expected. Contact Juniper Customer Support.
User
Configurations that
Impact Event
Processing
Depending on your STRM configuration, the event processing pipeline can be
severely impacted.
Administrators must review the following information to ensure that event
processing is not affected:
•DSM Extensions and Optimized Custom Properties
•Non-Optimized Custom Properties
•Rule Tests that Impact Performance
•Global Views

STRM Troubleshooting Guide
14 STRM SYSTEM NOTIFICATIONS
DSM Extensions and
Optimized Custom
Properties
STRM performance can be affected by the configuration of your DSM extensions
and optimized custom properties.
DSM Extensions
Using a DSM extension, you can create custom parsing methods, based on regex
pattern matching, to extract event data from unsupported log sources. As DSM
extensions are used by the STRM parsing engine, the regex patterns used in your
extension can impact event processing. For more information see, Identifying
DSM and Optimized Custom Property Issues.
Optimized Custom Properties
You can use regular expression patterns to extract data from events as they are
parsed. If regular expressions are written inefficiently, they can degrade the
performance of the STRM parsing engine and impact event processing.
Issues with DSMs or optimized custom properties can cause the following system
notification to be displayed. For more information see, Identifying DSM and
Optimized Custom Property Issues.
Performance degradation has been detected in the event pipeline.
Events were routed directly to storage.
Identifying DSM and
Optimized Custom
Property Issues
You can identify issues with any recently installed a DSM extension or newly
enabled custom property.
Procedure
1 Disable any recently installed DSM extension or custom property.
2 If STRM stops dropping events, but you continue to receive a system notification,
then review your DSM extensions or custom properties to identify inefficient regex
patterns.
3 If STRM continues dropping events, there might be multiple DSM extensions or
custom properties that are causing a problem with the event pipepline.
4 If the issue persists after you have disabled all DSM extensions and custom
properties, contact Juniper Customer Support.
Non-Optimized
Custom Properties Custom properties that are regularly used by STRM rules, or for searching and filtering, must be marked as Optimized.
In cases where they are not optimized, the data is parsed by the UI engine
(tomcat). This can affect search speeds and UI load times. For more information
on optimizing custom properties, see the STRM Users Guide.
If you experience performance impact, contact Juniper Customer Support.

STRM Troubleshooting Guide
Incomplete Report Results 15
Rule Tests that
Impact Performance The rules and tests that you configure in STRM can affect performance.
Regular expressions tests
Rules that test if the event payload contains or matches a regular expression,
perform a search of the entire payload and have a greater impact on STRM
performance.
Before you add a payload test to a rule, include filters in the rule that reduce the
number of events. For example, to search for a specific message that is only
contained in the Active Directory Logs, first apply the following filters to the rule:
•Log source type
•Log source group or specific log source filter
•Optional. Source IP
Host with port open tests
The host with port open test can impact STRM performance because it
compares passive and active ports with the events and flows received by STRM.
Before using this test, perform a bidirectional check to ensure that the host
responds to the communication request.
Global Views Creating a saved search that is grouped by multiple fields can generate a global
view with a large number of unique entries. As the volume of data increases, disk
usage, processing times, and search performance can be impacted.
To prevent this, only aggregate searches on fields that are necessary. You could
also reduce the impact on the accumulator by adding a filter to your search criteria.
Incomplete Report
Results Depending on how you configure and run STRM reports, the results you generate
might appear to be different from what you expect. It is common to assume that a
report is not displaying all the data that you require.
Data accumulation for a search only starts when the search is added to a
scheduled report. Therefore, a report that is created on Wednesday, but is
scheduled to run weekly on a Monday, will not display a full week of data.
NOTE
The next time the report runs it will contain a full week of data.
Using the Network Activity or Log Activity tabs, run the search again and make
a comparison with the generated report.
If the results are different, see Resolving Missing Report Data.

STRM Troubleshooting Guide
16 STRM SYSTEM NOTIFICATIONS
Resolving Missing
Report Data STRM 2012.1 implements the resolutions for report data issues.
Procedure
1 If STRM detects that your data is incomplete, a notification message is displayed
on the Reports tab.
2 To ensure you capture all the report data, you have the option to run your report
against raw data during the initial time period. For more information on how to
configure this option, see the STRM Users Guide.
Limited Disk Space
to Perform Backup A system notification occurs if there is limited disk space on the destination file
system. STRM cannot complete a backup if there is insufficient disk space.
You might receive the following system notification:
Backup: Not enough free disk space to perform backup.
System notifications about limited disk space are displayed when the partition used
for the backup is at greater than 90% capacity. This can be caused by the volume
of data and your backup retention period settings. For more information, see the
STRM Administration Guide.
Verifying the Backup
Partition Disk Levels You can verify the disk levels of your STRM backup partition.
About this task
Disk usage warnings can occur on the Console or any Managed Host in your
STRM deployment. To check disk usage levels, review the monitored partitions on
your STRM Console or Managed Hosts.
Procedure
Step 1 Using SSH, log in to the STRM Console or Managed Host as the root user:
Username: root
Password: <password>
Step 2 Type the following command:
df -PTh /store/backup
Step 3 Review the backup partition to check the disk utilization levels.
If the backup partition is at greater than 90% capacity, see Resolving Backup
Partition Usage.
Resolving Backup
Partition Usage You can reduce your backup disk usage levels.
About this task
Configuring the retention bucket storage settings has a global impact on the
storage across your STRM deployment.

STRM Troubleshooting Guide
Limited Disk Space to Perform Backup 17
Procedure
1 Reduce disk utilization on the /store file system. Choose from the following options:
- Remove the oldest data from the /store/ariel/events/ file system. If you are
not familiar with Unix file systems or performing large scale data removal,
then contact Juniper Customer Support.
- Reduce your data retention period by adjusting the default retention bucket
storage settings. For more information, see the STRM Administration Guide.
- Identify which log sources that you can retain for shorter periods and use the
retention buckets feature to manage this. For more information, see the
STRM Administration Guide.
- Consider an offboard storage solution. For example, iSCSI or Fibre
Channel. For more information, see the STRM Offboard Storage Guide.
2 If your STRM backup partition is mounted on an NFS share, the retention period
for the backup can be too high. By default, the backup retention period is two days.
For more information on configuring backup retention periods, see the STRM
Administration Guide.