VMware VCM 5.3 User manual

VMware vCenter Configuration
ManagerTroubleshooting Guide
VCM 5.3
This document supports the version of each product listed and supports all
subsequent versions until the document is replaced by a new edition. To
check for more recent editions of this document, see
http://www.vmware.com/support/pubs.
EN-000476-00

Copyright You can find the most up-to-date technical documentation on the VMware Web site at:
http://www.vmware.com/support/
The VMware Web site also provides the latest product updates.
If you have comments about this documentation, submit your feedback to:
[e-mail]
© 2006-2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and
intellectual property laws. VMware products are covered by one or more patents listed at
http://www.vmware.com/go/patents.
VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All
other marks and names mentioned herein may be trademarks of their respective companies.
VMware, Inc.
3401 Hillview Ave.
Palo Alto, CA 94304
www.vmware.com
VCM Troubleshooting Guide
2 VMware, Inc.

Contents
About This Book 5
General Troubleshooting Guidelines 7
Verify the Behavior is Negative 7
Isolate the Behavior 8
Identify External Factors 9
Check the Debug Log 10
Info Messages 11
Warning Messages 11
Error Messages 12
Exception Messages 12
Gather Information for VMware Customer Support 12
Types of Problems 13
User Interface 13
Security / Authentication 14
SQL Server 14
VCM Agent 15
UNIX Agent 15
Report Server 15
Internet Information Services (IIS) 16
Network Connectivity 17
Hardware and Performance Issues 17
How to Gather Diagnostic Files 19
Screenshots 19
Debug Logs 19
SQL Server Logs 20
IIS Logs 21
ARS Files 21
Syslog File (UNIX) 22
IE Tool Logs 22
Event Logs 22
System Information (msinfo32.exe) 23
ETL Logs (UNIX) 23
VCM Installation Logs 23
Patching Debug Information 23
Software Provisioning Troubleshooting 25
Troubleshooting Software Provisioning Repositories 25
When requesting a package from a repository, the package (.crate file) cannot be found 26
Repository.index is no longer valid 27
Crates.gz file is corrupt 27
Too many users adding new platforms and sections 27
Repository not found when reindexing or publishing packages to a repository 28
Troubleshooting Package Studio 28
Error when starting Package Studio: “Cannot create instance of ‘RepositoryEditorViewModel’
defined in assembly ‘PackageStudio…” 28
The Generate button is grayed out 29
VMware, Inc. 3

VCM Troubleshooting Guide
Error when adding a Provides: the dependency package name is invalid 30
When a package is created, signed, saved as project, and then generated, the package is no
longer signed 31
The installed size of the application is set to nnn, but systems with nnn run out of space during
installation 31
The package installs the application but does not uninstall 31
Files are added to the Project Data Directory, but they are not displayed in the list 31
I am uncertain if the package requires a reboot to install or I only want the installation to
reboot sometimes 32
Gathering Software Provisioning Logs on the Collector 32
Gathering Software Provisioning Logs on the Agent 32
Windows Agent Installation 35
Pre-Install Environment 35
Network 35
Collector 35
Manual Agent Install 44
Protocol Specific 45
UNIX Agent Troubleshooting 47
Agent Directory Structure 47
Collector Certificates 56
Patch Assessment 57
Directories Created During an Inspection 58
Saving Executed Scripts and Results 59
UNIX Agent Error Scenarios 60
If the installation reports and error 60
The Collector cannot ping the Agent 60
Agent fails to return data 61
Monitoring network traffic 63
The Collector reports the job succeeded, but there is still no data 66
Index 67
4 VMware, Inc.

About This Book
About This Book This manual, Troubleshooting Guide for VMware vCenter Configuration Manager, explains the types of
problems that may occur with VMware vCenter Configuration Manager, the diagnostic material VMware
Customer Supportwill need to analyze the problem, and the steps to retrieve that information.
Intended Audience
The information presented in this manual is written for system administrators who are experienced
Windows or UNIX/Linux/Mac OS X system administrators and who are familiar with managing network
users and resources, and performing system maintenance.
To use the information in this guide effectively, you must have a basic understanding of how to configure
network resources, install software, and administer operating systems. You also need to fully understand
your network’s topology and resource naming conventions.
Document Feedback
VMware welcomes your suggestions for improving our documentation. If you have comments, send
your feedback to [e-mail].
VMware VCM Documentation
The vCenter Configuration Manager (VCM) documentation consists of the VCM documentation set.
Technical Support and Education Resources
The following technical support resources are available to you. To access the current version of this book
and other books, go to http://www.vmware.com/support/pubs.
Online and Telephone
Support
To use online support to submit technical support requests, view your product
and contract information, and register your products, go to
http://www.vmware.com/support.
Customers with appropriate support contracts should use telephone support for
priority 1 issues. Go to http://www.vmware.com/support/phone_support.html.
Support Offerings To find out how VMware support offerings can help meet your business needs,
go to http://www.vmware.com/support/services.
VMware, Inc. 5

VMware Professional
Services
VMware Education Services courses offer extensive hands-on labs, case study
examples, and course materials designed to be used as on-the-job reference tools.
Courses are available onsite, in the classroom, and live online. For onsite pilot
programs and implementation best practices, VMware Consulting Services
provides offerings to help you assess, plan, build, and manage your virtual
environment. To access information about education classes, certification
programs, and consulting services, go to http://www.vmware.com/services.
VCM Troubleshooting Guide
6 VMware, Inc.

General Troubleshooting Guidelines 1
General Troubleshooting Guidelines This document attempts to describe some of the basic steps you can take before contacting VMware
Customer Support.Below is a simple flowchart that provides troubleshooting procedures, allowing you to
resolve simple issues on your own. Then, if necessary, you can contact VMware Customer Support for
additional support.
Verify the Behavior is Negative
There are a few messages that appear in VCM and VCM Patching that may be interpreted as errors, when
in fact they are merely warnings. Other behaviors may be operating exactly as defined, but since the result
is not what is expected, it could be interpreted as wrong behavior.
For example, when creating a Compliance rule, it is important to realize that the conditional statements
must be set up to highlight systems that fail to meet the described normal condition as non-compliant.
The VCM User Interface highlights this fact in the Compliance rule creation wizard on the condition
definition page displayed below:
VMware, Inc. 7

This construction may be confusing. A user might intuitively create a conditional statement that describes
the abnormal condition they want to eliminate. As a result, they find that VCM returns a list of nearly all
their machines and flags them as non-compliant with the abnormal rule. In reality, nearly all these systems
might be compliant and VCM has merely performed exactly as requested.
The first step to troubleshooting VCM, therefore, is to first verify that the suspect behavior is a negative
behavior or one that is not within the normal operating parameters of VCM. The VCM Help file is the
perfect tool to use when trying to determine normal behavior. There is a Help button on almost every
screen of each VCM wizard as well as in the upper-right corner of the Portal. The Help buttons, on the
Portal or in a wizard, display context-sensitive help topics relevant to the area of VCM in which you are
currently working.
Validate that the message or behavior seen is not described in the online Help before assuming the
behavior is negative or undesirable.
Isolate the Behavior
The next step to take when troubleshooting an undesired behavior is to try to isolate it. This is done by
eliminating, one by one, all factors that may be contributing to the behavior until the bare minimum
number of factors are in play when the error occurs.
For example, if you encounter some problem on a single machine during a collection for 50 machines and
10 data types per machine, the first step would be to eliminate the 49 machines that did not exhibit the
same behavior, and run a collection for the same 10 data types against the single problem machine. You
can usually isolate which machine failed by viewing the details of the job in the Job History screen on the
Administration slider. If you witness the same behavior, you then eliminate data types (one or two at a
time) until you find the data type that is causing the behavior. Again, the job details on the Job History
screen (as seen below) may give you some indication as to which data types failed.
VCM Troubleshooting Guide
8 VMware, Inc.

The goal for this trial-and-error elimination process is to understand when the undesired behavior occurs
so you can reproduce the behavior consistently and at will. Once you know exactly how to reproduce the
error at any time, be sure to document those steps for future investigation.
Identify External Factors
Sometimes the behavior is seemingly unpredictable and the above method will not succeed in isolating it.
For these types of issues, it is best to take special note of the status of the environment when the behavior
appears. Due to the amount of data processed by VCM, the systems running the database and the main
application are subject to periods of high resource consumption.
System load can be a good troubleshooting indicator when problems arise. When investigating an
undesired behavior, take note of the system load, memory usage, network traffic, time of day, other
running applications, or anything else that may indicate a correlation with the behavior on both the
Collector machine and the Agent machine. For Windows machines, Task Manager should provide you
with this information (as seen in the image below).
General Troubleshooting Guidelines
VMware, Inc. 9

For example, if collections are stalling at a specific step in the process, you may occasionally find that a
single CPU is running at 100% capacity. This may indicate any number of problems, but knowing that the
CPU running at 100% always correlates with the behavior you are investigating is the most important
thing at this stage. With this approach, you may be able to isolate a specific time of day, or a set of
conditions that must be present for the undesired behavior to occur.
Another good method for identifying external factors is reviewing Event Log. Many simple problems may
be detected by reviewing the Security and Application logs. The most common errors you will see in the
Event Log are authentication problems (as seen below), but any error or warning message occurring in
the Event Viewer during the time frame surrounding the undesired behavior is suspect, especially if the
same error message always accompanies the undesired behavior.
Errors and behaviors that involve hardware problems or failures may be extremely erratic. You may see
a series of seemingly unrelated errors occur in sequence, or you may see the same error at random times.
You may also see that the behavior of VCM appears to degrade over time. These sorts of behaviors may
indicate a need for system hardware diagnostics.
Check the Debug Log
VCM produces several files in various locations through its normal operation with the extension “.dbe”.
The .dbe files contain information about VCM operations. The Collector also writes debug information to
the database whenever jobs are running in the Portal.
The Collector log is the most important log file as it records the primary functionality of VCM, and when
VMware Customer Support technicians speak of “the debug log” in the singular, they are typically
referring to this Collector debug log. Debug logs are a common troubleshooting tool for VMware
Customer Support technicians, so you may already be very familiar with the process of gathering them.
You may even recognize several of the messages mentioned in the logs, which may or may not be helpful.
VCM Troubleshooting Guide
10 VMware, Inc.

It is important to remember that the debug logs produced by VCM were never designed for the average
user. The messages contained in the logs have been specifically created to help developers understand
why VCM is operating the way it is. Always review the debug logs yourself and when necessary, forward
the log in its entirety on to VMware Customer Support for analysis. A sample debug log is provided
below.
It is not necessary for any support technician to be familiar with every message produced in a debug log,
but it is extremely helpful to know what the different types of errors logged in the debug log can mean
when determining the cause for a specific behavior. There are four major categories of error messages:
nInfo messages
nWarning messages
nError messages
nException messages
Info Messages
Info messages are the most common message category that you will find if VCM is configured to save
them. By default, Info messages are discarded by VCM when writing to the debug logs because the
volume of information produced by VCM under normal circumstances will make these log files grow
substantially. If, however, Info messages are enabled under the Administration slider, nearly every action
VCM performs will be logged in the various debug logs. Info messages are critical when diagnosing an
undesired behavior because it gives the developers a context surrounding the behavior.
Warning Messages
Warning messages are the next most common message category you will find in debug logs. Warnings
indicate a point in the process where an unexpected situation arose, but do not necessarily identify a
problem with VCM, as many common situations such as network timeouts or authentication problems
will produce a warning. Also, warning messages should not stop VCM from executing normal processes,
but they are intended to alert the developers of a possible problem.
General Troubleshooting Guidelines
VMware, Inc. 11

Error Messages
Error messages indicate a more serious problem. VCM processes will not halt because of an error, but
they may not be able to continue normally. For example, if the collection of a specific data type produces a
value that is not of the same type the database is expecting, VCM may have to discard the collected
information for that entire data type. This would produce an error message to highlight the fact that
something did not work properly during the latest process. In this case, VCM was able to recover and
continue, but perhaps not with the desired result.
Exception Messages
Exceptions messages indicate a fatal problem that VCM cannot handle. These types of problems will likely
cause a complete failure of the current process. One common exception occurs with overseas customers.
When a collector machine is set up to use a regional date/time notation other than English (United States),
which uses a date/time notation of mm/dd/yyyy to represent the date, collections to UNIX machines will
usually fail. Many European date/time notations use dd/mm/yyyy to represent the date. The message
reported in the debug log is an exception with the message “Arithmetic overflow converting expression to
a data type datetime.” In essence, this message states that VCM could not convert the UNIX date/time
notation to the date/time notation used by the SQL database. With this exception occurring, complete
UNIX collections will fail.
Gather Information for VMware Customer Support
When reporting an issue to VMware Customer Support, you will save a great deal of time if you are
prepared with as much information about the problem as possible. Use the “Types of Problems” section of
this document to determine which diagnostic files VMware Customer Support will need in order to
analyze the behavior.
VCM Troubleshooting Guide
12 VMware, Inc.

Types of Problems 2
Types of Problems Most VCM problems will fall into one of several categories that may make them easier to identify. Each
category is representative of one of the areas critical for proper VCM functionality. When troubleshooting
a behavior, keep these categories in mind as they may add to your understanding of how VCM operates
as a whole.
User Interface
The User Interface (UI) is the main way users interact with VCM. The UI communicates primarily with the
VCM database to retrieve and submit information. While many unwanted behaviors may reveal
themselves in the UI, the UI is rarely the root cause. The most common problems directly associated with
the UI are display anomalies. Duplicate columns in the Data Grid, inappropriately disabled text fields, and
data formatting problems are all examples of UI-related problems. Screenshots typically provide the most
helpful information when relating these problems to VMware Customer Support. Below is an example of
a UI problem where the headers of a report are compressed into only a two or three character width.
VMware, Inc. 13

Security / Authentication
One of VCM's primary roles is multi-system administration. Therefore, VCM must be granted authority to
act as an administrator on all the machines under its control. Similarly, the number of places where that
authority may be removed or truncated is equally large. Common authentication problems may include
changing passwords, dropped administration access, and implementation of additional security measures
(such as a proxy server) without updating VCM. System access is only half of the access required by VCM.
VCM also requires access to the SQL Server database, with the authority to insert, modify, and delete data
within it. Access problems of both sorts may reveal themselves in the UI, in the Event Viewer, as well as in
the debug log. Screenshots of UI errors combined with debug logs are the best information to send
VMware Customer Support when reporting problems. Below is an example of an authentication error
found in the Event Viewer.
SQL Server
Nearly all data associated with VCM is stored in the SQL database. The VCM database contains the
collected system data from the VCM Agents.The VCM_Coll database contains information about the UI,
such as Collector settings and options. The VCM_Raw database is a performance-enhancing database used
to temporarily store information from collections before bulk insertion into the VCM database. Finally,
the VCM_UNIX database contains collected information from any UNIX Agents in the customer’s
environment. SQL Server errors may include resource problems, disk space problems, or authentication
problems, among others. Improper tuning of a SQL Server also may cause performance problems such as
data bottlenecks. Evidence of SQL Server errors may appear in the UI or in the debug log, but messages
directly relating to the SQL Server should appear in the SQL Server logs. If you suspect a SQL Server error,
the SQL Server logs, the system and application Event Logs, and the VCM debug log are the most helpful
sources of information for VMware Customer Support. Below is one example of an error you may see
associated with SQL Server.
VCM Troubleshooting Guide
14 VMware, Inc.

VCM Agent
The VCM Agents are the mechanism by which VCM collects information from managed machines. If the
Agent is not functioning properly, collections from that machine may fail. The Running Jobs UI is the first
component to identify problems with an Agent. A typical error message for an Agent that is unable to
start collecting might be “There was a problem parsing the agent instructions document.” This message
indicates there was some conflict in the instruction set sent to the Agent that it could not process, and failed
as a result. When troubleshooting Agent errors, a debug log and any available Agent ARS files is the best
information to send to VMware Customer Support when requesting assistance. Also, check the Event Log
for any application or system messages that may indicate why the Agent failed.
UNIX Agent
UNIX operates very differently from Windows. To accommodate the differences, VCM uses a separate
Agent for UNIX/Linux-based platforms. Troubleshooting UNIX Agent problems can be a little bit easier
than the standard Windows Agent because many operations available for Windows Agents are not yet
available on the UNIX side. Compliance enforcement is not yet available for the UNIX Agent, and
therefore should be eliminated from your troubleshooting process. For UNIX Agent errors a debug log,
UNIX Agent debug log, UNIX Agent ZRP files, and a system log are all helpful information to VMware
Customer Support technicians when troubleshooting UNIX Agent issues. Below is a typical UNIX error
shown in the debug log.
Report Server
The Report Server is responsible for the graphical displays of information in VCM and for the VCM
scheduled reports. Report error will normally appear in the UI.
Types of Problems
VMware, Inc. 15

You can easily determine if there is a problem with the Report Server specifically by navigating to the
Report Server homepage to see if the native interface for the Report Server is operational. This address is
usually just the name of the Report Server machine followed by “/Reports”. See the following example:
http://localhost/Reports
If the same type of problem you discovered in the UI also occurs on this Reports, then the problem is
almost certainly with the Report Server itself, and not with VCM. If the Report Server interface appears
without error, the cause is likely in VCM. For these errors, screenshots, any support-requested SQL
profiler trace files, and a debug log are the files the VMware Customer Support technicians should see.
Internet Information Services (IIS)
The VCM UI is displayed using Microsoft’s Internet Information Server (IIS). Occasionally you may see
error messages in the UI that you typically see when failing to connect to a webpage. Error messages like
“404 File Not Found” or “403 Forbidden” are typical IIS messages, and relate to the improper functioning
of the IIS configuration in relation to VCM or Report Server pages. You may also see ASP.NET errors
when IIS is involved as seen below.
When these errors occur, please send screenshots and any relevant entries in the IIS logs when requesting
assistance from VMware Customer Support.
VCM Troubleshooting Guide
16 VMware, Inc.

Network Connectivity
VCM relies heavily on network connectivity to all Agents. In systems where you have a split installation,
failed network connectivity can also cause problems with VCM. Failures in network connectivity are not
likely to occur across all systems involved with the operation of VCM at the same time, so issues with the
network are occasionally easier to diagnose. Issues with the network are usually first discovered in the UI
with the failure of some VCM or VCM Patching job to a single or subset of machines. A common error
message associated with connectivity problems is “Ping Failed” as seen below.
The simplest way to test network connectivity is to ping the Agent from the Collector. If the environment
contains a firewall, if the Agent is a UNIX agent, or if the Agent machine is using the HTTP protocol
instead of the DCOM protocol for its primary communication method, it may be necessary to connect to
the HTTP port of the Agent on the target machine from the Collector. This can be done by using the telnet
command in the following way:
C:\> telnet target_machine 26542
In the above example, the name of the machine we are testing is “target_machine” and the HTTP port
used is 26542. A successful test of this type should return nothing but a blank screen. This is the target
system answering the connection request and awaiting further instructions. You can break this connection
by typing “Ctrl-]” and then typing “quit” to exit the telnet program. If you receive any other message,
such as “Connection refused” or “Connection timeout”, that is a good indication of network
communication problems. These issues will need to be worked out with the local network support
individuals before VCM troubleshooting may continue.
Hardware and Performance Issues
Hardware and performance issues are some of the most difficult issues to diagnose because they are
usually sporadic and random. In fact, there may be times where the only constant you can count on is
random errors. This is the time to look at possible hardware and/or performance problems. Running out
of disk space is the most common hardware problem. Simply check the file manager to verify that
enough disk space is present for both the database and for VCM itself. Next, make sure that there is
enough memory and enough CPU cycles to ensure that the VCM services and processes can function
properly. If VCM is competing with several other processes on the system that are unrelated to VCM,
performance may degrade to the point that errors and exceptions are produced in the debug log. Finally,
it may be necessary to run diagnostics on the hardware components, such as the memory chips,
processor(s) and system boards. Refer to VCM documentation to see the recommended hardware
configurations needed to run VCM properly in the different possible environments.
Types of Problems
VMware, Inc. 17

VCM Troubleshooting Guide
18 VMware, Inc.

How to Gather Diagnostic Files 3
How to Gather Diagnostic Files You may need to review or gather one or more of the following files to use for diagnostic purposes. The
following sections provide the steps needed to locate the files.
Screenshots
Screenshots are one of the easiest ways to demonstrate the exact behavior you see on your system. A
single screenshot can show an error message, and a series of screenshots can show a change in behavior
over time. A screenshot series is also good for verifying the values you selected in a wizard screen-by-
screen.
To capture a full desktop screenshot:
1. Click the Print Screen button (PrtScr) on the keyboard.
2. Open an new email message or a Word document.
3. Press Ctrl-v to paste the image into the message or document.
To capture a screenshot of a single window:
1. Make certain the window you want to capture is active.
2. Hold down the Alt key and press the Print Screen key (PrtScr).
3. Open a new email message or Word document.
4. Press Ctrl-v to paste the image into the message or document.
Debug Logs
Debug logs are the primary way for our developers to trace the behavior of VCM. The most important
debug log is the Collector debug log, and must be extracted from the database using the
EcmDebugEventViewer.exe file located in <drive>:\Program Files\VMware\VCM\Tools. The following
procedure is used to extract the Collector debug log complete with INFO messages:
1. Start the VCM portal.
2. Select Administration | Settings | General Settings | Collector. The Collector Settings data grid
appears.
3. Select the Type of information that should be logged, and then click Edit Settings. The Edit Settings
page appears.
4. Select all the options: Exception, Error, Warning, and Info.
5. Complete the other pages in the wizard, and then click Finish.
VMware, Inc. 19

6. Select Administration | Settings | General Settings | Database. The Database data grid appears.
7.
8. Select the Type of information that should be logged, and then click Edit Settings. The Edit Settings
page appears.
9. Select all the options: Exception,Error,Warning, and Info.
10. Complete the other pages in the wizard, and then click Finish.
11. Select Administration | Settings | Windows | Agent - General. The Agent General Settings data
grid appears.
12. Select the Type of information that should be logged, and then click Edit Settings. The Edit Settings
page appears.
13. Select all the options: Exception,Error,Warning, and Info.
14. Complete the other pages in the wizard, and then click Finish.
15. Select Administration | Settings | UNIX | Agent - General. The Agent General Settings data grid
appears.
16. Select the Type of information that should be logged, and then click Edit Settings. The Edit Settings
page appears.
17. Select all the options: Exception,Error,Warning, and Info.
18. Complete the other pages in the wizard, and then click Finish.
19. Restart the VCM Collector service in the Services manager.
20. Now run the job in question again. You must wait 5 minutes after it completes before proceeding.
21. Minimize the console and navigate to <Drive>:\Program Files\VMware\VCM\Tools. In this
directory, double-click ECMDebugEventViewer.exe.
22. In ECMDebugEventViewer, click Filter Settings. The Filter Settings dialog box appears.
23. Select all the check boxes in the Message Type and Message Source areas. Click OK.
24. In the Data Source area, type the names of the servers and databases.
25. Click OK to close the dialog box.
26. Click Date/Time. The Data/Time dialog box appears.
27. Select the between option, specify the dates and add five or more minutes to the beginning and end
times during which the collection ran.
28. Click File, and then select Fetch.The displayed data is refreshed.
29. Click File, and then select Fetch Next. Continue the Fetch Next process until no additional data is
added to the displayed debug log.
30. Click File and select Save as Dbe. Save the file as a DBE file and note where it was saved for later use.
31. Repeat steps 1-4 and reverse the settings for the Collector, Database, and Windows Agent and UNIX
Agent objects back to their original logging levels (usually Exception, Error, and Warning).
SQL Server Logs
The SQL Server logs keep track of events related to the database operation. Use the following process to
extract them from SQLServer:
VCM Troubleshooting Guide
20 VMware, Inc.