Pivotal GemFire 6.6.x - 8.2.x
All Supported OS
This document contains guidelines for collecting basic information, such as logs, statistics files, thread dumps, and heap dumps, for GemFire related issues. Besides providing the artifacts it is important to provide a timeline or overview of the issue with details on impact and actions taken.
Please note that in GemFire 8.2.x you can use the provided Log Collection Utility to make it easier to collect artifacts from a GemFire cluster as described here.
What to collect when
- For all GemFire issues, support will need logs and statistics from all cluster members covering the time period when the issue occurred.
- For issues where members are hung (any time an unscheduled "restart" becomes necessary), support will also need thread dumps from, at a minimum, the members that appear unresponsive. It is important than more than one thread dump is taken on each host.
- For tuning issues, support will also require GC logs.
- For out-of-memory issues and memory leaks, a heap dump will be required. (If this is not feasible, a heap histogram is better than nothing.)
- Locator, CacheServer, and GemFire Agent logs
Copy logs from the location configured by the
log-fileproperty in the
gemfire.propertiesfile or given as a parameter in your startup script. For example (from the
Make sure to provide complete logs, including those that cover the header and startup information. This information is valuable when investigating an issue.
- Java Client logs
Copy any client logs from the location defined in the Java client code or
gemfire.propertiesfile. For example (in code):
ClientCache cache = new ClientCacheFactory() .set("name", "CqClient") .set("cache-xml-file", "xml/CqClient.xml") .set("log-level", "fine") .set("log-file", "cqclient.log") .create();
- Native Client logs
Copy logs from the path defined in the
log-fileproperty of the
gfcpp.propertiesfile or defined in native code. For example (in
- Databrowser logs
Copy the logs from the path defined in the Databrowser "Options" setting menu:
Options-->Preferences-->Miscellaneous-->Log file directory
- GFMon logs
Copy the logs from the path defined in the GFMon "Preferences" setting dialog:
Preferences-->General-->Log file directory
- Security logs
Copy the logs from the path defined by the
security-log-fileproperty of the
gfsecurity.propertiesfile. For example (
- GFSH logs
By default, gfsh session logging is disabled. To enable gfsh logging, you must set the Java system property
-Dgfsh.log-level=<desired_log_level>where desired_log level is one of the following values: severe, warning, info, config, fine, finer, finest. For example, in Linux:
$ export JAVA_ARGS=-Dgfsh.log-level=info
Then, start gfsh.
Copy any logs from the directory in which the gfsh command was run. For example, if the gfsh command was run from
/home/user1/GemWorkdir1, the gfsh log would be in a file like:
- GC Logs
GC logging is enabled with startup parameters added to the JVM. The following parameters should be added when enabling GC logging for GemFire GC tuning issues:
-XX:+PrintGC (or the alias: -verbose:gc) -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:<file>
Use Xloggc to specify the path and file name of the GC log file. Default is standard out.
The overhead of GC logging is usually rather small so it is generally recommended to have it enabled. However, it is good to know that it is not needed to decide on this at JVM startup. The JVM has a category of flags called "manageable". For manageable flags, it is possible to change their values at run time. All the GC logging flags that start with "PrintGC" belong to the "manageable" category. Thus, it is possible to activate or deactivate GC logging for a running JVM.
Manageable flags can be set with the use of a JMX client calling the setVMOption operation of the HotSpotDiagnostic MXBean or using the jinfo tool shipped with the JDK.
2. Statistics files:
- Locator Statistics files, CacheServer Statistics files
Copy any stats files from the location defined by the
gemfire.properties. For example (
statistic-sampling-enabled=true statistic-archive-file=myStatisticsArchiveFile.gfs enable-time-statistics=false
statistic-sample-ratecan be changed from the default sample rate of 1000 milliseconds, but this shouldn't be needed as impact is very small.
Note that time statistics should only be enabled in dev and QA environments and not in production as this setting has a relatively large impact on GemFire performance.
To setup rolling of statistics files use the following parameters:
This will makes gfs files roll when they reach 100MB and keep the last 10 files.
- Java Client Statistics files
Copy any client-side stats files from the path defined in code or in the
gemfire.propertiesfile. For example (in code):
ClientCache cache = new ClientCacheFactory() .set("name", "CqClient") .set("cache-xml-file", "xml/CqClient.xml") .set("log-level", "fine") .set("log-file", "cqclient.log") .set("statistic-archive-file", "myClientStats.gfs" .set("statistic-sampling-enabled", "true") .create();
- Native Client Statistics files
Copy any stats files from the location defined in the
statistic-archive-fileproperty of the
gfcpp.propertiesfile or defined in native code. For example (
For more about logging information, you can also refer the following document: http://pubs.vmware.com/vfabric53/topic/com.vmware.vfabric.gemfire.7.0/managing/statistics/setting_up...
3. Thread dumps
For some issues, such as hung systems or performance issues, thread dumps from the server or client are essential to analyzing the issue. It is very important that multiple thread dumps are taken periodically (i.e. every few seconds) over a period of time.
Thread dumps can be taken using the following procedure:
- Step 1. Find out the relevant GemFire process id, i.e:
$ jps -l 7904 sun.tools.jps.Jps 5388 sample.JClient
- Step 2. Generate the thread dump(s).
On Solaris, Linux, and other Unix platforms, sending a SIGQUIT signal to the GemFire Java process will generate a thread dump(s), i.e.:
kill -QUIT <pid>
In Windows, you can press the CTRL-Break keys in the command shell where the GemFire Java process was started.
Alternatively, these tools can be used to generate the thread dump:
- jstack command: jstack <pid>
- Java VisualVM (jvisualvm)
4. Heap dump
For investigating issues, such as an Out-of-Memory issue or Memory Leak, a heap dump will help track down underlying issues.
Generate the heap dump of gemfire process using the following:
- Step 1. First, identify the specific gemfire process id using a command like
jps(as in the procedure for getting thread dumps).
- Step 2. Generate the heap dump.
- Using jmap command
[JDK_INSTALLATION]/bin/jmap -dump:live,format=b,file=heap.dump.out <pid>
- Using Java VisualVM (jvisualvm):
- run [JDK_INSTALLATION]/bin/jvisualvm,
- select the target process and select,
[Application] menu-->[Heap Dump], and select the generated heap dump, then choose
[Save As] "to local disk."
- Getting a Heap Dump Automatically on an "Out Of Memory" error:
Add the following jvm parameter to the java process before it starts.
For example (on Windows):
JAVA_OPTS=%JAVA_OPTS% "-XX:+HeapDumpOnOutOfMemoryError" "-XX:HeapDumpPath=C:\TEMP"
- Using jmap command
All Help & Support > Pivotal GemFire Knowledge Base > About Getting Support