# Details of Metrics Collected by the JVM-Profiler 1. [CPU and Memory](#CPUAndMemory) 2. [IO](#IO) 3. [Process Info](#ProcessInfo) 4. [Stack Trace](#StackTrace) ## 1. CPU and Memory |**Name** |**Description** |**Sample Value** |**Note** | |--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------| |time |Time series value for grafana database. |1582504748538000000 | | |appId |Application id of the job that is running. |application_1576797426048_311346 |If using yarn, this is the yarn application id. | |bufferPools-direct-count |Total number of direct byte buffers used at the recorded time. |55 |Depending on your application, this can be a smaller number | |bufferPools-direct-memoryUsed |Total memory of the direct byte buffers used in bytes. |18950 |An estimate. This value should be very similar to the total capacity for a healthy JVM.| |bufferPools-direct-name |Name of the buffer pool. |direct |Direct buffer pools are always named direct. | |bufferPools-direct-totalCapacity |Available capacity in bytes in the direct buffer pool. |18949 |Capacity should be similar to memoryUsed for a healthy JVM. | |bufferPools-mapped-count |Total number of mapped byte buffers used at the recorded time. |0 |In general, spark applications will not require mapped buffer pools. | |bufferPools-mapped-memoryUsed |Total memory of the mapped byte buffers used in bytes. |0 |An estimate. This value should be very similar to the total capacity for a healthy JVM.| |bufferPools-mapped-name |Name of the buffer pool. |mapped |Mapped buffer pools are always named mapped. | |bufferPools-mapped-totalCapacity |Available capacity in bytes in the mapped buffer pool. |0 |Capacity should be similar to memoryUsed for a healthy JVM. | |epochMillis |System time in milliseconds when this value was recorded. |1582504781508 | | |gc-PSMarkSweep-collectionCount |Total number of collections that have occurred for PS MarkSweep. |7 |-1 if collection count is undefined for this collector. | |gc-PSMarkSweep-collectionTime |approximate accumulated collection elapsed time in milliseconds. |914 |-1 if collection count is undefined for this collector. | |gc-PSMarkSweep-name |Name of the collector. |PS MarkSweep |Constant value. | |gc-PSScavenge-collectionCount |Total number of collections that have occurred for PS Scavenge. |25 |-1 if collection count is undefined for this collector. | |gc-PSScavenge-collectionTime |approximate accumulated collection elapsed time in milliseconds. |1206 |-1 if collection count is undefined for this collector. | |gc-PSScavenge-name |Name of the collector. |PS Scavenge |Constant value. | |heapMemoryCommitted |Amount of memory in bytes guaranteed for the JVM to use. |375914496 | | |heapMemoryMax |Max amount of memory in bytes that can be used |11453595648 |Also refer to spark.memory.fraction for the fraction managed by spark | |heapMemoryTotalUsed |Total heap memory used in bytes |86386032 |Total memory used should be less than committed and max heap memory. | |host |Hostname of the executor |uber-bigdata-datanode2.uber.com | | |memoryPools-CodeCache-name |Name of the code cache. |Code Cache |Constant value. | |memoryPools-CodeCache-peakUsageCommitted |Peak committed memory of this pool since the JVM was started |2555904 | | |memoryPools-CodeCache-peakUsageMax |Peak value of usageMax of this pool since the JVM was started |251658240 | | |memoryPools-CodeCache-peakUsageUsed |Peak value of memory of this pool used since the JVM was started.|582656 | | |memoryPools-CodeCache-type |The type of this memory pool. |Non-heap memory |Code cache is a non-heap memory. So this value will always be constant. | |memoryPools-CodeCache-usageCommitted |Total memory in bytes committed for this pool. |2555904 | | |memoryPools-CodeCache-usageMax |The maximum amount of memory that can be used by this pool. |251658240 |-1 if undefined. Not guaranteed to be available if greater than the committed memory. | |memoryPools-CodeCache-usageUsed |Amount of memory used by this pool. |582656 |Having some code cache memory free ensures it is usable for JIT compilation. | |memoryPools-CompressedClassSpace-name |Name of the compressed class space. |Compressed Class Space |Is constant. This space is allocated in native memory for class metadata. | |memoryPools-CompressedClassSpace-peakUsageCommitted |Peak committed memory of this pool since JVM was started. |1572864 | | |memoryPools-CompressedClassSpace-peakUsageMax |Peak value of usageMax of this pool since the JVM was started. |1073741824 | | |memoryPools-CompressedClassSpace-peakUsageUsed |Peak value of usage of this pool since the JVM was started. |1389800 | | |memoryPools-CompressedClassSpace-type |Type of memory of this memory pool. |Non-heap memory |Compressed class space is native memory but is managed by the JVM | |memoryPools-CompressedClassSpace-usageCommitted |Total memory in bytes committed for this pool. |1572864 | | |memoryPools-CompressedClassSpace-usageMax |The maximum amount of memory that can be used by this pool |1073741824 |-1 if undefined. Not guaranteed to be available if greater than the committed memory. | |memoryPools-CompressedClassSpace-usageUsed |Amount of memory used by this pool. |1389800 | | |memoryPools-Metaspace-name |Name of the Metaspace. |Metaspace |Is constant. Metaspace is always called metaspace. | |memoryPools-Metaspace-peakUsageCommitted |Peak committed memory of this pool since JVM was started. |12320768 | | |memoryPools-Metaspace-peakUsageMax |Peak value of usageMax of this pool since the JVM was started. |-1 |-1 if undefined. | |memoryPools-Metaspace-peakUsageUsed |Peak value of usage of this pool since the JVM was started. |11768752 | | |memoryPools-Metaspace-type |Type of memory of this memory pool. |Non-heap memory |Metaspace is native memory (since JDK 8) but is managed by the JVM | |memoryPools-Metaspace-usageCommitted |Total memory in bytes committed for this pool. |12320768 | | |memoryPools-Metaspace-usageMax |The maximum amount of memory that can be used by this pool |-1 |-1 if undefined. Not guaranteed to be available if greater than the committed memory. | |memoryPools-Metaspace-usageUsed |Amount of memory used by this pool. |11768752 |The amount of space used by loaded classes. | |memoryPools-PSEdenSpace-name |Name of the young generation heap space. |PS Eden Space |Is constant. PS Eden space is always called PS Eden Space. | |memoryPools-PSEdenSpace-peakUsageCommitted |Peak committed memory of this pool since JVM was started. |98041856 | | |memoryPools-PSEdenSpace-peakUsageMax |Peak value of usageMax of this pool since the JVM was started. |4262461440 |-1 if undefined. | |memoryPools-PSEdenSpace-peakUsageUsed |Peak value of usage of this pool since the JVM was started. |92270136 | | |memoryPools-PSEdenSpace-type |Type of memory of this memory pool. |Heap memory |PS Eden space is a portion of the heap memory that is managed by the JVM | |memoryPools-PSEdenSpace-usageCommitted |Total memory in bytes committed for this pool. |98041856 | | |memoryPools-PSEdenSpace-usageMax |The maximum amount of memory that can be used by this pool |8589934592 |-1 if undefined. Not guaranteed to be available if greater than the committed memory. | |memoryPools-PSEdenSpace-usageUsed |Amount of memory used by this pool (and by newly loaded classes).|92270136 |A continuously increasing value indicates an unhealthy garbage collection. | |memoryPools-PSOldGen-name |Name of the old generation heap space. |PS Old Gen |Is constant. PS Old Gen space is always called PS Old Gen. | |memoryPools-PSOldGen-peakUsageCommitted |Peak committed memory of this pool since JVM was started. |261619712 | | |memoryPools-PSOldGen-peakUsageMax |Peak value of usageMax of this pool since the JVM was started. |8589934592 |-1 if undefined. | |memoryPools-PSOldGen-peakUsageUsed |Peak value of usage of this pool since the JVM was started. |0 | | |memoryPools-PSOldGen-type |Type of memory of this memory pool. |Heap memory |PS Old Gen space is a portion of the heap memory that is managed by the JVM | |memoryPools-PSOldGen-usageCommitted |Total memory in bytes committed for this pool. |261619712 | | |memoryPools-PSOldGen-usageMax |The maximum amount of memory that can be used by this pool |8589934592 |-1 if undefined. Not guaranteed to be available if greater than the committed memory. | |memoryPools-PSOldGen-usageUsed |Amount of memory used by this pool (by objects beyond threshold).|0 |A continuously increasing value indicates an unhealthy garbage collection. | |memoryPools-PSSurvivorSpace-name |Name of the survivor heap space. |PS Survivor Space |Is constant. PS Survivor Space is always called PS Survivor Space. | |memoryPools-PSSurvivorSpace-peakUsageCommitted |Peak committed memory of this pool since JVM was started. |16252928 | | |memoryPools-PSSurvivorSpace-peakUsageMax |Peak value of usageMax of this pool since the JVM was started. |16252928 |-1 if undefined. | |memoryPools-PSSurvivorSpace-peakUsageUsed |Peak value of usage of this pool since the JVM was started. |0 | | |memoryPools-PSSurvivorSpace-type |Type of memory of this memory pool. |Heap memory |PS Survivor Space is a portion of the heap memory that is managed by the JVM | |memoryPools-PSSurvivorSpace-usageCommitted |Total memory in bytes committed for this pool. |16252928 | | |memoryPools-PSSurvivorSpace-usageMax |The maximum amount of memory that can be used by this pool |16252928 |-1 if undefined. Not guaranteed to be available if greater than the committed memory. | |memoryPools-PSSurvivorSpace-usageUsed |Amount of memory used by this pool (and by newly loaded classes).|0 |A continuously increasing value indicates an unhealthy garbage collection. | |name |Process name (number) |12345@uber-bigdata-datanode2.uber.com |Also contains host name. | |nonHeapMemoryCommitted |Amount of non heap memory in bytes committed for the JVM to use. |16187392 |Total of committed of all non-heap memory pools. | |nonHeapMemoryMax |Maximum amount of memory in bytes that can be used. |-1 |-1 if undefined. | |nonHeapMemoryTotalUsed |Amount of non heap memory (metaspace, code cache) used. |13510368 |Total of usage of all non-heap memory pools. | |processCpuLoad |The recent cpu usage for the JVM process. |4.335671366438364e-9 | | |processCpuTime |CPU time used by JVM process in nanoseconds. |2340000000 | | |processUuid |UUID of the JVM process. |48abf05b-dfd4-4069-86c6-8cdbc195bd40 | | |processUuid_1 |UUID of the JVM process. |48abf05b-dfd4-4069-86c6-8cdbc195bd40 |Appears to be a duplicate metric. | |role |Role of this spark executor. |executor |Can be either executor or driver. | |systemCpuLoad |The recent cpu usage for the whole system. |0.03703887412577616 | | |vmHWM |Peak resident set size ("High Water Mark") in bytes for the JVM. |142876672 |An indication of actual physical memory (RAM) usage in linux by the JVM. | |vmPeak |Peak of the virtual memory usage in bytes for the JVM. |15130554368 | | |vmRSS |JVM physical memory usage (resident set size) in bytes. |142876672 | | |vmSize |Virtual memory size for the JVM. |15130550272 |This memory includes swap space, so a large value can imply high swap memory usage. | [Back to Top](#Title) ## 2. IO |**Name** |**Description** |**Sample Value** |**Notes** | |--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------| |time |Time series value for grafana database. |1582504713009000000 | | |appId |Application id for the job that is running. |application_1576797426048_311346 |If using yarn, this is the yarn application id. | |epochMillis |System time in milliseconds when this value was recorded. |1582504713001 | | |host |Hostname of the executor. |uber-bigdata-datanode2.uber.com | | |name |Process name (number). |12345@uber-bigdata-datanode2.uber.com |Includes the hostname as well. | |processUuid |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 | | |processUuid_1 |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate metric. | |role |Role of this spark executor. |executor |Can be either executor or driver. | |self-io-rchar-1 |The number of characters read by this JVM process. |8192340 |The "-1" is coming from the formatter for influx which was used when writting this. | |self-io-read_bytes-1 |Attempt to count the bytes read by this JVM process from storage.|1892352 | | |self-io-wchar-1 |The number of characters written to disk by this JVM process. |51016 | | |self-io-write_bytes-1 |Attempt to count the bytes written by JVM process to storage. |86016 | | |stat-cpu-N |CPU entry found in /proc//stat. |cpu, cpu0, cpu1, cpu2 ... |N is the CPU number starting at cpu-1 (i.e. cpu) which aggregates all other CPU lines. | |stat-idle-N |Time the N-th CPU spent in the idle task. |53679018 |Time is in USER_HZ (1/100th of a second). | |stat-iowait-N |Time spent by N-th CPU waiting for I/O to complete. |137645010 | | |stat-nice-N |Time spent by N-th CPU in user mode with low priority (nice). |104 |Low idle time with high nice time are indications of busy background tasks running. | |stat-system-N |Time spent by N-th CPU in system mode. |32700364 |Time spent at kernel level. | |stat-user-N |Time spent by N-th CPU at the user (application) level |279816973 | | [Back to Top](#Title) ## 3. Process Info |**Name** |**Description** |**Sample Value** |**Notes** | |--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------| |time |Time series value for grafana database. |1582504713009000000 | | |agentVersion |A constant value defined in code. |1.0.0 |Agent version will always be 1.0.0. | |appId |Application id for the job that is running. |application_1576797426048_311346 |If using yarn, this is the yarn application id. | |cmdline |Contains JVM command executed and input arguments and classpath. |/usr/jdk64/jdk1.8.0_112/bin/java -javaagent...| | |epochMillis |System time in milliseconds when this value was recorded. |1582504713001 | | |host |Hostname of the executor. |uber-bigdata-datanode2.uber.com | | |jvmClassPath |Contains JVM classpath entries. |--user-class-path .../jvm-profiler-1.0.0.jar |This entry is emtpy if cmdLine is not empty. This is to avoid duplicate information. | |jvmInputArguments |Contains JVM input arguments. |metricInterval=5000,sampleInterval=5000 ... |This entry is emtpy if cmdLine is not empty. This is to avoid duplicate information. | |name |Process name (number). |12345@uber-bigdata-datanode2.uber.com |Includes the hostname as well. | |processUuid |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 | | |processUuid_1 |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate metric. | |role |Role of this spark executor. |executor |Can be either executor or driver. | |xmxBytes |Maximum JVM heap size in bytes. |12884901888 | | [Back to Top](#Title) ## 4. Stack Trace |**Name** |**Description** |**Sample Value** |**Notes** | |--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------| |time |Time series value for grafana database. |1582504713009000000 | | |appId |Application id for the job that is running. |application_1576797426048_311346 |If using yarn, this is the yarn application id. | |count |Number of elements appended to the metrics buffer. |1 |This value is usually 1, but not guaranteed to be constant. It is thread-safe. | |endEpoch |System time in milliseconds after resetting the metrics buffer. |1582504712723 |The metrics buffer is an internal buffer used by this profiling application. | |host |Hostname of the executor. |uber-bigdata-datanode2.uber.com | | |name |Process name (number). |12345@uber-bigdata-datanode2.uber.com |Includes the hostname as well. | |processUuid |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 | | |processUuid_1 |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate metric. | |role |Role of this spark executor. |executor |Can be either executor or driver. | |stacktrace |Stacktrace starting from bottom of the stacktrace. |java.lang.Object.wait,java.lang.ref ... |Trims the top method (most nested method) if the size is too large (800k characters) | |startEpoch |System time in milliseconds before resetting the metrics buffer. |1582504710523 |The metrics buffer is an internal buffer used by this profiling application. | |threadName |Name of the thread for which stacktrace is displayed. |Signal Dispatcher | | |threadState |State of the thread when this stacktrace was captured. |NEW, RUNNABLE, BLOCKED, WAITING ... | | [Back to Top](#Title) ## 5. Thread Info |**Name** |**Description** |**Sample Value** |**Notes** | |--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------| |time |Time series value for grafana database. |1582504713009000000 | | |epochMillis |System time in milliseconds when this value was recorded. |1582504781508 | | |host |Hostname of the executor. |uber-bigdata-datanode2.uber.com | | |name |Process name (number). |12345@uber-bigdata-datanode2.uber.com |Includes the hostname as well. | |appId |Application id for the job that is running. |application_1576797426048_311346 |If using yarn, this is the yarn application id. | |processUuid |UUID of the JVM process. |89a256c7-96c3-4e48-b5ba-f55e886ba743 | | |role |Role of this spark executor. |executor |Can be either executor or driver. | |totalStartedThreadCount |Total Number of Thread created so far since JVm Launch. |10 | | |liveThreadCount |Number of thread which are currently active. |8 | | |newThreadCount |Number of thread which are newly Created. |3 | Derived from last Cycle Total Threads. | |peakThreadCount |the peak live thread count since the JVM start |8 | | [Back to Top](#Title)