Allocating Memory to Rhapsody

Memory recommendations are based on allocation of half of the available physical memory to Rhapsody, with the remaining memory available for Java native memory, OS memory and disk caching requirements. This configuration is recommended to minimize problems with garbage collection, and is especially important on virtual machine-based machines. If other applications are running on the same server as Rhapsody (not recommended), the memory usage of the other applications must be taken into account, for example:

• Medium standalone:
  • 8 GB server memory.
  • 4 GB allocation to Rhapsody.
• Medium with other application:
  • 12 GB server memory.
  • 4 GB recommended by other applications’ documentation.
  • 4 GB allocated to Rhapsody.

Rhapsody is subject to the Java memory management model. Consequently, allocating more memory to the heap does not necessarily improve performance. A large heap allocated to Rhapsody may result in noticeable performance degradation while the full garbage collection process operates. Memory tuning, therefore, aims to provide sufficient memory for the base configuration load as well as sufficient headroom to manage the message load efficiently, while minimizing both the frequency and length of the full garbage collection events.

Rhapsody is sensitive to memory performance. The host should be configured such that memory swapping or paging is not required, as this would impact the engine performance.

Rhapsody captures a range of statistical memory related data which is useful in managing the health of your Rhapsody engine. Refer to System Performance for details.

Java Memory Layout

In order to diagnose out-of-memory (OOM) errors or performance problems in a Java application, one must have a fundamental understanding of a Java process's memory structure. When an application is launched, one instance of a JVM (Java Virtual Machine) is instantiated on the system. So, typically each individual Java application is hosted in its own JVM (there are exceptions such as applications launched within an Application Server hosted on one JVM).

Upon application startup, the JVM allocates a Java Heap for use by managed Java objects in the application (the size is configurable using the -Xms and -Xmx settings). The Java Heap is allocated as one continuous block of memory in the process virtual address space. The maximum value (-Xmx) is reserved and the minimum value (-Xms) is also committed. In other words, they are backed by actual memory on the system as opposed to just reserved in the process address space. The Java Heap itself is allocated as a portion of the JVM Process Heap. Therefore, if there is an upper limit to the size of the JVM Process Heap, the Java Heap settings reduce the size of the available JVM Process Heap. This is not a concern on 64-bit systems. But, on 32-bit systems which use 32-bit pointers, this limits the range of virtual addresses available on the machine, thereby restricting the size of the JVM Process Heap and consequently the size of the Java Heap.

Java Heap

The JVM allocates a separate Java Heap for use by the application instead of the entire JVM Process Heap. There are a number of benefits to the JVM handling the memory management on behalf of the application:

• Reduced heap fragmentation (when coupled with JVM Garbage Collection (GC) compaction algorithms).
• As memory allocations or operations in the Java Heap are managed by the JVM, the JVM can throw Null Pointer Exceptions when invalid memory access occurs in the application. In contrast, this would result in a Segmentation Fault within a native application which would crash the application.
• Memory is committed up front as the heap expands according to the GC algorithms in place. As a result, allocation for individual Java objects from the already committed Java Heap is very fast.

Java Heap Structure

The Java Heap is divided into the Young Generation and the Old Generation. The Young Generation is further divided into an Eden Space and a Survivor Space. The Eden Space represents the portion of the heap where new objects are created. In a multi-threaded environment, this can lead to lock contention as objects are allocated by different threads within the Eden Space.

Objects are moved to the Old Generation memory space known as the Tenured Space if they are not garbage collected as a result of one or more minor GC cycles (they first transition to the Survivor Space). The significance of this from a debugging perspective is that a full GC cycle only occurs once the Tenured Space is exhausted and the Tenured Space is only exhausted if there are sufficient objects with long-lived references that have transitioned from the Young Generation to the Old Generation. Therefore, if the Java Heap is exhausted, there is either insufficient heap allocated for the application's needs or there are objects present with references that should not exist.

JVM Process Heap Structure

The JVM Process Heap is comprised of the following memory blocks:

• The Java Heap tuned with the -Xms and -Xmx options.
• Code Generation - used to convert and store bytecode to native code (this is typically a very small portion of the JVM Process Heap).
• Socket buffers - each socket connection will include a send and receive buffer (the size can be set in the application). Again, this is typically small but for an application with thousands of socket connections, it may become significant. For example, a buffer size of 64kb per connection for 1,000 connections leads to approximately 64Mb being used.
• Thread stacks - they can become significant for an application such as Rhapsody which uses many threads. Each thread has a separate memory area assigned for its stack (which represents the instruction stack and data of that thread's execution). This is configurable on a global basis using -Xss and on a per-thread basis as an option in the constructor (however, in practice this is not possible as the JVM is free to disregard this setting). Default stack sizes vary on a per-platform basis and are between 320kb and 1Mb.
• Direct Memory Space for memory mapped files and direct allocations of memory in non-Java Heap (in other words, through certain ByteBuffer operations).
• Java Native Interface (JNI) space including code as well as native requirements.
• Memory requirements for the Garbage Collector such as thread and GC information.

Sufficient physical memory should exist on the system to accommodate the entire JVM Process Heap, not just the Java Process Heap and the other applications on the system. Over-committing a system running Rhapsody by encroaching onto virtual memory is strongly discouraged because of the impact swapping has on full GC times. Refer to the interactive diagram for details.

Garbage Collection and Memory Tuning

The Java Virtual Machine Garbage Collector has configurable modes of operation, each of which can have a significant impact on both throughput and garbage collection times. The Java Virtual Machine can be tuned and configured by setting various options in the wrapper.conf file found under the Rhapsody installation directory in the bin directory.

To enable an option, add the following line to the wrapper.conf file:

wrapper.java.additional.<n>=<OPTION>

where:

• <n> follows sequentially after the last wrapper option in wrapper.conf.

• <OPTION> refers to one of the following garbage collection or memory tuning options.

Java Heap Size Tuning Options

• -Xms - initial Java Heap size (committed on startup).
• -Xmx - maximum heap size (reserved on startup).

MetaSpace Size Tuning Options

• -XX:MaxMetaspaceSize - maximum MetaSpace size.

Stack Size Tuning Option

• -Xss - stack size for each thread rounded according to Platform settings.

Tuning Options Usage Example

#********************************************************************
# Common VM Options (sequence range 1-49)
#********************************************************************
wrapper.java.additional.1=-server
wrapper.java.additional.2=-Djava.awt.headless=true
wrapper.java.additional.3=-XX:+DisableExplicitGC
# Separate config.ini location and bundle deployer working directories (into shared/config areas)
wrapper.java.additional.4=-Dosgi.configuration.cascaded=true
wrapper.java.additional.5=-Dosgi.sharedConfiguration.area=bin
wrapper.java.additional.6=-Dosgi.configuration.area=equinox
# https://bugs.eclipse.org/bugs/show_bug.cgi?id=243383 - default configuration directory is created at top level
wrapper.java.additional.7=-Dosgi.configuration.area.default=equinox
# Override default installation location so startup.jar doesn't have to live in root
wrapper.java.additional.8=-Dosgi.install.area=.
wrapper.java.additional.9=-Dosgi.classloader.lock=classname
wrapper.java.additional.10=-Dorg.eclipse.equinox.http.jetty.autostart=false
wrapper.java.additional.11=-Dosgi.classloader.singleThreadLoads=true
wrapper.java.additional.12=-verbose:gc

# Do not change this. Changes to this property should only be made using the environment variable
# stipulated
wrapper.java.additional.13=-Dwrapper.log.location="%WRAPPER_LOG_LOCATION%"
wrapper.java.additional.13.stripquotes=TRUE

Rhapsody Thread Usage

Rhapsody thread usage is highly dependent on the number of communication points and connections that have been configured. There is a base overhead of threads, for example 30 threads, and an additional overhead of thread per direction per connection for each communication point and a controller thread for the communication point. Therefore, a system with two bi-directional communication points, which support five connections each, have a total of 22 threads (2 x 2 x 5 + 2) plus approximately 30 other threads for various other components. As such, a system which makes use of a large number of communication points or connections has a correspondingly increased number of threads allocated, and this in turn results in a larger amount of memory required in the JVM Process Heap.