SAP NetWeaver Java on IBM i

From the hardware to the operating system to the application and workload many factors will influence how a JVM is tuned. Due to the different environments that an SAP NetWeaver system might operate in and the different workloads these systems might run, it is difficult to predict how to optimally configure a JVM without any workload analysis. Through customer experience, benchmarks, and other tests we have learned what levers to pull and knobs to turn in order to tune the JVM. However, the actual settings that are optimal for each environment should be determined by analyzing and making adjustments specifically for that particular environment and workload. Here are some guidelines when tuning a JVM for SAP NetWeaver Java applications.

1. Have reasonable expectations
   Performance is a relative term. Most people tend to assess performance based on some sort of comparison. Whether the comparison is valid or invalid it doesn’t matter. It is important to be able to set reasonable expectations for performance and possible improvements based on the current health of your system, resources available, and the workload.
2. Establish performance goals and stick with them
   It is important that you consider and decide what your performance goals are. For example, are you tuning for peak load or average load? Are you tuning for optimal flexibility because your workloads vary significantly or are you tuning for optimal performance for one specific workload? Decide what your performance goals are and then tune accordingly.
3. Begin with the recommended settings as a starting point
   Your SAP NetWeaver Java system is installed with default settings that provide acceptable performance settings for many workloads. These settings are not meant to be optimal for every workload, but they are meant to be the starting point for tuning the JVM for every workload.
4. Understand tuning and configuration options
   It is important for you to understand how to tune your JVM. This includes knowing which settings will have the biggest impact, how different settings are related, and understanding trade-offs.
5. Rules of thumb can help you get started, but are usually not conclusive
   A good rule of thumb is not meant to be exact, but rather it is designed to help move you in the right direction. A rule of thumb usually takes into consideration a few of the many variables impacting your system. As such a rule of thumb might move you closer to your target, but there is no substitute for actual workload analysis.
6. Try to measure performance consistently
   Measuring performance can be difficult outside of a controlled lab environment. It is important to try and establish some levels of control and consistency if you choose to quantify performance. This includes attempting to make measurements using a workload that is repeatable and fairly consistent.

Most SAP NetWeaver Java Application Server configurations are platform independent. However, with certain settings, the operating system, JVM, and the hardware should be considered. The following recommendations are specific to IBM i and the IT4J JVM, for other configurations consult SAP documentation.

• Number of Server Nodes
• Number of Threads Per Node

Due to the scalability of the IT4J 64-bit JVM, it is possible to achieve a high level of throughput for each server node. Since each additional server node adds significant memory and CPU overhead, limiting the number of server nodes will decrease the memory footprint and processor utilization. A smaller number of server nodes will also reduce the amount of configuration and maintenance.

Recommendation
Add additional server nodes only when the existing, tuned server node(s) does not have sufficient capacity. An indication that an additional server node might be necessary is when performance of the existing server node(s) becomes degraded under an increased load even though the partition where the server node(s) is running has plenty of excess memory and processor capacity. Before adding additional server nodes ensure that the existing nodes have been tuned and are operating efficiently.

Rule of Thumb
Two server nodes per processor

For most workloads it is not necessary to configure more than two server nodes for each processor dedicated to that workload. For workloads that don’t demand an entire processor, less than two server nodes per processor can be configured.

Adding Server Nodes

1. Open the SAP J2EE Engine Config Tool
2. Select the instance where the server node will be added
3. From the menu select Server -> Add Server
4. Click "Yes" if prompted to add another server
5. Click "Ok"

1. Open the SAP J2EE Engine Config Tool
2. Expand the new server node
3. Expand managers
4. Select ConfigurationManager
5. Select rdbms.driverLocation
6. Enter the location of the Native JDBC driver jar file
7. Click "Set"
8. Click "Save"

What Are Server Nodes
Server nodes in the Java Application Server are similar in function to work processes in the ABAP Application Server, but are very different in structure. Both server nodes and work processes are responsible for performing the real work done by the SAP system and both receive that work from the dispatcher.

Whereas a work process is a single threaded job that can only handle one task at a time, a server node consists of a Java Virtual Machine (JVM) that by its nature is multi-threaded and can handle multiple tasks at one time. Because of the multi-threaded nature of the server node, the number of server nodes in a Java Application Server instance will be quite small compared to the number of work processes in an ABAP Application Server instance doing the same amount of work.

Since additional server nodes consume resources, it is important to not add additional server nodes if they are not necessary. The number of server nodes required is dependent on the number of processors available and the expected load. It is important to note that the number of server nodes will not always correspond to the number of processors in a partition.

Overview
To maximize the scalability and the capacity of a particular SAP dispatcher or server process, it is possible and sometimes necessary to adjust the number of threads available. Both, server and dispatcher processes have a system threads pool and server processes have an additional application threads pool. For those thread pools you can define an initial number of threads, a minimum value and a maximum value. The system starts with the initial number of threads and adjusts the number of threads available based upon the load between the minimum and maximum value. Ideally, there should always be free threads that are available for application processing.

However, a thread is a resource that has a certain footprint and therefore the number of idling threads should not be excessively high.

To be able to decide if you need to adjust the thread values, you can use the Monitoring application in the Visual Administrator. Open the node that you want to watch, open the services node and click on Monitoring. Then, under root/Kernel you’ll find nodes called “Application Threads Pool” or “System Threads Pool”. If you see the “WaitingTasksCount” measure being greater than 0 you might want to increment your MaxThreadCount value. If the “CurrentThreadPoolSize” is constantly higher InitialThreadPoolSize you might consider to increase InitialThreadCount.

In the following you’ll find the default values and the instruction on how to change the Thread Pool values.

Default Values for Thread Pools

1. Open the SAP J2EE Engine Config Tool
2. Expand Global dispatcher/server configuration, depending on the node that you want to configure
3. Expand managers
4. Select ThreadManager for the System Threads Pool or ApplicationThreadManager for the Application Threads Pool
5. Select InitialThreadCount, MaxThreadCount or MinThreadCount
6. Change the value
7. Click "Set"
8. Click "Save"

The minimum hardware requirements are amount of hardware resources, particularly memory and processors, required to meet the demands of a workload. Note the following about minimum hardware requirements:

• All applications require a certain amount of resources in order to function
• Running an application in an environment that does not meet minimum resource recommendations leads to an increased risk of functional and performance problems if the application runs at all.
• Resource requirements vary from application to application and vary within the same application depending on configuration, workload, etc...

• Memory requirements include the memory necessary for Java (GC, JIT, JVM), the OS, and the application to start and function properly with acceptable performance.
• Memory usage in the SAP NetWeaver Java environment is coupled tightly with the workload and JVM configuration
  • Memory needs change as the type of workload changes and as the volume of work changes
• Additional memory beyond minimum requirements is recommended to compensate for peak loads and changes in workload.

• Processor requirements include the processing capacity necessary for Java (GC, JIT, JVM), the OS, and the application to start and function properly with acceptable performance.
• Processor utilization generally increases when sufficient memory is unavailable.
• Additional processing capacity beyond the minimum requirements is recommended to compensate for peak loads and changes in workload.

Shared memory pools provide a method for the operating system to dedicate memory to a specific subsystem or job. Because of the Java Garbage Collection mechanism which touches the whole Java heap once in a while you will want to keep the Java heap in main memory and avoid paging. Therefore, running your Java workload in its own memory pool decreases contention of memory due to other running jobs and reduces the likelihood of parts of the heap being paged out.

The disadvantage of configuring a separate memory pool is that the memory is removed from the *BASE pool, which limits how the operating system can manage memory. This could potentially lead to less efficient use of memory resources by the operating system.

Recommendations

• Configure a shared memory pool if your Java instance is mission critical and belongs to a productive system.
• Configure a shared memory pool if your Java instance is consistently memory starved (indicated by excessive paging) due to contention from other workloads on the partition.
• Configure a shared memory pool if it is essential that your Java workload be isolated from other workloads running on your partition.
• Do not configure a shared memory pool if your Java instances are used intermittingly such as in a test or sandbox environment.
• Do not configure a shared memory pool for your Java workload if your system is dedicated to running only that Java workload.

We recommend performance adjuster to be turned off (system value QPFRADJ set to 0).
If the performance adjuster is set (QPFRADJ not set to 0), the shared pool containing SAP NetWeaver Java installations must have a minimum size set according to the SAP system's minimum memory requirements.

How to Change Performance Adjuster
The performance adjustment system value is QPFRADJ. When this system value is set to '2' or '3,' the system periodically checks the performance of all the active shared pools and adjusts or rearranges the storage and activity levels as needed.
The settings on the WRKSHRPOOL display affect the performance adjuster algorithm.

What Is the Performance Adjuster
The performance adjuster allows IBM i to automatically manage the shared memory pools without any user interaction. This function is active by default (the shipped value of QPFRADJ is '2' which means 'Adjustment at IPL and automatic adjustment').
For more details see Java memory tuning tips on the IBM i Information Center (Version 5 Release 4).

Why Turn Off Performance Adjuster
It is important that the SAP NetWeaver Java Application Server always has the minimum required memory available. Running with less than the recommended minimum could lead to functional and performance problems. The Performance Adjuster attempts to balance memory between memory pools based on historical data. In certain situations the Performance Adjuster could allocate less memory than is required for the Java Application Server.

The thread activity level of the storage pool containing the SAP NetWeaver Java Application Server instance should be set based on your workload. Increase the max active value as necessary when threads are marked as ineligible (this can be determined using WRKSYSSTS).

Recommendation
If at any time threads are marked as ineligible increase the max active value in increment of 100 until no threads are marked as ineligible.

In most environments and under most workloads it is not necessary to set max active higher than 1000.

How To Change Max Active

1. Enter WRKSHRPOOL command and change the activity level for a specific memory pool under "Max Active" column. This can also be done using the WRKSYSSTS command.

What Is Max Active
The activity level of a storage pool refers to the number of active threads within that storage pool. If the activity level is too low for the number of threads and jobs that are active, your threads are paged out of main storage and marked as ineligible for a short time. This can drastically affect the performance of your application. In prior releases the value of Max Active had an impact on the database optimizer. This is no longer the case.

How to determine if threads are in ineligible state

1. Enter the WRKSYSSTS command
2. Press F11 once to view the Wait -> Inel and Active -> Inel columns
3. Columns should have value of 0. If the value is greater than 0 the max active value should be increased.

For more details see the Memory pool activity level on the IBM i Information Center (Version 5 Release 4).

Most of the tuning for the Java stack is done with individual JVM system properties. Even though many Java system properties exist and can be modified, most should be left unchanged. For the most part the only JVM system properties that should be maintained are those properties related to the JVM garbage collector, these include maximum heap size (Xmx), initial heap size (Xms) and nursery area size (Xmn).

For most of the recommendations provided, analysis of the workload, specifically the garbage collector output, will be required in order to determine the appropriate settings.

Recommendations
SAP note 1234382 "IBM i: VM Settings for SAP WebAS Java (IBM J9 JVM)" provides initial settings. Since it is impossible to provide a single set of values that work optimally in all situations, it is recommended to use these settings as an initial starting point and then make adjustments based on your workload and analysis.

For checking and cleaning up the Java VM parameters you can use the tool CONFIGJVM. IF you already made adjustments to the Java heap, you should specify the option KEEPHEAP(*YES) when calling the tool. This preserves any changes made to the Java heap. Details about CONFIGJVM can be found in SAP note 1262494.

Also make sure SAP note 990892 is implemented.

Xmx (maximum heap size)
For most workloads the default Xmx setting will be sufficient, however, it may be necessary to adjust Xmx on occasion. Xmx can be adjusted up or down depending on the workload. A high occurrence of global garbage collections and compactions usually indicate that the Xmx setting is insufficient and should be increased. Since IT4J is 64-bit JVM, the heap size can exceed 2 GB in size. In practice it is usually not necessary to expand the heap size beyond 2.5 GB. You should consider adding another server node if a JVM heap needs to expand beyond 2.5 GB.

Consider reducing the Xmx setting if you have a light workload and the JVM is not utilizing the heap.

Xms (initial heap size)
For optimal performance of a specific workload set Xms equal to Xmx. This immediately allocates the maximum heap size for use. Setting Xmx in this fashion is appropriate if you are familiar with the behavior of your workload and if the workload is fairly consistent.

For maximum flexibility set Xms to 25% of maximum heap size. This allows the heap to grow or shrink as required by the workload. Setting Xms in this fashion is appropriate for a load that varies significantly or if you are unsure of how your workload behaves.

Xmn (nursery area size)
Because most garbage collections occur in the nursery area, this setting can potentially impact performance. The size of the nursery area should be set according to the frequency that short lived objects are created. Consider increasing the Xmn setting if the tenured area of the heap is not being fully utilized. Consider reducing the Xmn setting if the tenured area of the heap is being heavily utilized leading to an increased frequency of global garbage collections. Note that nursery area garbage collections (scavenger collections) may outnumber tenured area collections (global collections) by a ratio of up to 50 to 1.

Xlp64k (large pages)
This value should always be set if both the operating system and hardware support large pages. Large pages are supported beginning with IBM i V6.1 and POWER 6 hardware.

Xgcpolicy:gencon (garbage collection policy)
The garbage collection policy should be set to generational for SAP NetWeaver Java workloads. The behavior of SAP NetWeaver Java makes generational the optimal garbage collection policy.

Changing System Properties

1. Open the SAP J2EE Engine Config Tool
2. Expand "cluster-data"
3. Expand the instance
4. Click on the component to be changed
5. Modify or add the property in the pane on the right
6. To set the bootstrap JVM properties, click the "Bootstrap" tab