SAP Community - NW ABAP Monitoring Tools

Monitoring of ABAP System on AWS Using CloudWatch – Part II

2021-04-11T09:38:07+02:00

This is in continuation of my previous blog post Monitoring of ABAP System on AWS Using CloudWatch – Part I

4. DEPLOYMENT OF LAMBDA FUNCTION

4.1 Using Serverless Application Repository

Go to Serverless Application Repository from AWS Console and then go to Available applications → Public Applications and search for sap-monitor

Fill out all the required details for the connectivity to the SAP system which has following

Application Name: This is the stack name of the application created via AWS Cloud Formation

SAPSIDParameter: SAP System ID

LayerVersionParameter: need to provide sapjco Lambda Layer Version

SecurityGroupParameter: Need to provide Security Group IDs which needs to be assigned to the Lambda Function

SubnetParameter: Need to provide Subnet IDs which will provide Internal IP address for the Lambda execution

SAPClientParameter: Need to Provide Client Number of SAP

SAPHostParameter: Need to provide hostname/IP-Address of Primary Application Server

SAPInstanceIDParameter: Need to provide the SAP Instance Number

SAPRFC0UserParameter: Need to provide RFC User for Lambda Function

SAPRFC1PasswordParameter: Need to provide the password the RFC user

ScheduleParameter: This will schedule /SDF/MON job automatically into the system

This will start the deployment at AWS Cloud formation of the Lambda functions with the parameters provided

Below screen will appear when the deployment gets finished

We can see the deployed Lambda function in AWS Console.

4.2 Using AWS CloudFormation

It could happen that the application in Serverless Application Repository is not available for some of the regions then in that case we can also use AWS Cloud Formation template to deploy the same

Go to AWS Cloud Formation from the AWS console → Go to Create Stack and then click on With new resources (standard)

Manually upload the below Cloud Formation template

AWSTemplateFormatVersion: '2010-09-09'

Transform: AWS::Serverless-2016-10-31

Description: SAP NetWeaver ABAP Monitoring sap-monitor-<SID>

Globals:

  Function:

    Timeout: 60

Metadata:

  AWS::ServerlessRepo::Application:

    Name: sap-monitor

    Description: Amazon CloudWatch Monitoring for SAP NetWeaver ABAP-based environments.

    Author: mtoerpe

    SpdxLicenseId: Apache-2.0

    LicenseUrl: s3://sap-monitor/34400b68072d710fecd0a2940a0d1658

    ReadmeUrl: s3://sap-monitor/cb03873fab47b4d2e631b2ecfdd66ff3

    Labels:

    - sap

    - netweaver

    - monitoring

    - cloudwatch

    HomePageUrl: https://github.com/aws-samples/amazon-cloudwatch-monitor-for-sap-netweaver

    SemanticVersion: 1.0.5

    SourceCodeUrl: https://github.com/aws-samples/amazon-cloudwatch-monitor-for-sap-netweaver

  AWS::CloudFormation::Interface:

    ParameterGroups:

    - Label:

        default: Network Configuration

      Parameters:

      - SubnetParameter

      - SecurityGroupParameter

    - Label:

        default: SAP Configuration

      Parameters:

      - SAPSIDParameter

      - SAPHostParameter

      - SAPInstanceIDParameter

      - SAPClientParameter

      - SAPRFC0UserParameter

      - SAPRFC1PasswordParameter

    ParameterLabels:

      SubnetParameter:

        default: Which VPC Subnet(s) should this be deployed to?

      SecurityGroupParameter:

        default: Which Security Groups should be applied?

Parameters:

  SubnetParameter:

    Type: List<AWS::EC2::Subnet::Id>

    Description: Provide Subnet(s) IDs, separate by comma

  SecurityGroupParameter:

    Type: List<AWS::EC2::SecurityGroup::Id>

    Description: Provide Security Group(s) IDs, separate by comma

  LayerVersionParameter:

    Type: String

    Description: Provide <sapjco> Lambda layer version

    Default: '1'

  ScheduleParameter:

    Type: Number

    Description: Schedule /SDF/SMON job automatically (true = 1, false = 0)

    Default: 1

    MinValue: 0

    MaxValue: 1

  SAPSIDParameter:

    Type: String

    Description: Provide SAP System ID (used as namespace/identification only)

    MinLength: 3

    MaxLength: 3

  SAPHostParameter:

    Type: String

    Description: Provide Hostname/IP-Address of Primary Application Server

  SAPInstanceIDParameter:

    Type: String

    Description: Provide SAP Instance Number

    MinLength: 2

    MaxLength: 2

  SAPClientParameter:

    Type: String

    Description: Provide SAP Client

    Default: '000'

    MinLength: 3

    MaxLength: 3

  SAPRFC0UserParameter:

    Type: String

    Description: Provide RFC User

  SAPRFC1PasswordParameter:

    Type: String

    Description: Provide RFC Password

    NoEcho: true

Resources:

  SecretsManager:

    Type: AWS::SecretsManager::Secret

    Properties:

      Name:

        Fn::Sub:

        - sap-monitor-${SID}

        - SID:

            Ref: SAPSIDParameter

      Description:

        Fn::Sub:

        - Secret for SAP Monitoring <${SID}>

        - SID:

            Ref: SAPSIDParameter

      SecretString:

        Fn::Sub:

        - '{ "name":"${SID}", "host":"${HOST}", "sys_id":"${INSTANCEID}", "client":"${CLIENT}",

          "user":"${USER}", "password": "${PASSWORD}", "language":"en", "/SDF/SMON_DESC":"AWSCW",

          "/SDF/SMON_SCHEDULE":${SCHED} }'

        - SID:

            Ref: SAPSIDParameter

          HOST:

            Ref: SAPHostParameter

          INSTANCEID:

            Ref: SAPInstanceIDParameter

          CLIENT:

            Ref: SAPClientParameter

          USER:

            Ref: SAPRFC0UserParameter

          PASSWORD:

            Ref: SAPRFC1PasswordParameter

          SCHED:

            Ref: ScheduleParameter

      Tags:

      - Key: AppName

        Value: SAPMONITOR

  ScheduledRule:

    Type: AWS::Events::Rule

    Properties:

      Name:

        Fn::Sub:

        - sap-monitor-${SID}

        - SID:

            Ref: SAPSIDParameter

      Description:

        Fn::Sub:

        - Scheduler for SAP Monitoring <${SID}>

        - SID:

            Ref: SAPSIDParameter

      ScheduleExpression: rate(1 minute)

      State: DISABLED

      Targets:

      - Arn:

          Fn::GetAtt:

          - CWLambdaFunction

          - Arn

        Id: latest

  PermissionForEventsToInvokeLambda:

    Type: AWS::Lambda::Permission

    Properties:

      FunctionName:

        Fn::GetAtt:

        - CWLambdaFunction

        - Arn

      Action: lambda:InvokeFunction

      Principal: events.amazonaws.com

      SourceArn:

        Fn::GetAtt:

        - ScheduledRule

        - Arn

  CWLambdaFunction:

    Type: AWS::Serverless::Function

    Properties:

      CodeUri: s3://sap-monitor/c716c19711241c175058825388cbeae2

      Layers:

      - Fn::Sub:

        - arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:layer:sapjco:${LLY}

        - LLY:

            Ref: LayerVersionParameter

      Handler: init::handleRequest

      FunctionName:

        Fn::Sub:

        - sap-monitor-${SID}

        - SID:

            Ref: SAPSIDParameter

      Description:

        Fn::Sub:

        - SAP Monitoring for <${SID}>

        - SID:

            Ref: SAPSIDParameter

      Runtime: java11

      MemorySize: 512

      Timeout: 30

      VpcConfig:

        SecurityGroupIds:

          Ref: SecurityGroupParameter

        SubnetIds:

          Ref: SubnetParameter

      Environment:

        Variables:

          SECRET:

            Fn::Sub:

            - sap-monitor-${SID}

            - SID:

                Ref: SAPSIDParameter

      Policies:

      - Version: '2012-10-17'

        Statement:

        - Effect: Allow

          Action:

          - secretsmanager:GetSecretValue

          Resource:

            Ref: SecretsManager

        - Effect: Allow

          Action:

          - cloudwatch:PutMetricData

          Resource: '*'

      Tags:

        AppName: SAPMONITOR

Outputs:

  CWLambdaFunction:

    Description: SAP Monitor Lambda Function ARN

    Value:

      Fn::GetAtt:

      - CWLambdaFunction

      - Arn

Upon uploading the above template then click on Next

Specify the Stack Name for the execution

Specify all the required parameters for execution and then click Next

Specify the IAM Role that is required for cloud formation stack execution

Specify the advanced options for Stack Policy and Rollback Configuration

Specify Notification Options and Stack creation options(if required) and then click Next

Review the settings and then click Create Stack

This will create and execute the stack, and below screen will appear after successful finish of the Stack

5. TESTING DEPLOYMENT

As our lambda function is in place to monitor our ABAP system, we need to test the lambda function that if Lambda function is getting though the SAP System

5.1 Lambda Execution Configuration

We need to check the configuration that will be used for the Lambda execution which should be as per our environment

5.1.1 Memory

We need to configure optimized memory that is allowed for Lambda function to use during the execution. Default is 512 MB and must be changed as per the requirement and test cases

5.1.2 Timeout

We need to configure optimized timeout value after which the Lambda function will be forcefully stopped. Default is 30 seconds and must be changed as per the requirement and test cases

5.1.3 VPC

We need to configure the required value for the VPC of the Lambda function as execution will happen using the configured network settings. VPC, Subnets and Security groups needs to be assigned as per the infrastructure

5.2 Lambda Test Event

First, we need to configure the Test Event for Lambda function which we can feed for lambda execution

Go to Lambda from AWS Console and then click on the newly created function

Click on Test and then put below text and click on Invoke

{

“refresh”:true

}

After a successful execution below screen will appear

5.3 Checking Metrics

After successful execution we need to be sure that Statistical data has been fetched and stored by AWS CloudWatch from the SAP system.

To check the same, go to CloudWatch from AWS Console and then click on Metrics

Then we must see the metric for SAP System under Custom namespace → sap-monitor → by SID

This confirms that test is successful.

This is end of Part – II, in next part Monitoring of ABAP System on AWS Using CloudWatch – Part III of this article you can find more information about Post Configuration, Dashboard, Notifications and Logs of Monitoring ABAP Systems via AWS.

Monitoring of ABAP System on AWS Using CloudWatch – Part III

2021-04-11T10:00:40+02:00

This is in continuation of my previous blog post Monitoring of ABAP System on AWS Using CloudWatch – Part II

6. POST-CONFIGRUATION

As a part of post configuration, we must do the below steps for regular execution

6.1 Enable Scheduler

We must enable the scheduler so that AWS Lambda function can be triggered after regular interval of time to fetch the required data for CloudWatch

To do the same, go to Amazon EventBridge → Event and click on Rules. We can find sap-monitor’s Rule already configured.

Select the Rule and then click on Enable, confirm the action to Enable the Rule

Once the Rule is enabled, then as per the schedule Lambda Function will fetch the data from SAP.

7. DASHBOARD

For monitoring multiple metrics at the same time, we can create and configure the our own dashboard using AWS CloudWatch.

7.1 Creation of Dashboard

To-do the same, go to AWS CloudWatch from AWS Console and then click on Create Dashboard

Then we need to specify the name of the Dashboard which will be created and click Create Dashboard

Dashboard has been created for Monitoring of the system.

7.2 Adding Widgets

Once the dashboard has been created then we need to add different Widgets as per the requirement. To do the same, we need to go to the newly created Dashboard and click on Add Widget

Then we need to specify the data source that will be used to create widgets which is Metrics

Then we need to select the Metric Name that needs to be displayed on the CloudWatch Dashboard and click on Create Widget

We can now see the Graphed Metrics on the Dashboard

Similarly, we can select more widgets to pin the graphical metrics on the CloudWatch Dashboard as below

We must save the dashboard so that widgets can be saved and can be utilized for quicker view.

8. NOTIFICATION

We can also specify the settings by which a notification can be sent as per the requirement.

8.1 Setting Up Alarms

We can setup alarms for sending any type of notification as per the requirement, to do so we need to go to CloudWatch → Alarms and click on Create alarm

Then we need to Specify the metric and conditions to trigger the notification, for that we need to click on Select Metric

Specify the Metric for which we need to generate the Alarm and then click on Select metric

Then we need to specify the time period and statistics for which alarm needs to be set. Here we need time period of 5 mins

Then we need to specify the condition on which alarm needs to be raised by the CloudWatch. Here we have defined the threshold value to 60 and click Next

Then we need to specify the configuration actions when Alarm get a catch. Here we have specified the Amazon SNS topic

We can also specify the Auto scaling group, EC2 action any systems manager Ops Center action, if required and then click on Next

Then we need to specify the name and description of the Alarm and click Next

Then we need to review the specified parameter and create the alarm

Below screen will appear after successful creation of Alarm, but state will be Insufficient Data till the first recording of data

After the first recording of the statistics from the CloudWatch, the status should change to OK

We can also see the Data collection in graphical manner for Alarm

When there is any breach in the threshold specified in the alarm settings then it gets recorded in AWS EventBridge

And information of this breach has been sent via email to the stakeholders as per the SNS configuration

9. CONSIDERATIONS

For using this solution in live customer environment, we have below points which needs to be considered

9.1 Performance

From performance point of view, this solution uses two ways to connect to SAP System Regularity and collect insights: -

One Minute interval for /SDF/MON data with a 30 second granularity which is scheduled daily on demand. According to SAP, CPU and memory overhead on the SAP system caused by /SDF/SMON is negligible and can safely be run in production. Link can be referred for further details

Five Minute interval for ST03 “Last Minute Load”. SAP statistical records (STAD) and related aggregation (ST03) are usually enabled by default, if it’s not enabled then need to check SAP note 2369736. They are the foundation of any SAP monitoring, especially for SAP GoingLive Check, SAP EarlyWatch Alert and SAP Solution Manager.

9.2 Cost

There no addition cost for using this service, but one need to pay for the underlining AWS services which is getting used, such as AWS Lambda, CloudWatch, AWS SNS etc. For a single SAP system with two application servers, the estimated cost is 16 USD per month per SID without considering any free tiers. These are estimated cost which is depend on various factors, please check link for calculation details

To limit the stored log data and reduce the cost in the long run, make sure to also reduce the retention period for /aws/lambda/sap-monitor-<SID> to one week. For more details on this please refer link.

9.3 Capture Metric

A full list of currently captured metrics and their importance can be found below:

Generic (1 - per SID):

"PING" = SAP Monitor RFC Execution Time (/SDF/SMON) "TOTAL_APP_SERVERS" = Number of active SAP Application Servers



/SDF/SMON (14 - per Instance), granularity 1 min:

"ACT_WPS" = Number of Active Work Processes

"ACT_DIA" = Number of Active Dialog Work Processes

"TRFC_FREE" = Number of Available Work Processes for RFCs

"SESSIONS" = Number of Sessions

"USERS" = Number of Logins

"UPDQ" = Update Queue Length

"ENQQ" = Enqueue Queue Length

"DIAQ" = Dialog Queue Length

"PRIVWPNO" = Number of Work Processes in Priv Modes

"DIAAVG20" = Average Load last 20s

"DIAAVG60" = Average Load last 60s

"FREE_MEM_PERC" = Free Memory in % of RAM

"USR_TOTAL" = CPU Utilization (User) in %

"SYS_TOTAL" = CPU Utilization (System) in %



SWNC_GET_WORKLOAD_SNAPSHOT = st03 Last Minute Load (12 - per SID), granularity 5 min



Currently enabled for Tasktypes DIA & RFC:



"ST03_AVG_SNAP" = Average Response Time per Step in ms (ideally < 1s)

"ST03CPU_TIME_PERC_SNAP" = Average CPU Time in % as part of average response time (ideally ca. 40%)

"ST03DB_TIME_PERC_SNAP" = Average DB Time in % as part of average response time (ideally ca. 40%)

"ST03AVG_DB_DIR_SNAP" = Average Database Request Time per Direct reads in ms (ideally max. 10ms)

"ST03AVG_DB_SEQ_AVG_SNAP" = Average Database Request Time per Sequential reads in ms (ideally max. 10ms)

"ST03_AVG_DB_CHG_AVG_SNAP" = Average Database Request Time per Changes in ms (ideally max. 10ms)

If we also like to monitor other OS-related metrics such as free hard disk space, then we can also deploy the CloudWatch Agent on top.

10. LOGS

10.1 CloudFormation

As the deployment triggers Cloud Formation stack internally, hence we can navigate to CloudFormation → Stacks → Events which can provide details of the steps which were carried out during the execution

10.2 CloudWatch

We also have detailed logs which we can see from AWS Cloudwatch, for which need to navigate to proper log groups from CloudWatch

Now SAP System installed on AWS EC2 is getting monitored using AWS Services.

BEST OF LUCK!!

11. CONCLUSION:

In this 3 parts of article, we have discussed how we can Deploy, Test, Post Configuration, Dashboard of Monitoring ABAP Systems via AWS.

THANK YOU..

Monitoring of SAP Systems Using Azure Monitors - Part I

2021-05-10T07:04:19+02:00

This Article is referenced from the approach announced by Microsoft in this blog. Microsoft has also provided documentation on this to help customers on Azure to enhance the monitoring of the SAP systems on Azure.

In this Article we will discuss about the Monitoring Service "Azure Monitor for SAP Solutions" delivered by Microsoft and how we can take benefit from this service.

Azure Monitor for SAP Solutions is an Azure-native monitoring product for customers, running their SAP landscapes on Azure Virtual Machines and Azure Large Instances. With Azure Monitor for SAP Solutions, we can collect telemetry data from Azure infrastructure and databases in one central location and visually the telemetry data. SAP has provided several notes which support SAP products on Azure:-

2015553 - SAP on Microsoft Azure: Support prerequisites

1928533 - SAP Applications on Azure: Supported Products and Azure VM types

We can monitor different components of an SAP landscape such as OS, High-availability cluster, SAP HANA , SAP NetWeaver and etc. Currently, Azure Monitor for SAP Solutions is under public preview and there is high probability of changes in coming days on this service, kindly refer to Microsoft documentation in advance

1. AZURE SERVICES

Below are services that will be used by Azure to deliver the solution

1.1 Azure Monitors

Azure Monitor for SAP Solutions uses the power of existing Azure Monitor capabilities which contains Log Analytics and Workbooks to provide more monitoring capabilities. It read the SAP data available in the Log Analytics and make data available for the workbooks to show it in a graphical Manner.

1.2 Log Analytics Workspace

A Log Analytics workspace is a unique environment for Azure Monitor log data. Each workspace has its own data repository and configuration, and data sources and solutions are configured to store their data in a workspace. All the data which is extracted from SAP Solutions will be stored in this space.

1.3 WorkBooks

Workbooks can query data from multiple sources within Azure. In Workbooks we can transform this data to provide insights into the availability, performance, usage, and overall health of the underlying components. For instance, analyzing performance logs from virtual machines to identify high CPU or low memory instances and displaying the results as a grid in an interactive report. Workbook will use the data from the Log Analytics Workspace and then transforms into Text, charts, Grids, Tiles, Trees, Graphs etc...

1.4 Key Vault

Azure Key Vault is a cloud service for securely storing and accessing secrets. A secret is anything that we want to tightly control access to, such as API keys, passwords, certificates, or cryptographic keys. Vaults support storing software and HSM-backed keys, secrets, and certificates. All the secrets for the SAP systems will be stored in the Key Vault which will be used by the providers to access SAP solutions for collecting the data.

1.5 Managed Identities

Managed identities eliminate the need for services to manage credentials. Managed identities provide an identity for applications to use when connecting to resources that support Azure Active Directory (Azure AD) authentication. Applications may use the managed identity to obtain Azure AD tokens. Managed Identities allow provides to use AD Authentications for retrieving the SAP system secrets from the Key Vaults

1.6 Azure Queue Storage

Azure Queue storage is a service for storing large numbers of messages that can be accessed from anywhere via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account. Queue storage is often used to create a backlog of work to process asynchronously. Queue Storage will be used by the providers to store the messages which came while connecting to the SAP Solutions.

1.7 Azure Virtual Machine

Azure Virtual Machines (VM) is one of several types of on-demand, scalable computing resources that Azure offers. VM will be deployed by the Azure Monitors. This is a Standard_B2ms VM. Responsibility of this VM is to host the Monitoring Payload.

2. ARCHITECTURE OVERVIEW

The following diagram explains about the collection of telemetry by Azure Monitor for SAP Solutions from SAP HANA database. The architecture is agnostic to whether SAP HANA is deployed on Azure Virtual Machines or Azure Large Instances.

2.1 Architectural Components

Azure portal – We can go to marketplace within Azure portal and discover using Azure Monitor for SAP Solutions

Azure Monitor for SAP Solutions resource – this is a landing place for customers to view monitoring telemetry

Managed resource group – This will be deployed automatically as part of the Azure Monitor for SAP Solutions resource deployment. Resources which are deployed and their purpose are:
- Azure Virtual Machine: This is a Collector VM with size Standard_B2ms. The main purpose of this VM is to host the Monitoring Payload. In the above diagram, the monitoring payload contains the logic to connect to SAP HANA database over SQL port.
- Azure Key Vault: This resource is deployed to securely hold SAP HANA database credentials and to store information about providers.
- Log Analytics Workspace: This is the destination where the telemetry data resides.
  - Visualization is built on top of telemetry in Log Analytics using Azure Workbooks.
  - We can use their existing workspace within the same subscription as SAP monitor resource by choosing this option at the time of deployment.
  - We can use Kusto query language (KQL) to run queries against the raw tables inside Log Analytics workspace.

2.2 Architectural Highlights

Multi-instance - We can create monitor for multiple instances of a given component type (for example, HANA DB, HA cluster, Microsoft SQL server, SAP NetWeaver) across multiple SAP SIDs within a VNET with a single resource of Azure Monitor for SAP Solutions.

Multi-provider - The above architecture diagram shows the SAP HANA provider as an example. Similarly, we can configure more providers for corresponding components (for example, HANA DB, HA cluster, Microsoft SQL server, SAP NetWeaver) to collect data from those components.

Open source - The source code of Azure Monitor for SAP Solutions is available in GitHub. We can refer to the provider code and learn more about the product, contribute or share feedback.

Extensible query framework - We can request specific telemetry data to be added to Azure Monitor for SAP Solutions.

3. SOLUTION OVERVIEW

With Azure Monitor for SAP Solutions, we can collect telemetry data from Azure infrastructure and databases in one central location and visually see telemetry data for faster troubleshooting. Azure Monitor for SAP Solutions is an Azure-native monitoring product for customers, running their SAP landscapes on Azure.

3.1 Support

3.1.1 Infrastructure

Azure Virtual Machine

Azure Large Instance

3.1.2 Databases

SAP HANA Database
- Version
  - HANA 1.0 SPS 12 Rev. 120 or higher
  - HANA 2.0 SPS03 or higher
- Topology
  - Single node (scale-up) and multi-node (scale-out)
  - Single database container (HANA 1.0 SPS 12) and multiple database containers (HANA 1.0 SPS 12 or HANA 2.0)
  - Auto host failover (n+1) and HSR
- Microsoft SQL Server
  - Version
    - SQL Server 2012 SP4 or higher
  - Topology
    - Default or named standalone instances in a virtual machine
    - Clustered instances or instances in an AlwaysOn configuration when either using the virtual name of the clustered resource or the AlwaysOn listener name. Currently no cluster or AlwaysOn specific metrics are collected
    - Azure SQL Database (PAAS) is currently not supported

3.1.3 Application

SAP NetWeaver
- Version
  - SAP NetWeaver 7.0 or higher
- Configurations
  - ABAP
  - JAVA
  - Dual-Stack SAP NetWeaver Application Server

3.2 Data Collection

Data collection in Azure Monitor for SAP Solutions depends on the providers that are configured. The following data is being collected

3.2.1 SAP NetWeaver

SAP system and application server availability including instance process availability of Dispatcher, ICM, Gateway, Message Server, Enqueue Server, IGS Watchdog

Work process utilization statistics and trends

Enqueue Lock statistics and trends

Queue Utilization statistics and trends

3.2.2 SAP HANA Database

CPU, memory, disk, and network utilization

HANA System Replication (HSR)

HANA backup

HANA host status

Index server and Name server roles

3.2.3 Linux Operating System

CPU utilization, fork's count, running and blocked processes.

Memory utilization and distribution among utilized, cached, buffered.

Swap utilization, Paging, and swap rate.

Filesystems utilization, Number of bytes read and written per block device.

Read/write latency per block device.

Ongoing I/O count, Persistent memory read/write bytes.

Network packets in/out, Network bytes in/out

3.2.4 Pacemaker High Availability Cluster

Node, resource, and SBD device status

Pacemaker location constraints

Quorum votes and ring status

3.2.5 Microsoft SQL Server

CPU, memory, disk utilization

Hostname, SQL Instance name, SAP System ID

Batch Requests, Compilations, and page Life Expectancy over time

Top 10 most expensive SQL statements over time

Top 12 largest table in the SAP system

Problems recorded in the SQL Server Error log

Blocking processes and SQL Wait Statistics over time

3.3 Regions

As Azure released it for public preview, this service will be available in East US 2, West US 2, East US and West Europe

3.4 Pricing

Azure Monitor for SAP Solutions is a free product (no license fee). We are responsible for paying the cost for the underlying components in the managed resource group such as below: -

Azure Virtual Machine and disk associated with it

Azure Key Vault

Log Analytics workspace

Azure Managed Identity

Storage Account

3.5 SLA

These are released for public previews are excluded from service level agreements.

4. PREREQUISITE

For taking advantage of the above architecture below are some of the pre-requisite

4.1 Authorization

Configuring user must have access for the below: -

Creation of Azure Virtual Machines

Creation and Deletion of secrets from Azure Key Vault

Creation of Managed Identity

Creation and assignment of Network Interface to the VM

Creation of Network Security Groups

Creation of Azure Storage account

Creation of Log Analytics Workspace

4.2 Namespace Registration

Below Namespace should be registered beforehand to perform an uninterrupted configuration: -

4.2.1 Managed Identities

We must register our subscription for using Managed Identities, to check the same we need to go to Subscription → Resource Providers and then search for ManagedIdentity. We must register Microsoft.ManagedIdentity to our Subscription

4.2.2 Operational Insights

We must register our subscription for using Operational Insights, to check the same we need to go to Subscription → Resource Providers and then search for OperationalInsights. We must register Microsoft.OperationalInsights to our Subscription

This is end of Part – I, in next part Monitoring of SAP Systems Using Azure Monitors - Part II of this article you can find more information about Deployment and Configuration of Azure Monitors for SAP Solutions.

Monitoring of SAP Systems Using Azure Monitors – Part II

2021-05-10T07:08:12+02:00

This is in continuation of my previous Article Monitoring of SAP Systems Using Azure Monitors – Part I

6. DEPLOYMENT AND CONFIGRUATION

6.1 Configuring Azure Monitors

Go to Search for Azure Monitors for SAP Solutions

Click on New

Fill up all the requested details and Next

We can add the providers later, hence click on Next

Specify the Tags, if any and click Next

Review the settings and click on Review + Create

This will be initializing the deployment

Wait for the deployment to complete and then click on Go to resource

Now we can see the deployed Azure Monitors for SAP Solutions deployed

5.2 Creating Key Vault

As we need to securely store the secrets for accessing the SAP systems, for that we need to create Key Vault. To do so, Search for Key Vault and then click on Add

Fill out all the requested fields and then click on Next: Access Policy

Enable the access and click Next: Networking

Select the required network and then click on Next: Tags

Specify the tags if required and then click Next: Review + Create

Review all the settings and then click on Create

Wait for the deployment to get finished

Add the secret to Key vault and click on Create

Wait for the secret to save inside the key vault

5.3 Adding Providers

A provider contains the connection information for the corresponding component and helps to collect telemetry data from that component. One Azure Monitor for SAP Solutions resource can be configured with multiple providers of the same provider type or multiple providers of multiple provider types.

We can choose to configure different provider types to enable data collection from corresponding component in their SAP landscape.

We can also choose to configure multiple providers of a specific provider type to reuse the same SAP monitor resource and associated managed group.

It is recommended to configure at least one provider from the available provider types at the time of deploying the SAP Monitor resource. By configuring a provider, we can initiate data collection from the corresponding component for which the provider is configured.

If we don't configure any providers at the time of deploying SAP monitor resource, although the SAP monitor resource will be successfully deployed, no telemetry data will be collected.

5.3.1 Configuring SAP HANA

We can configure one or more providers of provider type SAP HANA to enable data collection from SAP HANA database. The SAP HANA provider connects to the SAP HANA database over SQL port, pulls telemetry data from the database, and pushes it to the Log Analytics workspace in the subscription. The SAP HANA provider collects data every 1 minute from the SAP HANA database.

To add provider, we need to go to Providers in deployed Azure Monitor resource and click on Add

Select Type as SAP HANA and further details will be asked

Fill all the below connectivity details

Input the Private IP for the HANA server.

Input the name of the Database tenant you want to use. We can choose any tenant however, we are using SYSTEMDB here as it enables a wider array of monitoring areas.

Input the SQL port number associated with your HANA database. The port number should be in the format of [3] + [instance#] + [13]. For example, 31013.

Input the Database username you want to use. Ensure that database user has the monitoring and catalog read roles assigned.

Select for key vault which contains the secret for accessing HANA

Select the Key Vault and then click on Secret

Now click on Create

Wait for the connection to be Successful

5.3.2 Linux Operating System

We can configure one or more providers of provider type OS (Linux) to enable data collection from BareMetal or VM Node. The OS (Linux) provider connects to BareMetal or VM Nodes, using Node_Exporter endpoint, pulls telemetry data from the Nodes and pushes it to Log Analytics workspace in the customer subscription. OS (Linux) provider collects data every 60 seconds for most of the metrics from Nodes.

5.3.2.1 Installing Node Explorer

To install Node Explorer we need to go to page https://prometheus.io/download/ and download the required package

Transfer the package to OS and then extract the package

Navigate to the location of the extraction and start the Node Explorer

We can see the telemetry data populated on 9100 port

5.3.2.2 Adding OS Provider to Azure Monitor

To configure the OS (Linux) Provider, we need to add provide with following details and then click on Create

Name – A name for this provider. It should be unique for this Azure Monitor for SAP solutions instance.

Node Exporter Endpoint – Usually it should be http://<hostname>:9100/metrics

Wait for the connection to be established.

5.3.3 Configuring SAP NetWeaver

We can configure one or more providers of provider type SAP NetWeaver to enable data collection from SAP NetWeaver. NetWeaver provider leverages the existing SAPControl webservice interface to retrieve the appropriate telemetry information.

For the current release, below are the standard out-of-box SOAP web methods invoked by Azure Monitor.

Web Method	ABAP	JAVA	Metrics
GetSystemInstanceList	X	X	Instance Availability, Message Server, Gateway, ICM, ABAP Availability
GetProcessList	X	X	If instance List is RED, we can get what Process causing that server to be RED
GetQueueStatistics	X	X	Queue Statistics (DIA/BATCH/UPD)
ABAPGetWPTable	X	-	Work Process Utilization
EnqGetStatistics	X	X	Locks

To add provider, we need to go to Provider and then click on Add

Fill out the requested details: -

Input the hostname of the SAP system and Subdomain (if applicable)

Enter the Instance number corresponding to the hostname entered

Enter the System ID (SID)

Wait for the provider to get successful connection to the system

5.3.4 Configuring High-Availability Pacemaker Cluster

We can configure one or more providers of provider type High-availability cluster to enable data collection from Pacemaker cluster within the SAP landscape. The High-availability cluster provider connects to Pacemaker, using ha_cluster_exporter endpoint, pulls telemetry data from the database and pushes it to Log Analytics workspace in the customer subscription. High-availability cluster provider collects data every 60 seconds from Pacemaker.

5.3.4.1 Installing HA Cluster Explorer

Install ha_cluster_exporter in each node within the Pacemaker cluster.

Clone the github repository to the machine

Use Zypper command to install the HA Cluster Explorer

Run the explorer in background so that data collection can be done

5.3.4.2 Adding HA Pacemaker Cluster to Azure Monitor

Configure a High-availability cluster provider for each node within the Pacemaker cluster.

Fill the following information and then click Create

Name – A name for this provider. It should be unique for this Azure Monitor for SAP solutions instance.

Prometheus Endpoint – http://<servername or ip address>:9664/metrics.

SID – Use the SAP SID.

Cluster name – The cluster name used when creating the cluster. The cluster name can be found in the cluster property cluster-name.

Hostname – The Linux hostname of the VM.

Wait for the provider for creating a successful connection to Cluster

5.3.5 Configuring Microsoft SQL Server

We can configure one or more providers of provider type Microsoft SQL Server to enable data collection from SQL Server on Virtual Machines. SQL Server provider connects to Microsoft SQL Server over the SQL port, pulls telemetry data from the database, and pushes it to the Log Analytics workspace in the customer subscription. The SQL Server must be configured for SQL authentication and a SQL Server login, with the SAP DB as the default database for the provider, must be created. SQL Server provider collects data between every 60 seconds up to every hour from SQL server.

5.3.5.1 Creation of User

To connect to MSSQL server, a user must be created at the database level so that providers from Azure Monitor can connect to MSSQL Server using this user. User must have all the required authorizations and role to extract the required telemetry data.

We can run the below MS script in SQL Server Management Studio to create a user with the appropriate permissions needed to configure the provider

USE [<Database to monitor>]

DROP USER [<user name>]

GO

USE [master]

DROP USER [<user name>]

DROP LOGIN [<user name>]

GO

CREATE LOGIN [<user name>] WITH PASSWORD=N'<password>', DEFAULT_DATABASE=[<Database to monitor>], DEFAULT_LANGUAGE=[us_english], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF

CREATE USER <user name> FOR LOGIN <user name>

ALTER ROLE [db_datareader] ADD MEMBER [<user name>]

ALTER ROLE [db_denydatawriter] ADD MEMBER [<user name>]

GRANT CONNECT TO <user name>

GRANT VIEW SERVER STATE TO <user name>

GRANT VIEW SERVER STATE TO <user name>

GRANT VIEW ANY DEFINITION TO <user name>

GRANT EXEC ON xp_readerrorlog TO <user name>

GO

USE [<Database to monitor>]

CREATE USER [<user name>] FOR LOGIN [<user name>]

ALTER ROLE [db_datareader] ADD MEMBER [<user name>]

ALTER ROLE [db_denydatawriter] ADD MEMBER [<user name>]

GO

5.3.5.2 Adding MSSQL Provider to Azure Monitor

After selecting the Add Provider we need to select Microsoft SQL Server from the drop-down menu. And then fill out all the requested details so that Provider can connect to MSSQL Server

Wait for the connection to be successful.

This is end of Part – II, in next part Monitoring of SAP Systems Using Azure Monitors – Part III of this article you can find more information about Dashboards, Workbooks and Displaying Graphs using Azure Monitors for SAP Solutions.

Monitoring of SAP Systems Using Azure Monitors – Part III

2021-05-10T07:11:11+02:00

This is in continuation of my previous Article Monitoring of SAP Systems Using Azure Monitors – Part II

6. WORKBOOKS

Workbooks provide a flexible canvas for data analysis and the creation of rich visual reports within the Azure portal. They allow you to tap into multiple data sources from across Azure and combine them into unified interactive experiences.

6.1 SAP HANA

For checking the Workbook for SAP HANA, we can select SAP HANA under the Monitoring TAB and then select the Monitored SAP HANA instances which we need to display

This will first display the Overview of the SAP HANA Instances in which we can see an overview for Highest CPU and Memory touched so far

We can also see further details to the CPU, Memory, Disk and Network in and out along with the services running on the HANA Server

We can see detailed CPU utilization timeline under the Utilization Tab

In same tab we can have the detailed CPU utilization with MAX touched so far

On selecting the Memory button, we can see the Memory timeline for the Server.

In the same tab, we can see the detailed utilization of the Memory on HANA and MAX touched so far

In Backup tab, we can see all the successful backups of Database as well as Log area.

In same tab, we can see the Backup size and duration in a graphical view to check the trend of the backups

Going forward in same tab we can also see more detailing about each backup happened for the Hana

Under Replication Tab, we can also see the graph contains the details about replication tier for HANA Database

In same tab we can see further statistics of the replication of SYSTEMDB as well as its associated tenants

6.2 Linux Operating System

For checking the Workbook of Linux Operating System, we can select the OS(Linux) under the Monitoring Tab

In the same tab we can see the Average CPU Usage per mode in graphical view

Go further to the same tab we can also see overview Average Memory Usage, file system occupied space and Network in and out of the VM

In CPU Tab, we can see the details about CPU usage and timeline.

In the same tab we can see the Process and Forks details with count

In Memory Tab, we can see the Memory timeline usage and distribution

In same tab we can see the usage and timeline for the SWAP which is getting used

Going further in the same tab, we can see the details about the page and swap rate

In disk tab, we can see the mount points and respective details

In the same tab, we can see Read/Write Bytes and time along with the Output I/O

In Network Tab, we can see the inbound/outbound traffic, Network Packets and Network Bytes transferred

6.3 SAP NetWeaver

To check the workbook of SAP NetWeaver, we need to go to SAP NetWeaver Blade under Monitoring Tab

Under Overview tab we can find the availability graph of the system. We can also group the same by SID

We can also group the availability by application server

In Queue Statistics tab we can see the details about the dispatcher queues

6.4 High-Availability Pacemaker Cluster

To see the Workbook for HA Pacemaker Cluster, we need to go to the High-Availability Cluster(pacemaker) cluster blade under the Monitoring tab

In this we can see the cluster availability status in a graphical view

In the same tab, we can see the Node status and Resource Status which is attached to the Pacemaker Cluster

Going further to the same tab we can see the status of the primary node over time and resources under the primary node

Same way, we can see the status of the secondary node over time and resources attached to the Secondary node

6.5 Microsoft SQL Server

To see the Workbook of Microsoft SQL Server, we need to go to the Microsoft SQL Server blade under tab Monitoring

We can see the Batch request per sec, Compilations and Re-Compilations per second, CPU Usage and Latencies of the Microsoft SQL Server

In the same tab we can also see the hit ration and page life in a graphical view

Under Statistics Tab, we can see Memory overview as well as top 10 waits statistics

We can also see the top 10 SQL Statements and Top 12 Largest Tables in the SQL Server

Under Backup Tab, we can see the backup duration and sizes in the graphical view

This is end of Part – III, in next part Monitoring of SAP Systems Using Azure Monitors – Part IV of this article you can find more information about Alerts, Notifications and Troubleshooting Logs for Azure Monitors for SAP Solutions.

Monitoring of SAP Systems Using Azure Monitors – Part IV

2021-05-10T07:14:32+02:00

This is in continuation of my previous Article Monitoring of SAP Systems Using Azure Monitors – Part III

7. ALERTS

We can also setup alerts on the data that is being stored in the Log Analytics, we can design the alert as per our thresholds and requirements

7.1 Create Action Group

For sending any notification we first need to create action group in azure Monitor, to do so we need to go to Monitor → Alerts → Manage Actions and then click on New Action Group

Fill out all the requested details and then click on Next: Notifications

Fill out all important details such as Name, Notification type, email id etc.… and then click on Next: Action

Fill the details about the Actions if any and then click on Next: Tags

Specify the Tags if any and then click on Next: Review + Create

Review all the settings of the Action Group and click on Create

Wait for the deployment to finish

7.2 Setting Up Notification

For settings up alert, we first go to Azure Monitor → Alert and click on New Alert Rule

Select the resource to Log Analytics Workspace in which the telemetry data is getting stored

We also need to specify the Condition by clicking Add Condition and then click on Custom log search

Specify the query which needs to be executed for triggering alert, here we are only using the query to check if system is up and running fine.

Specify the condition after which email will be triggered and click Done

Condition will be added to the Alert

Now add action group to which email will be triggered

Specify the other requested details and click on Create

Wait for the Rules to create which we can see in the Alert Rules

So, Once Log Analytics receive GRAY from the sapcontrol as below

Alert will catch this anomaly and display it on Alert Dashboard

Further details about Alert can be checked by clicking on this.

Below is the sample email sent with details about the alert

8. LOGS

8.1 Log Analytics Workspace

We can see and check the telemetry data which is been extracted by the providers

8.2 Azure Monitor Activity Log

We can also check the deployment logs about the providers in the Activity log of Azure Monitor

8.3 Extraction Log

We also see the extraction log of each call to system in the queue of Storage account.

Now SAP System installed on Azure is getting monitored using Azure Monitors for SAP Solutions.

BEST OF LUCK!!

9. CONCLUSION:

In this 4 parts of article, we have discussed how we can Deploy, Configure, Workbooks, Alerts, Notifications and Troubleshooting logs for Azure Monitors for SAP Solutions.

THANK YOU..

Do you know the Deadlock Monitor?

2021-08-26T09:27:50+02:00

Purpose

Deadlocks can occur in any SQL Server multiuser application. The occurrences of deadlocks are a problem of the application. It is not a SQL Server problem. SQL Server and SAP provide tools to detect, monitor and analyze the deadlocks. But in the end the solution must lie with the application development and process management.

Sometimes the deadlocks are handled properly by the application and sometimes not. Unexpected deadlocks can occur if performance problems cause transactions to hold locks longer than expected, or if wrong query plans are used, or if ad hoc statements interfere with the system. There are many reasons why deadlocks could cause system problems so it's vital to have a good tool to analyze them.

Below you can learn how to use the DBACOCKPIT -> Diagnostics -> Deadlock Monitor action in SAP NetWeaver which is available when SAP NetWeaver is running on SQL Server or a remote DBACOCKPIT connection to SQL Server is set up for monitoring.

Setup

The Deadlock Monitor is always available for the local system. If the DBACOCKPIT is used to monitor a remote SQL Server, some features may not be available.

Using the Deadlock Monitor

When you enter the Deadlock Monitor action for the local system, you will see two tabs:

Deadlocks Reported by ABAP Stack

The first tab is the old deadlock monitor. It is based on the MSSDEADLCK table. This table exists in the SAP NetWeaver dictionary and it is filled by DBSL (the DB interface of SQL Server) when SAP NetWeaver encounters deadlocks. This feature only tracks deadlocks generated by the SAP NetWeaver kernel. If deadlocks were encountered by any other software, they will not appear here.

The SAP NetWeaver system running on the schema to be monitored records the deadlocks in table MSSDEADLCK and it also deletes the records after one month using an SM37 job (report RSMSSDCN).

In this context only two features possible: Table detail and ABAP code. Those commands are available in the ALV toolbar. The table detail button navigates you to the table detail action for the table listed in the selected deadlock line. The table detail button is always enabled. The ABAP code button is only available when monitoring the local system. It opens an ABAP editor for the OPEN SQL statement where the error occurred.

Deadlocks Recorded in X-Events

There are two generations of this screen. the first is a simple list of deadlocks recorded in the SQL Server Systemhealth (syshealth) X-events session. The second is a more sophisticated summary list, with the original detailed list below.

First Release

First we will describe the earlier version:

Deadlock List

This shows that a deadlock has occurred, and it shows the "victim" of the deadlock, i.e. the session that was selected by SQL Server to trigger an error and reset the transaction. It also shows the "survivor" of the deadlock, i.e. the other process which was involved but was not affected by the deadlock. Using the toolbar buttons, you can view the details of the tables involved (owner object and waiter object), examine the ABAP code of either the victim or the survivor, and view the full XML graph in a browser. If SSMS (the SQL Server Management Studio) is installed on the local PC, the deadlock can be opened in SSMS showing a graphical view of the deadlock chain.

When you press the "Owner object detail" button, the DBACOCKPIT action "Single Table Analysis" is invoked. The same action can be called for the "Waiter object detail" which is the object that the process is trying to lock but has to wait because the resource is locked by another process.

New Version

The newer version is called "Deadlocks from X-Events Session". Instead of a simple list, the screen now has three sections:

Deadlock Summary

Inputbuffer of the session which encountered the deadlock

Deadlock Details. This is the same list as the one described in the earlier version.

This new screen has several advantages over the old one. The top summary view is a grouping of the most common deadlocks that have occurred on the system, with the most common at the top. The list is ordered by the victim flag (victims at the top, survivors on the bottom) and the number of occurrences of that particular deadlock.

The reason this list was provided is that with the old simple list it was very difficult in some cases to determine which deadlock occurred most often in order to focus on the most problematic type of deadlock.

The middle section shows the full SQL statement. If the deadlock entry above is a victim of a deadlock, then it is the statement that failed, otherwise it is the statement that was executing in the surviving process when the deadlock occurred.

The bottom section is the same as before, but when you click on the "Details for selected deadlock" button in the summary list, this detail list will only show you the occurrences for that deadlock. This can be reverted by clicking on "All Deadlocks" in the detail list.

The X-Events Session

The older version of the X-Events tab used the SQL Server Systemhealth (syshealth) session. The new version introduces a new separate X-Events session named "SAP_Deadlocks". After the support package of the new version is applied to a system, a new SQL Server "Extended Events" session appears in the SQL Server Management Studio under Management -> Extended Events -> Sessions. This new session is more efficient because it records only deadlock events and nothing else. The data is also kept longer than in the syshealth session, since only deadlocks are recorded.

The old screen sometimes took a long time to read data from the syshealth session because the syshealth session grows very large on busy systems. The new screen should always appear much faster.

Usage Scenarios

The starting point for understanding deadlocks, how and why they occur and how they can be tracked and analyzed is SAP Note 32129. This note refers to other notes which provide detailed information about various aspects of handling deadlocks in SAP systems running on SQL Server. This information can sometimes be overwhelming, so we provide only a brief overview here.

Some reasons for deadlocks (not a complete list):

Missing index to support a DML statement
This can cause the statement (UPDATE, DELETE) to lock many more rows in the table than are actually modified in the end. Usually there are scans and searches that lock a lot of rows. In this case, use the SQL DML statement WHERE condition to determine whether there is a fitting index. Use the object detail buttons to jump to single table analysis to view the existing indexes.

Application does not serialize execution of critical changes
This allows parallel execution, which can result in deadlocks. It is important to try to understand the application context in which the deadlock occurred. Use the "ABAP code" button to jump to the Open SQL statement and examine the surrounding code. Try to understand how the two processes interfered with each other, why they were running simultaneously (parallel jobs etc.) and how this can be avoided either by scheduling jobs at a different time or with a different data set, or by modifying the code to synchronize access to a critical path.

Long running statements
This can be a SELECT or any other statement. If a statement is slow due to system performance and the transaction remains open for a longer period, the system is at risk of deadlock. This is a common scenario. Long running statements are inefficient and consume a lot of resources and time, but they can cause even worse problems if they occur within a transaction which has already acquired some locks. This drastically increases the likelihood of deadlocks in some cases. So, resolving performance issues (adding indexes, hints, altering the Open SQL statement, etc.) can often magically reduce the occurrences of deadlocks in a process.

Application does not commit often enough
If there is a long running batch job that runs mass modifications within a single transaction, this leads to many open locks. This can then interfere with normal short user transactions that modify the same tables.
To monitor this, SAP has provided a new feature in DBACOCKPIT: Performance -> History -> Open Transaction History. This feature is available in the same support packages as the newer Deadlock Monitor user interface. We will not go into detail on this feature here. This will be covered in a separate blog post. A link will be added here soon.
Open transactions need to be avoided for two reasons. First to avoid holding locks too long, which increases the likelihood of deadlocks, and second because a long transaction consumes excessive log space and may cause the database to run out of log space.
To solve this problem, the application must be analyzed to either increase the frequency of COMMITs or use a smaller data set.

Different access order
In this case, the application modifies the data in more than one process, each of which uses a different access order. Or different indexes are used in different DML statements for the same table. Sometimes there is an ABAP internal table that contains a number of rows, and the content of the table is not consistently ordered the same way when the concurrent processes scan the table and execute modifications (perhaps in multiple tables) for each row.
In some cases, the solution can be as simple as adding a "sort" statement in ABAP to make the access more consistent.

To summarize: The Deadlock Monitor can be used to determine the various reasons for deadlocks and then one or more of the following actions can be taken:

Add an index to get a better access plan for queries (to shorten a transaction) and/or DML operations (to have a smaller footprint).

Optimize queries using database hints or update statistics on tables. SQL Server automatically updates statistics on tables, but in rare cases, manual updating of statistics is required.

Recommend code changes in Z customer code or open an SAP case if the problem occurs in SAP code.

If this is not sufficient and a more detailed analysis is needed, we recommend three blogs by Clas Hortien from Microsoft Germany:

Deadlock Analysis (basic)

Deadlock Analysis (advanced)

Deadlock Analysis (examples)

These blogs are tailored for SAP workload, but mainly describe native SQL Server tools. The blogs can be accessed here.

Release Information

The older version of the deadlock monitor, with the deadlocks recorded in ABAP stack and X-events have been part of DBACOCKPIT for many years. However, the new features (summary list, inputbuffer) were delivered in mid-2021. The SP levels of the new feature are as follows:

SAP NetWeaver 756 SP01
SAP NetWeaver 755 SP03
SAP NetWeaver 754 SP05
SAP NetWeaver 753 SP07
SAP NetWeaver 752 SP10
SAP NetWeaver 751 SP13
SAP NetWeaver 750 SP23
SAP NetWeaver 740 SP27
SAP NetWeaver 731 SP30

The new features are not supported in earlier releases than SAP NetWeaver 7.30 EhP1 (SAP_BASIS 731).

Conclusion

I hope this overview of the Deadlock Monitor, and how to use it to address various types of deadlock issues, proves helpful to you. If you have further questions about the Monitor, or about a deadlock problem you are encountering, I encourage you to ask your questions in our SAP Community using this link.

If you want to read more about SAP on SQL server then I invite you to view our topic area here.

Why is my SQL Server database log growing out of control risking system standstill?

2021-11-18T23:09:42+01:00

Purpose

It is very important to monitor the transaction log size of SQL Server. If SQL Server runs out of log space the system will come to a standstill. The log space is released when a log backup is performed. But the space is only made available up to the starting point of the longest running open transaction. That means that if transactions are held open for a long time, the log will grow out of control. We have seen cases where transactions were held open for days in some SAP background jobs which resulted eventually in a full log and system standstill.

If transactions are open for a long time, and they also hold many locks on database objects, then un-necessary blocking can occur resulting in unexplained bad performance. This also increases the risk of deadlocks in the system.

Until now the only indicator in DBACOCKPIT that a transaction has been open too long has been the DB02 main screen (DBACOCKPIT -> Space -> Overview). The middle section of the screen looks like this:

DB02

What is new is the "History" button. That is the topic of this blog post. This button takes you to the new DBACOCKPIT action where you can view a history of long running transactions and examine the details of not only the processes which held a transaction for a longer time, but also the locks which were held by the session and blocked other sessions.

The screen is also available under DBACOCKPIT -> Performance -> History -> Open Transaction History

Setup

The Open Transaction History is always available for the local system if it's running on SQL Server, or if the DBACOCKPIT is used to monitor a remote SQL Server.

The action is supported for SQL Server 2005 and higher.

Using the Open Transaction History

The DBACOCKPIT action is available under Performance -> History -> Open Transaction History. The initial screen looks like this:

Open Transaction History

The top part of the screen shows a list of sessions which held a transaction open for more than 1 minute. The system checks this once every 20 minutes. The list shows when the data was collected, the session ID and which application program was using the session. The start time of the transaction and the duration in minutes so far at the collection time is also shown.

There is a column in the list which contains the first part of the SQL statement which was the last one executed on the session when the data was collected (the inputbuffer). Usually this statement is much longer than the area provided on the list. You can view the full statement by selecting the row and clicking on the "Details for selected row" button. Double-clicking on the line has the same effect. This will cause the full statement to appear in an edit control in the middle part of the screen.

The list also shows some attributes of the session such as the CPU time and number of reads. Also the current log size and the allocated space within the log at the time of collection are shown.

The toolbar of the ALV control has a couple of useful buttons: "ABAP Code" and "Table Detail". The system tries to determine the ABAP report name and location and also the primary table name to allow navigation to either view the ABAP code in an editor or view the table definition. The "ABAP Code" button is disabled when monitoring remote servers.

Active Request

When a session is checked there may be a SQL statement currently running on the session or it may be inactive. This is indicated by the column "Active request". If there is no active request it means that at the time the session was just waiting for the next SQL statement to be issued and no activity was occurring in that session on SQL Server. There can be many reasons for this. The ABAP program may be executing a long running calculation, it may be waiting in a remote function call (RFC), waiting for an enqueue object or working on executing SQL statements on another connection. If there is no active statement then the statement shown in the inputbuffer edit control is just the last statement executed prior to the data collection time, it wasn't being executed at the time.

If there was an active request then the columns to the right contain information about the request. CPU and I/O numbers will show how many resources were consumed by the request, what the wait type was and how long the wait had been.

If there was an active request on the session which was saved in the history, then the button "Show Active Request" in the toolbar will navigate to the statement analysis screen to examine the statement and the query plan of the request that was active.

Locks

The lower third of the screen shows a list of sessions which were blocked by the selected top level session which was determined to have been running the longest. If there was a hierarchy of blocked processes then the "SQL statement text" column will show which sessions were blocked in the hierarchy by showing arrows of different lengths indicating the depth of the session in the blocking hierarchy. This display is similar to the Performance -> Locks screen which shows currently blocked sessions.

For each of the sessions which were blocked the statement is shown and the button "SQL Statement" allows you to view the full statement and the full query plan in the statement monitor. This also allows navigation to the ABAP code and all the other statement analysis features such as viewing table and indexing details.

Usage Scenarios

Long running transactions can cause two different types of problems on the database server:

Allocated log space grows out of control. If the log runs out of space then no changes can be made in the database anymore.

Blocking locks which are held too long can cause performance bottlenecks and in some cases deadlocks can occur. This can happen even on systems which have more than sufficient CPU and I/O resources.

It is important to monitor the allocated log space over time and make sure there is always plenty of space available on the disk. In DBACOCKPIT go to Space -> History -> Database Size History and check the "Allocated log space (MB)" column. You will see that the log space allocated is reduced when log backups occur and then gradually increases until the next backup. Always make sure that the peak usage is well within the total available log space on the disk. If the allocated space in the log behaves erratically with extreme peaks, then it's an indication that transactions are held too long. In that case use the Open Transaction History screen to find the longest running transactions that have been caught by the history job. Use ALV features to sort the data according to the number of minutes the transaction was held open. Make a note of when these transactions start, which jobs open the transactions and whether there are performance problems or other artificial delays which could cause a transaction to remain open for an extended period.

Sometimes excessive blocking can cause performance problems. If users complain that some of their transaction are intermittently very slow, whereas most other transactions are fast, that's an indication that excessive blocking is occurring. In that case you can use DBACOCKPIT -> Performance -> SQL Statements and sort the data by "Wait ratio: elapsed/CPU time". There you will see statements which run for a long time, but don't use much CPU time, i.e. they are blocked. In addition you can visit the Open Transaction History screen and see if the "Blocking" column is checked. Then double click on those lines where "Blocking" is indicated. On the bottom ALV list you will be able to see all the sessions which were blocked by that statement. You may see a long list of blocking and blocked sessions. You can see which work process was used, whether it was a dialog or batch process. When you click on the "SQL Statement" button in the toolbar you can see which ABAP report was used and which table was used. This may help you determine how and why the job or frontend process was blocked and possibly how to avoid such a situation.

Release Information

This DBACOCKPIT feature was delivered with the following SAP_BASIS support packages:

756 SP01
755 SP03
754 SP05
753 SP07
752 SP10
751 SP13
750 SP23
740 SP27
731 SP30

The new feature is not supported in releases earlier than NetWeaver 7.30 SP1 (SAP_BASIS 731).

Conclusion

I hope this overview of the Open Transaction History monitor proves helpful to you. If you have further questions about the Monitor, or about a long running open transaction you are observing in your system, I encourage you to ask your questions in our SAP Community using this link.

If you want to read more about SAP on SQL server then I invite you to view our topic area here.

OData Calls Statistics in ST03

2021-11-29T16:26:18+01:00

ST03/ST03N – SAP Workload Monitor, contains performance statistics for different task types, including HTTP(s) calls.

ST03 - Workload Overview

When you try to identify the top HTTP(s) calls, influencing your system, you can probably see the following picture.

ST03 - Transaction Profile

There will be a report SAPMHTTP on top of the list, consuming most of the total response time, and then some ABAP WebDynpro Applications (starting with symbol: <{>) and BSP Applications / WebUI Components (with names as is or starting with BSPWD*).

But you won’t see any OData calls because this statistic is hidden behind the report SAPMHTTP.

This is default configuration and it’s done so in order to limit the number of aggregated records, so that the data set is not getting too big, which may lead to ABAP memory dumps.

If you want to see the performance statistics for HTTP / OData calls, you need to adjust some settings.

First execute report SWNC_CONFIG_URL and chose the rule for aggregating HTTP calls. Select either “Break-Down by URL” or “Break-Down by URL and Page”.

Report SWNC_CONFIG_URL

If you configure this way, the Odata calls will be aggregated with complete URL, like “/sap/opu/odata/sap/ZMY_ODATA_SRV/MyEntitySet”.

You can leave them as is, or you can also remove or replace unnecessary prefix “/sap/opu/odata/sap” with a special symbol or characters. This setting is done in the report SWNC_CONFIG_PATH.

Report SWNC_CONFIG_PATH

Press F1 (“Help”) to get more help regarding the settings.

Alternatively refer to the SAP Note: 1631033 – “Normalization of background job names for statistics data”.

After all these settings you will be capable to see more detailed performance statistics per HTTP(s) / OData calls.

ST03 - Aggregation of HTTP Calls

Note: But bear in mind that the change of configuration may lead to ABAP memory dumps.

Please check the following blog post: https://blogs.sap.com/2015/05/06/sapcollectorforperfmonitor-dumps-with-exporttoomuchdata-or-tsvtnewpageallocfailed/

P.S. If something is not working, don't report here! Please, raise SAP incident instead.

BR, Dima

Measure your Application's Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS

2021-12-23T13:26:34+01:00

Imagine that you have designed and implemented this great app your users have been waiting for. But if it is too slow, they will not like it. System administrators will hate it if its resource consumption is too high. How do you know ahead of time that your app offers great functions and also runs fast and lean? You use transaction STATS to evaluate statistics records associated with your application's executions.

In this blog post I explain how you obtain statistics records with high quality and how you analyze them with transaction STATS. In the best case, such an analysis proves that your app combines excellent single user performance with low resource consumption and is scalable. In all other situations, the analysis will indicate the bottleneck where optimizations shall be applied.

As owner and developer of transaction STATS I am more than happy to share tips and tricks that will help you to efficiently work with the tool and to obtain insights into your application's performance.
You can use STATS since SAP NetWeaver 7.40. I am continuously improving it and this blog post presents the tool’s state for SAP NetWeaver 7.57 as of late December 2021.

What are Statistics Records?

Statistics records are logs of activities performed in SAP NetWeaver Application Server for ABAP or on the ABAP platform. During the execution of any task by a work process in an ABAP instance, the SAP kernel collects header information to identify the task, and captures various measurements, like the task’s response time and total memory consumption. When the task ends, the gathered data is combined into a statistics record. These records are stored chronologically in a file in the application server instance’s file system. Collecting statistics records is a technical feature of the ABAP runtime environment and requires no manual effort.

The measurements in these records provide useful insights into the performance and resource consumption of the application whose execution triggered the records’ capture. Especially helpful is the breakdown of the response times of the associated tasks into DB requests, ABAP processing, remote communication, or other contributors.

The value of your performance analysis depends on the quality of the underlying measurements. While the collection of data into the statistics records is performed autonomously by the SAP kernel, you need some preparation to ensure that the captured information accurately reflects the performance of the application.

Ensure that the test system you will use for the measurements is configured and customized correctly.

Verify that the test system is not under high load from concurrently running processes during your measurements

Carefully define the test scenario to be executed by your application. It must adequately represent the application’s behavior in production.

Provide a set of test data that is representative of your productive data. Only then will the business scenario execution resemble everyday use.

Execute the scenario a few times to fill the buffers and caches of all the involved components (e.g., DB cache, application server’s table buffer, web browser cache). Otherwise, your measurements will not be reproducible, but will be impaired by one-off effects that load data into these buffers and caches. It is then much harder to draw reliable conclusions.

After this preparation, you execute the measurement run, during which the SAP kernel writes the statistics records that you will use for the analysis.

How Do You Analyze Statistics Records?

To display the statistics records of your measurement run, you call transaction STATS. Its start screen (Fig. 1) consists of four areas, where you specify criteria for the subset of statistics records you want to view and analyze.

Figure 1: On the STATS start screen, you define filter conditions for the subset of statistics records you want to analyze, specify from where the records are retrieved, and select the layout of the data display.

In the topmost area, you determine the Monitoring Interval. By default, it extends 10 minutes into the past and 1 minute into the future. Records written during this period are displayed if they fulfill the conditions specified in the other areas of the start screen. Adjust this interval based on the start and end times of your measurement run so that STATS shows as few unrelated records as possible.

In the Record Filter area, you define additional criteria that the records must meet—for example, client, user, or lower thresholds for measurement data, such as response time or memory consumption. Be as specific and restrictive as possible, so that only records relevant for your investigation will be displayed.

By default, statistics records are read from all application instances of the system. In the Configuration section, you can change this to the local instance, or to any subset of instances within the current system. Restricting the statistics records retrieval to the instance (or instances) where the application was executed shortens the runtime of STATS. The option Include Statistics Records from Memory is selected by default, so that STATS will also process records that have not yet been flushed from the memory buffer into the file system.

Under Display Layout, select the domain you want to focus on and how the associated subset of key performance indicators (KPIs)—the captured data—will be arranged in the tabular display of statistics records. The Main KPIs layouts provide an initial overview with the most important data and is a good starting point.

What do You Learn from Statistics Records?

Fig. 2 shows the statistics record display based on the settings specified in the STATS start screen (Fig. 1).

Figure 2: Statistics records matching the criteria specified on the STATS start screen.
The list shows statistics records that correspond to the creation of a sales order with SAP GUI transaction VA01.

The table lists the selected statistics records in chronological order and contains their main KPIs. The header columns—shown with a blue background—uniquely link each record to the corresponding task that was executed by the work process. The data columns contain the KPIs and indicate the performance and resource consumption of the tasks. Measurements for times and durations are given in milliseconds (ms). Memory consumptions and data transfers are measured in kilobytes (KB).

The table of statistics records is displayed within an ALV grid control and inherits all functions of this SAP GUI tool: You can sort or filter records; rearrange, include, or exclude columns; calculate totals and subtotals; or export the entire list. You can also switch to another display layout or modify the currently used layout. To access these and other standard functions, expand the toolbar by clicking on the Show Standard ALV Functions button (the black triangle that points to the right).

The measurements most relevant for assessing performance and resource consumption are the task’s Response Time and Total Memory Consumption. The Response Time measurement starts on the application server instance when the request enters the dispatcher queue and ends when the response is returned. It does not include navigation or rendering times on the front end, or network times for data transfers between the front end and the back end. It is strictly server Response Time. (Exception: For SAP GUI transactions that use GUI controls, the RFC-based communication with them occurs in roundtrips. Their total time is measured as GUI Time and included in the Response Time. Parts of GUI Time may be contained in Roll Wait Time, the remainder will contribute to Processing Time.) The end-to-end response time experienced by your application’s users may be significantly longer than the server Response Time. The most important contributors to server Response Time are Processing Time (mostly the time it takes for the task’s ABAP statements to be handled in the work process) and DB Request Time (the time that elapses while database requests triggered by the application are processed). In most cases, Total Memory Consumption is identical to the Extended Memory Consumption, but Paging Memory or Heap Memory may also contribute to the total.

From this list of statistics records, using criteria appropriate to your investigation, you find and inspect statistics records for tasks that are too slow or require too many resources.

Since even the most basic statistics record contains too much data to include in a tabular display, STATS gives you access to all measurements—most notably the breakdowns of the total server Response Time and the DB Request Time, and the individual contributions to Total Memory Consumption—of a selected record by double-clicking any of its columns. This leads you to an itemized view of the record’s measurements in a pop-up window (Fig. 3).

Figure 3: Pop-up window with all available measurements for a statistics record, subdivided into semantic categories on separate tabs.

At the top, it identifies the statistics record via its header data. With the up and down triangles to the left of the header data you can navigate from record to record within this pop-up. The available technical data is grouped into categories, such as Time, DB, or Memory and Data. Use the tabs to switch between categories. Tabs for categories without data for the current statistics record are inactive and grayed out. Documentation is available by single-clicking the field name.

To assess the data captured in a statistics record, consider the purpose that the corresponding task serves.
OLTP (Online Transaction Processing) applications usually spend about 25% of their server Response Time as DB Request Time and the remainder as Processing Time on the application server. For tasks that invoke synchronous RFCs or communication with SAP GUI controls on the front end, associated Roll Wait Time may also contribute significantly to server Response Time. For OLTP applications, the expected end-to-end response time is about 1,000 ms and the typical order of magnitude for Total Memory Consumption is 10,000 KB. Records that show significant upward deviations may indicate a performance problem in your application. You should analyze them carefully and use dedicated analysis tools such as transactions ST05 (Performance Trace) or SAT (Runtime Analysis).
In comparison, OLAP (Online Analytical Processing) applications typically create more load on the database (absolute as well as relative) and may consume more memory on the application server. Users of OLAP applications may have slightly relaxed demands of their end-to-end response times.

Conclusions

You want to optimize the performance of your applications and you want to ensure their linear scalability. For this you need reliable data that indicates the software’s response times and resource consumption. For ABAP-based applications, this data is contained in the statistics records that the SAP kernel captures automatically for every task handled by a work process. Using this information, you can identify critical steps in an application, determine which component is the bottleneck, and define the best approach for optimization.

The Performance Monitor (transaction STATS) is a tool for selecting, displaying, and analyzing statistics records. It helps developers and operators find the best balance between fast response times, large data throughput, high concurrency, and low hardware cost. The tool is easy and convenient to use and comes with a feature-rich user interface framework so that you can focus on the data and its interpretation.

Outlook

As a developer, and in the interest of your application's users, you are primarily interested in short end-to-end response times, but STATS focusses very much on server Response Time. Especially for modern browser-based or mobile applications, front-end and network times dominate the total duration of many user interaction steps, but their contributions are not contained in the statistics records.

Additionally, one of your interactions with the application may trigger multiple HTTP round trips into the ABAP back end. All associated statistics records must be considered for a complete account of the resource consumption, but their attribution is not always straightforward.

To fill these gaps, SAP's UI frameworks (especially SAPUI5 underlying SAP Fiori applications) collect relevant data on the front end and subsequently send it in front-end statistics records to the ABAP back end. There these front-end statistics records can be combined with the corresponding main statistics records.

I will describe the front-end statistics records in a sequel to this blog post. I will also introduce and explain transaction STATS_FE as the tool to display and analyze them.

References

Parts of this blog post are taken from my SAPinsider article on transaction STATS:
Identifying Performance Problems in ABAP Applications

Details on the collection and interpretation of statistics records are explained in SAP Notes:
579462 - Runtime parameter of the statistics collection
8963 - Definition of SAP response time/processing time/CPU time
1063061 - Information about response time in STAD/ST03
364625 - Interpretation of response time in 4.6
552845 - FAQ: RFC Statistics in Transactions ST03/ST03N and STAD

STATS is the successor of transaction STAD. As both tools work with the very same statistics records, the following two blog posts are still relevant:
Statistical Records Part 1: Inside STAD
Statistical Records Part 2: RFC Statistics

Use Transaction STATS_FE to Analyze the Performance of your Web Application

2022-01-27T11:40:54+01:00

As a business application developer, you need to ensure that your apps run with optimal performance. While this holds for all types of applications, in this blog post I am specifically looking at web apps, and in particular I am focusing on SAP S/4HANA applications using web-based access.

Your app's users demand short end-to-end response times to work efficiently. For each user interaction of your web application, you can break down its end-to-end response time into three main contributors:

Front end (web browser or native mobile)

Network

Back end (for SAP S/4HANA web applications this is further subdivided into the ABAP business logic and the SAP HANA database persistency)

You need reliable measurements on these components to assess a dialog step's duration. (Some user interactions may be handled exclusively on the front end. These front-end-only steps do not have any contributions from network and back end.) Most user interactions in Online Transaction Processing (OLTP) applications have their end-to-end response times dominated by the front end plus the network contributions.

Another relevant aspect of your web application's performance is its back-end resource consumption. Here the challenge is that one user interaction triggers an unpredictable number of HTTP roundtrips into the back end. They must be identified and aggregated to get dependable data on consumed hardware resources like CPU time or main memory.

In my previous blog post Measure your Application’s Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS I have described the statistics records collected by the ABAP kernel and I have explained how you use my transaction STATS to display and interpret the relevant statistics records. I have also mentioned some limitations of the statistics records, especially that they focus on server response time and that they correspond to individual tasks executed in the ABAP back end.

In this blog post I explain how front-end statistics records fill these gaps and how my transaction STATS_FE gives you a complete end-to-end picture of the performance and scalability of your web application. You can use STATS_FE since SAP NetWeaver 7.56. I am continuously improving it and this blog post presents the tool’s state for SAP NetWeaver 7.57 as of late January 2022.

What are Front-End Statistics Records?

SAP's most important UI frameworks (SAP GUI for HTML, SAPUI5, and Web Dynpro ABAP) have been equipped to take performance-related measurements on front-end devices. Client-specific performance information is collected per user interaction. Even some---but not all---of the previously mentioned front-end-only steps (user interactions without associated HTTP requests) trigger the logging of front-end statistics. At the end of a user interaction the gathered data is assembled into a front-end statistics records which will subsequently be sent into the ABAP back end.

The available meta data

identify the application and the user interaction step,

indicate the front end's operating system, and the HTML browser with its version,

specify the UI framework,

establish the relation with the back-end statistics records that correspond to the HTTP requests.
(In my blog post on STATS, they were simply called statistics records. From now on I will use the more precise terms back-end or main statistics record.)

Time durations are available for

end-to-end response time
difference between the start and end times of the user interaction, as determined by the UI framework

client navigation time
sum of all HTTP request’s navigation times, corrected for overlaps

client round trip time
sum of all client-side HTTP request durations, corrected for overlaps

client processing time
calculated as end-to-end response time – client navigation time – client round trip time

back-end request time
sum of all client-side HTTP request durations, measured on the front end

server response time
sum of all server-side HTTP request durations, measured on the back end and sent to the front end with the HTTP response

network time
sum of all differences (client-side HTTP request durations – server-side HTTP request durations)

KPIs related to data transfer include

number of HTTP requests

amount of data sent

amount of data received

a flag to indicate whether HTTP requests were compressed

As I have stressed in my previous post the "value of your performance analysis depends on the quality of the underlying measurements. While the collection of data into the statistics records is performed autonomously by the SAP kernel, you need some preparation to ensure that the captured information accurately reflects the performance of the application." This continues to hold, and you should take the same preparatory steps as mentioned there.

How Do You Analyze Front-End Statistics Records?

To view the front-end statistics records related to an execution of your web application you run SAP GUI transaction STATS_FE in the ABAP system that acts as the back end for your web application's execution. On the start screen (Fig. 1) you specify criteria for the subset of front-end statistics records you want to analyze. The filter criteria are subdivided into four areas.

Figure 1: On the STATS_FE start screen, you define filter conditions for the subset of front-end statistics records you want to analyze, specify from where the records are retrieved, and select the layout of the data display.

You determine the Monitoring Interval in the topmost area. By default, it extends 10 minutes into the past and 1 minute into the future. Records written during this period are displayed if they fulfill the conditions specified in the other areas of the start screen. You want STATS_FE to show as few unrelated records as possible. Therefore, you adjust this interval based on the start and end times of your application's execution.

You define additional criteria that the records must meet in the Record Filter area. Be as specific and restrictive as possible, so that only records relevant for your investigation will be displayed. Particularly helpful are client and user. If you want to focus on expensive user interactions in your web application, you can specify lower thresholds for measurement data, such as end-to-end response time or number of network roundtrips. The other conditions support advanced analyses and comparisons.

By default, statistics records are read from all application server instances of the system. In the Configuration section, you can change this to the local instance, or to any subset of instances within the current system. Restricting the statistics records retrieval to the instance (or instances) where the application was executed shortens the runtime of STATS_FE. The option Include Statistics Records from Memory is selected by default, so that STATS_FE will also process records that have not yet been flushed from the memory buffer into the file system.

Under Display Layout, select how the associated subset of key performance indicators (KPIs)—the captured data—will be arranged in the tabular display of front-end statistics records. The pre-selected layout shows all KPIs from the front-end statistics records and the most important aggregated KPIs of the associated back-end statistics records.

What do You Learn from Front-End Statistics Records?

Fig. 2 shows the list of front-end statistics records based on the settings specified in the STATS_FE start screen (Fig. 1).

Figure 2: Front-end statistics records matching the criteria specified on the STATS_FE start screen (Fig. 1). The list shows front-end statistics records that correspond to the creation of a sales order with SAP S/4HANA application F3893.

The table displays the selected front-end statistics records in chronological order and contains their main KPIs. The header columns—shown with a blue background—uniquely link each record to the corresponding user interaction. The data columns contain the KPIs that indicate the performance of the user interactions. These columns are subdivided into two groups: Those with a gray background show the performance-related measurements taken on the front end. They are the true front-end statistics records. The columns with the yellow background cover the most important aggregated KPIs of the associated back-end statistics records.
Measurements for times and durations are given in milliseconds (ms). Memory consumptions and data transfers are measured in kilobytes (KB).

The table of statistics records is displayed within an ALV grid control and inherits all functions of this SAP GUI tool. To access them, expand the toolbar by clicking on the Show Standard ALV Functions button (the black triangle that points to the right).

The End-to-End Response Time is split into Client Navigation Time, Client Round Trip Time, and Client Processing Time. This breakdown tells you if a slow user interaction suffers from technical problems to start the first HTTP request, in which case Client Navigation Time would be high. A large Client Processing Time typically indicates that the interpretation or rendering of the HTTP requests takes too long. Slow HTTP requests show up as high Client Round Trip Time. This KPI takes into account if HTTP requests overlap. The related KPI Server Request Time does not apply such a correction, but simply adds the individual request's durations. HTTP requests can be slow because their processing in the back end takes too much time, or because the network communication is too slow. This latter case is indicated by a high Total Network Time.
For most dialog steps in SAP S/4HANA applications, users expect an End-to-End Response Time below 1 s. To reliably achieve this, Client Navigation Time plus Client Processing Time shall be below 300 ms, and Total Network Time in a wide area network must not exceed 350 ms. The remaining 350 ms are available for back-end processing.

Further data on the communication between front end and back end, are the Network Roundtrips, and the transferred amount of data, split into Sent Data and Received Data. The Compressed flag is blank if there is at least one uncompressed response body with size above 1 KB.
To achieve an End-to-End Response Time less than 1 s a user interaction must not trigger more than two non-overlapping synchronous roundtrips, and ideally does not transfer more than 10 kB of data.

For each front-end statistics record, STATS_FE finds the associated main statistics records and aggregates their most important KPIs: Durations are summed; for memory consumptions, the maximum is determined. The values (together with the number of detected main records in column Count of Main Records) are displayed with a yellow background. With these measurements you can assess the user interaction's overall resource consumption in the back end.
As a rule of thumb, a simple dialog step (expected End-to-End Response Time below 1 s) shall use less than 350 ms ABAP CPU Time and less than 150 ms HANA CPU Time.

You might expect that Network Roundtrips always equals Count of Main Records, but the front-end statistics may include requests that have been handled by the browser cache, or requests that were sent to other back ends than the one where you run STATS_FE, or requests that serve generic purposes not directly relevant to the application's business objective. On the other hand, the back end may internally trigger tasks (e.g., RFCs, UPDATEs) resulting in separate main statistics records. Therefore, deviations in both directions are possible. Front-end statistics for which Network Roundtrips equals zero indicate front-end-only steps.

Based on this list of front-end statistics records you identify and focus on user interactions that do not meet expectations---either because they are too slow or because they require too many resources.

Figure 3: Main statistics records associated with the seventh front-end statistics record from Fig. 2.

Double-clicking an entry in the list of front-end statistics records takes you to transaction STATS and there displays the main statistics records associated with the user interaction (Fig. 3). You analyze this set of back-end statistics records as describe in my STATS blog post to understand the back end performance and resource consumption of all activities that were triggered by the user interaction represented by the chosen front-end statistics record and that were handled by the ABAP system where you currently work with transactions STATS_FE and STATS.

Conclusions

Front-end statistics records contain performance-related data for user interactions on front-end devices. They are collected by SAP's most frequently used UI frameworks and then sent into the ABAP back end. There you can use the Front End Performance Monitor (transaction STATS_FE) to select, display, and analyze the front-end statistics records related to your application. As developer or operator, you use this tool to assess the contributions to the end-to-end response time of user interactions in web applications whose performance you want to ensure. The integration of the main statistics records and the coupling with transaction STATS enables the simultaneous evaluation of the resource consumption in the back end.

STATS_FE supports a holistic end-to-end view on the performance, resource consumption, and scalability of web applications. Starting with the set of front-end statistics records of a complete business process, you obtain the overall resource consumption, and you identify the critical user interactions. Focusing on them you understand which component (front end, network, or back end) is the bottle neck. If you conclude that this is the back end, a drill-down into the associated main statistics records will get you even closer to the root cause.

Disclaimer

The statistics records (front-end as well as back-end) constitute measurements and therefore can have neither absolute precision nor absolute accuracy. To improve reliability and confidence in the data, reduce the impact of variability with an increased sample size (i.e., take averages of multiple measurements), and detect conceivable bias by comparing with data from other tools.

References

You find an introduction to statistics records and transaction STATS in my blog post Measure your Application’s Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS.

I have published an SAPinsider article on transaction STATS:
Identifying Performance Problems in ABAP Applications

ST05: Basic Use

2023-02-24T12:24:31+01:00

This is the first blog post in a series of posts that cover my SAP GUI transaction ST05. Here, I show the straightforward use of the tool to get you started for ad hoc performance analyses of your business applications.

If you have not yet done so, please read the introductory post Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources. There, I explain how my transactions STATS and STATS_FE help you to detect performance bugs in your applications. That post also introduces ST05 as an essential analysis tool to find root causes for applications that are too slow or consume too many resources. In particular, it captures and logs communications and data transfers between the ABAP work process and external components. By default, these transmissions are not recorded to avoid the associated overhead.

You start the Performance Trace (sometimes also called Performance Analysis) with transaction code ST05. On its start screen, in area Select Trace Type (Fig. 1), mark the options that correspond to the types of communication you need to analyze. This choice shall correspond to the critical interfaces identified by previous measurements of the application under investigation, e.g., with transactions STATS or STATS_FE.

Figure 1: In area Select Trace Type on the ST05 start screen, you mark the relevant trace types. Determine whether the ABAP call hierarchy shall be recorded in section Configure Trace. Then switch on tracing on the local application server instance by clicking Activate Trace. The Trace State panel displays the current settings.

ST05 supports seven communication interfaces across which the ABAP work process can exchange information with external components. They are listed in Table 1.

**Table 1:** Trace types covered by `ST05`.
SQL Buffer Enqueue RFC HTTP APC AMC	DB accesses calls to buffered objects handled by the application server’s buffers set or release SAP business locks (enqueues) remote function calls HTTP(S) communication WebSocket communication message exchange via publish/subscribe channels

The most common trace type (pre-selected by default) is the SQL Trace. It addresses communication with the database server. A more appropriate name would be DB Trace, because only SQL statements sent to the database will be recorded. To capture SQL statements handled by the table buffer in the ABAP application server instance you must mark option Buffer Trace.
In the Configure Trace area you can decide whether relevant communication activities shall be recorded together with their triggering ABAP call hierarchy (Stack Trace = On). During subsequent trace analysis, this is especially helpful for events encapsulated within and called via generic technical frameworks. As a developer of a specific business application's coding you can do little to improve the immediate invocation of the event, but you are obliged to optimize your call to the framework. This call's source code is accessible by the ABAP call stack. If you set Stack Trace = Off (default in customer systems), only the immediate caller will be recorded.

To start tracing, click button Activate Trace. This switches on the trace recording only on the local ABAP application server instance, and exclusively for the current user. Activities on other instances or by other users are not traced. Area Trace State indicates the date and time of the last change, and the name of the user who did the change (Fig. 2). It also shows the active trace types and the name of the user whose activities are traced. (With the current method to activate tracing, this is your user's name.)
If recording is already active on the instance, you cannot start another recording. You must wait until the other user ends the recording.

Figure 2: On the ST05 start screen, the Trace State area indicates the current situation on the local application server instance. When the process under investigation has finished, Deactivate Trace. Then click Display Trace to work on the records.

Execute the process in the normal way. You must ensure that it runs for the same back-end user who activated tracing, and that it gets executed on the application server instance where you have switched on the recording.

Once the application has completed, return to the ST05 start screen (Fig. 2). Switch off the recording by clicking Deactivate Trace. To start the analysis, click button Display Trace. This takes you to the Filter Conditions for Trace Records (Fig. 3). There you can specify restrictions on the subset of captured trace records to be examined. The appropriate filter conditions are pre-populated as much as possible with values that correspond to the previous activate - deactivate cycle. This applies in particular to the Trace Types, and to the Trace Period, which starts at the time of the last trace activation and ends at the time of the last trace deactivation.

Figure 3: Define filter conditions for trace records to be displayed and analyzed. Entries in areas Trace Types and Trace Period are prefilled with data matching the previous trace recording. With entries in section Trace Filter, you can concentrate on a subset of the trace with specific properties.

In the Configuration area, you may overrule the default to work on trace records only from the Local application server instance. Alternatives are all Active instances, or an Individual subset of all application server instances. (Recall that the click on Activate Trace records only on the local instance.) Due to load balancing, after a new logon you might be working on another application server instance. If you know the instance where your trace was recorded, choose it from the Instance Selection popup that comes up after clicking Individual.
The option Max. Number of Records guards against traces with an unexpectedly large number of records. A popup will indicate when the given number of records has been read and processed. You can then decide whether you want to continue or abort. The default value is 5000 records. The system will remember your most recent value. You may also set your personal default with user profile parameter ST05_MAX_REC_NUMBER.

In area Trace Filter you may set restrictions (Tab. 2) to focus your attention on particular details. This is helpful in exceptional cases, but rarely used. Ensure that you do not suppress important information by applying a filter that is too restrictive. Also be aware that it might interfere with displaying the ABAP call stack.

**Table 2:** Filter conditions for the display of trace records.
Trace Types	Select the trace types you want to analyze.
Trace Period	After recording a trace, the default time interval extends from the time when the trace recording was started until it was ended. For previously recorded traces, specify the time interval manually.
Client	Client where the traced activity was executed.
User Name	Name of the user who executed the traced activity.
Object Name	Name of the object that was accessed. (SQL, BUF: table or view; ENQ: lock object; RFC: function; HTTP, APC: path; AMC: channel ID)
Duration [Microsecs]	Elapsed execution time of the operation in microseconds (µs).
Executed Operation	Operation that was executed.
Database Connection Name	Database connection that was used to access the data.
Database Connection ID	Database connection ID used to process the statement.
Work Process Type	Type of the work process that executed the traced activity.
Work Process ID	Number of the work process that executed the traced activity.
Transaction ID	GUID that designates an SAP LUW.
EPP Root Context ID	The extended passport (EPP) correlates requests in complex business processes throughout the system landscape. The root context ID identifies the initial context.
EPP Connection ID	The connection ID distinguishes involved remote destinations.
EPP Connection Counter	The connection counter enumerates multiple calls that reuse a connection.

When you execute, the available trace records that meet the filter conditions are retrieved, handled, and displayed in an ALV grid control. The standard trace display (Fig. 4) is a chronologically sorted list of main records, each corresponding to one execution of an ABAP statement. (The processing of the statements in your application’s ABAP source code may involve several contributing operations that are captured as individual, detailed records in a trace. For the list of Trace Main Records, ST05 attempts to aggregate all these detailed records so that ideally there is a one-to-one correspondence between list entries and implemented ABAP statements. Individual records that cannot be aggregated are collected into dummy main records with the Statement set to OTHER (<trace_type>) and empty Start Time.) All types of trace records are shown in one common table, using only those fields that are relevant for the respective record's type, and leaving the others empty.

Figure 4: In the ALV list of Trace Main Records, each entry reflects one execution of an ABAP statement calling an external resource, or the processing of an incoming request.

Table 3 documents the fields that are visible by default.

**Table 3:** Fields in the default layout of the list of trace main records.
Start Time	Start time of the event in the user’s time zone.
Duration	Elapsed execution time of the event in microseconds (µs). (Measured by the ABAP work process.)
CPU Time	CPU time in microseconds (µs) used during the statement's execution. (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Memory	Maximum memory consumption in kilobytes (kB) during the statement's execution. (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Records	Number of records affected by the event. (SQL, BUF: number of table or view rows; ENQ: number of lock granules) (Not for RFC, HTTP, APC, and AMC trace records.)
Program Name	Name of the ABAP program that called the event.
Object Name	Name of the object that was accessed. (SQL, BUF: table or view; ENQ: lock object; RFC: function; HTTP, APC: path; AMC: channel ID)
Statement	Edited statement. (SQL: field list and FROM clause removed, variable names replaced by values)
DB Cursor	Number of the cursor used for the operation. (Only for SQL trace records.)
SQL Statement Hash	Hash value of the SQL statement. (Only for SQL trace records on SAP HANA.)
Array	Maximum number of records that can be transferred from the database to the ABAP work process within one communication package. (Only for SQL trace records.)
Connection Name	Name of the logical database connection (default = R/3). (Only for SQL trace records.)
User Name	Name of the user who executed the event.
Client	Client where the event was executed.
WP Type	Type of the work process that executed the event.

Every trace record contains the abbreviated statement, focussing on the parts with an impact on performance, and the object accessed by the statement. Also available is the calling program, and some details on how the statement was processed. The Duration (in microseconds; μs) or the number of affected table rows (Records) are likely the most relevant values for judging performance. If your system is based on the SAP HANA database, SQL trace records include measurements for the CPU Time and for the Memory Consumption on the database server, and identify the statement with its SQL Statement Hash.

The second header line at the top of the list summarizes the visible trace records by showing totals for Duration, CPU Time, and Records, and the maximum for Memory Consumption. Note that the total Duration is not necessarily related to the application's elapsed time. In the example of Fig. 4, the incoming HTTP request triggered SQL statements which were executed during the HTTP request's duration. Thus, the total Duration may overestimate the elapsed time.

Further fields (listed in Table 4) may be shown by customizing the ALV grid control's layout.

**Table 4:** Further fields for the list of trace main records, available on demand by changing the ALV grid control's default layout.
Start Date	Start date of the event in the user’s time zone.
Instance	Application server instance where the event was executed.
Processing Time	Elapsed execution time of the event in microseconds (µs). (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Transaction Code	Name of the transaction that triggered the traced activity.
DB connection ID	ID of the logical database connection. (Only for SQL trace records.)
WP ID	Number of the work process that executed the traced activity.
Transaction ID	The transaction GUID designates an SAP LUW.
EPP Root Context ID	The extended passport (EPP) correlates requests in complex business processes throughout the system landscape. The root context ID identifies the initial context.
EPP Connection ID	The connection ID distinguishes involved remote destinations.
EPP Connection Counter	The connection counter enumerates multiple calls that reuse a connection.

The list of main records helps you to judge the choreography of executed statements. In particular, it answers the question: Which event was executed when and where? This may allow you to conclude that statements are executed although they do not contribute anything to your application's business logic. These superfluous statements slow down your application, and at the same time waste precious resources. You must eliminate them. In my follow-up blog post ST05: Aggregate Trace Records, I present an easy and reliable method to find these unnecessary statements in a trace and in your application's ABAP source code.
For the remaining essential events, you focus on the statements that have a long Duration, affect many Records, or consume a lot of HANA hardware resources (CPU Time, Memory). They offer the largest optimization potential. The sequel blog post quoted above also discusses how to identify statements that would benefit the most from tuning. In my most recent blog post ST05: Analyze Individual Trace Records I describe their in-depth analysis.

Summary

Getting started with ST05 is really easy and takes only a few clicks. Just start the tool, and click button Activate Trace. Then execute the application you want to analyze. When it has finished, return to ST05 and click button Deactivate Trace. Finally, click Display Trace and execute to evaluate the trace records. The analysis of the trace records will discover hot spots and bottlenecks of your application related to its communications with external resources. The most important caveat is that all steps have to be done on the same application server instance and by the same back-end user.

Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources

2023-02-24T12:27:16+01:00

In my previous blog posts Measure your Application's Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS and Use Transaction STATS_FE to Analyze the Performance of your Web Application, I have put you as a software developer in charge of your business applications' performance. Here, performance encompasses end-to-end response time and scalability, which is impacted by your application's resource consumption. Users expect short response times and high business data throughput. To meet these non-functional requirements, you need tools to find root causes for poor performance.

With my transaction STATS_FE, you break down your applications' end-to-end response times into three main contributors: front end, network, and back end. The ABAP back-end portion is covered by the ABAP kernel's main statistics records. You display and evaluate them with my transaction STATS. There, you might discover that an individual task in the back end is too slow or requires too many resources. Further, you might conclude that this is due to communication of the ABAP work process executing your applications' ABAP code with external resources - most prominently the database, but also remote servers accessed via RFC or HTTP calls, or other entities engaged through appropriate interfaces. Beyond this conclusion the back-end statistics records cannot be very helpful. They are a monitoring mechanism and a measurement tool, thus do not contain sufficient detail to identify or analyze root causes.

To pin down bottlenecks and to derive efficient optimization approaches, you need a dedicated analysis tool that captures and logs the communication across the boundaries of the ABAP work process. This is the purpose of the Performance Trace (SAP GUI transaction ST05), which not only registers the communication with external entities via the relevant interfaces, but also enriches the corresponding records with measurement values (e.g., duration, resource consumption, amount of transferred data). ST05 is the ABAP stack's interface trace tool. It captures events during application execution that leave a work process to request data or services from external resources, or that enter a work process to perform services for external callers. The set of the ensuing records is a trace of your applications' outgoing or incoming data transmissions.

ST05 is a tool for the analysis of individual applications and their single user runtime behaviour. It investigates applications with respect to the resources they request from outside of the work process, and is also applicable to services provided for remote clients. ST05 is not suitable for load or mass tests.

The information obtained from ST05 helps you to fulfill the expectations of your software's users or operators. It identifies hot spots in an application that may be due to ...

… expensive SQL statements

… unnecessary accesses to the persistence layer

… long running SAP locks (enqueues)

… RFC or HTTP issues

In contrast to the statistics records, which are always and autonomously recorded by the ABAP kernel, the ABAP work process' communications are traced only on demand. Otherwise the associated overhead may slow down applications in productive use too much.

As I have already emphasized in my blog post Measure your Application's Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS, the value of your performance analysis depends on the quality of the underlying measurements. You need some preparation to ensure that the captured information accurately and reproducibly reflects the performance of the application. Only then can you derive reliable conclusions.

Confirm that the test system is configured and customized correctly.

Verify that the test system is not under high load from processes running concurrently with your analyses.

Carefully define the business scenario to be tested by your application. It must adequately represent the application’s behavior in production.

Provide a set of test data that is representative of productive data. Only then will the scenario execution resemble everyday use.

Execute the scenario a few times to fill the buffers and caches of all involved components (e.g., DB cache, application server’s table buffer, web browser cache). Without these pre-runs or warm-up runs, your recorded traces will not be reproducible, but will capture one-time or first-use effects like loading data into buffers and caches. This may lead you to incorrect diagnoses, or make it more difficult to identify the real issues. It is then much harder to draw reliable conclusions.

After this preparation, you activate the trace recording, then execute the application (this is the measurement run), and afterwards deactivate the trace recording. Next display the trace. The subsequent evaluation of the trace records will identify hot spots and bottlenecks of your application that are caused by the communications and associated data transmissions of the ABAP work process with external resources.

List of Blog Posts on `ST05`

In a sequence of individual posts, I cover the main features of ST05, explain the necessary technical background, and share best practices for efficiently working with the tool.

Basic Use
How do you work with the tool?

Technical Background of Trace Recording and Analysis
What is the technology behind ST05 and how does it affect its behaviour?

Trace Directory
How do you save and recover traces?

Advanced Features of ST05
1. Activate Trace Recording with Filter
  How can you control the trace recording?
2. Aggregate Trace Records
  How do you aggregate related trace records to get a better overview?
3. Analyze Individual Trace Records
  What can you learn about the processing of communication steps?

Examples shown in these blog posts were recorded while running the transactional S/4HANA Fiori application F0997 (Audit Journal) from the Finance line of business in an internal test system. The test case displayed the changes for Company Code 1010 and Fiscal Year 2023.

Transaction ST05 is part of SAP_BASIS since decades. I am continuously improving it and in this series of blog posts I am presenting the tool's state for SAP NetWeaver 7.58 as of early 2023. Depending on your system's release, some of the features may not yet be available.

References

I have published several articles in SAPinsider which are relevant to measuring and analyzing the performance of ABAP applications:

Manfred Mensch:
Identifying Performance Problems in ABAP Applications: Analyze Statistics Records Using the Performance Monitor (Transaction STATS)

Manfred Mensch:
Track Down the Root of Performance Problems with Transaction ST05

Manfred Mensch:
Pinpoint Performance Bottlenecks Using ABAP Profiling: Analyze Your Application with Transaction SAT

The copyright to these three articles is with SAPinsider and they appear here with permission from the publisher.

There are two previous blog posts related to ST05, based on an older SAP_BASIS release, but still useful:

Siegfried Boes:
The SQL Trace (ST05) Quick and Easy

Randolf Eilenberger:
Code Inspector Integration into SQL Trace of ST05

Refer to my blog posts on transactions STATS and STATS_FE for measuring your application's end-to-end response time and resource consumption:

Manfred Mensch:
Use Transaction STATS_FE to Analyze the Performance of your Web Application

Manfred Mensch:
Measure your Application's Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS

ST05: Activate Trace Recording with Filter

2023-03-08T10:22:35+01:00

This is the second blog post in my series of posts about SAP GUI transaction ST05. Here, I explain that you can control and customize the trace recording with filter conditions, and how this is useful for analyzing your applications’ performance. With a small case study I show this feature in action.

In the introduction to the sequence of blog posts (Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources) I have referred to my transactions STATS and STATS_FE as measurement tools, which help you to find performance hot spots in your applications. To identify root causes of poor performance (long response time or high resource consumption), you need dedicated analysis tools. With respect to communications and data transfers between the ABAP work process and external components, ST05 is the analysis tool of choice. I have described the straightforward way of working with it in my previous post ST05: Basic Use. Be sure to read it for a quick review.

The simplest way to switch on tracing involves the selection of the desired trace types, followed by a click on Activate Trace. Then trace recording is active only on the local ABAP application server instance and exclusively for the current user. Activities on other instances or by other users are not captured.
Now imagine that you want to support co-workers with the performance analysis of their business application. Maybe they do not have authorization to run ST05, while you are not allowed to work with the application. Or you do not know ahead of time which of the system's application server instances will handle the back-end portion of the application—the system’s load balancer may dispatch incoming requests to any instance. Or you need to investigate a batch job that runs at night time, when you do not want to manually switch on or off the trace recording. Furthermore, within a trace type, i.e., for a particular communication interface of the ABAP work process, you might be interested only in a subset of all associated communication events. All these situations are not covered by the ordinary Activate Trace. This is where the option to Activate Trace with Filter (Fig. 1) comes in.
Figure 1: Clicking button Activate Trace with Filter opens a popup with Conditions for Trace Recording. There you can override some of ST05’s defaults, or specify restrictions on the communication events to be captured. Scheduling the trace recording is convenient for the analysis of batch jobs that run while you are not in the office.
The running text explains all options in detail.

With an entry in field User Name you can activate tracing for another user. Then her or his activities are recorded instead of yours. User name patterns with the asterisk * as a terminating wildcard character are supported.

To trace events originating from or handled by a non-local application server instance within the current system, expand the dropdown list box for the Instance field to see all active instances. The local instance is explicitly indicated. If you know the relevant instance, select it. Otherwise, to activate tracing on all instances, choose <ALL>. This tries to switch on tracing, but if it is already active for another user on an instance, the attempt fails for this instance. A popup will inform about this failure. The brute force option <ENFORCE> unconditionally terminates any tracing in progress on all instances and then activates trace recording per your conditions. Use this option sparingly and carefully. It is convenient and safe for you, but may be disturbing and annoying for your colleagues.

With the group of fields Transaction Name, Program Name, RFC Function Name, or URL, you can limit the captured events to those that are triggered while executing a particular application. Note that this application must be at the very top of the call hierarchy. Therefore, a value can be given for at most one of the fields. Patterns with the asterisk * as a terminating wildcard are allowed.

Occasionally, an application (most often a batch job) may occupy a work process for a very long time. With transaction SM50 you can determine the work process’ number and then enter it into the Process Number field of the filter conditions popup. Thereby, you confine the trace recording to events related to the given ABAP work process. The resulting trace may identify the reason for the long duration. There is one caveat: A long running database operation that started before the trace was switched on will not trigger the creation of an associated trace record.
(To the best of my knowledge, the value of this filter criterion is rather limited, and it is not used very often. Please add a comment to this blog post if you know of an important use case.)

For an SQL trace, you can restrict the recording of SQL statements to those that work on certain database tables (or database views, including CDS views). You may define up to 5 tables to be included. Similarly, you can exclude up to 5 database tables. Statements accessing these excluded tables will then not be recorded. For both options, you may filter on groups of tables by terminating entries in columns Include or Exclude with an asterisk *. For implemented Open SQL JOINs, the table name filter applies to the first of the JOINed database tables. If a database table matches entries in both columns, Include and Exclude, SQL statements against this table will not be recorded.
If you specify a table in the Include column, but also want to trace all statements that do not directly affect an individual table (e.g., COMMITs, ROLLBACKs, SET SESSION VARIABLE, SET CLIENT INFO), you must mark the check box Include Statements with Empty Table Names.
You may identify tables to be included with transaction STATS(cf. my blog post Measure your Application’s Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS): The Top N Table Accesses sub record in the DB area shows the top N most expensive tables. You may also decide to focus on a table with central importance for your line of business, so that you get an overview and deeper understanding of where or how this table is used. Based on this knowledge, you can derive approaches for optimizing the accesses to this table.
Excluding certain tables from trace recording may be helpful if your application triggers many statements against these tables and you have already verified and confirmed that they are all necessary and processed in the best possible way. Then the corresponding trace records would only clutter up your trace, without adding any useful information. Even worse, they increase the risk that the available trace files get filled up to their capacity and then are successively overwritten, thereby evicting potentially important records. (Refer to my blog post ST05: Technical Background of Trace Recording and Analysis for a detailed description and explanation of the underlying mechanism.)
Make sure that you do not by mistake exclude a table that your application accesses via potentially expensive statements.

Imagine that you need to investigate the communications triggered by a batch job. You know its scheduled start time and have an idea about its expected duration. If the job starts or ends well outside of your work time, you can use ST05’s Scheduling capability. Mark the check box Schedule Trace Recording, specify the relevant Start Date and Start Time as well as End Date and End Time. This schedules and releases jobs named ST05_Trace_On and ST05_Trace_Off, which respectively activate and deactivate tracing on the appropriate dates and times. Start Date and Start Time should be slightly earlier than the beginning of the batch job you want to analyze, and End Date and End Time should be a little later than its projected end. Only then will the trace recording be active for the job's entire duration so that the resulting trace can cover all relevant events.
If you cannot analyze the trace shortly after it was recorded, other subsequent tracing activities may overwrite it from its beginning. To protect against this, mark the check box Save Trace in DB and provide a Description. This will add a second step to job ST05_Trace_Off, which processes all available trace records in the prescribed time frame and persists them with your description into the trace directory. There they will expire only after 120 days, giving you plenty of time for your evaluation. Please note that an active trace recording may overwrite its own beginning. The danger for this increases with the time during which tracing is switched on. Option Save Trace in DB cannot prevent this. It will nevertheless store the remaining trace records into the trace directory so that you can work at least with those that are still available, and do not lose even more records to other tracing activities. Of course, the previously explained approach to limit traced SQL statements to those that do or do not access specified database tables is very efficient in reducing the risk that a scheduled trace overwrites its own leading records.

Once you have defined your filter conditions, switch on trace recording by pressing the Enter key. The Trace State area on ST05’s main screen will then indicate that recording has been activated with filter. To review the current settings, select Display Filter for Trace Recording from the Edit menu (Fig. 2). If your only restriction affects the User Name, ST05 does not consider this a genuine filtered activation, but of course still respects it, and already displays it in the Trace State.
The User Name must not be changed while recording a trace. The Transaction Name, Program Name, RFC Function Name, or URL to be traced, as well as the list of tables to be Included or Excluded, can be modified while tracing.
If you Schedule Trace Recording, a message in the SAP GUI’s status bar indicates the result. On success the corresponding batch jobs show up in the job overview of transaction SM37.
Figure 2: Review the Conditions for Trace Recording, either in the Trace State area on ST05's main screen, or in full detail via a dedicated popup.

When you have switched on trace recording on all application server instances, make sure that you click button Deactivate Trace on All Instances to switch it off. Otherwise, it gets deactivated only on the local instance, but keeps going on all remote instances.

Example

Let's assume that we want to analyze the start of the SAP S/4HANA Fiori application F0997 (Audit Journal) with an automatically applied default filter on Company Code 1010, Fiscal Year 2023, and Ledger 0L. We start the app by selecting it from our favorites in the SAP S/4HANA Product Home Page, and then immediately go back after the list of journal entries has been displayed. We repeat this back and forth pattern several times to create a reliable set of statistics records.
We begin our investigation with transaction STATS_FE(Use Transaction STATS_FE to Analyze the Performance of your Web Application) to measure the end-to-end performance of this front-end user interaction and to understand its overall impact on the ABAP back end (Fig. 3). We identify one front-end statistics record as representative for the user's activity. Forward navigation by double click takes us from there to transaction STATS, which resolves the user interaction's total effect into individual back-end tasks and their corresponding main statistics records (Fig. 4). In our example scenario, we observe 8 ABAP tasks, triggered by incoming HTTP server requests. One of them clearly dominates the load on the back end, and this is obviously due to database requests . The corresponding Top N Table Accesses sub record in the main statistics record indicates that sequential reads from two database tables account for almost all the time.
Figure 3: Transaction STATS_FE displays the front-end statistics records corresponding to user interactions. Fields with a green background characterize the front-end activity. Fields with a gray background contain measurements taken on the browser by the UI framework. The fields with the yellow background (normally shown to the right, here copied into an overlay) show aggregated values of the associated back-end statistics records.
The first record corresponds to the initial logon to the SAP S/4HANA Product Home Page. Then, 11 pairs of records represent calls from there to the Audit Journal and back. Observe that the first call to the application is significantly slower, presumably because buffers and caches have not been filled. Also note that only multiple repeated executions allow the identification of a failed measurement (crossed out in the screen shot).
The red arrows mark the front-end statistics record analyzed in more detail (Fig. 4).

Figure 4: A double-click on the front-end statistics record marked by the red arrows in Fig. 3 shows the associated back-end statistics records in transaction STATS. This resolves the back-end load triggered by the user interaction into separate tasks. Analyze the critical tasks, i.e., those that are slow or consume many resources. The records' details provide deeper insights and suggest the next steps in a root cause analysis. These details are displayed in a popup by double-click on the back-end statistics record.

For this scenario, our investigation so far has used solely the monitoring transactions STATS_FE and STATS based on the statistics records, which are always available. As a result, we know that one of the five calls to OData service /sap/opu/odata/sap/FAC_AUDIT_JOURNAL_SRV dominates the performance, and that accesses to tables IFIACCDOCJRNL and IFICMPTJRNL are the decisive factors (Fig. 4). To come closer to the root cause, we will use this information, together with knowledge of the back-end user's name, to set Conditions for Trace Recording in ST05 (Fig. 5).

Figure 5: Conditions for Trace Recording for a focussed investigation of Fiori application F0997 (Audit Journal), deduced from the monitoring data contained in front-end and back-end statistics records (Fig. 3 and Fig. 4).

The resulting trace (Fig. 6) is very much to the point. It contains only the HTTP requests serving the business logic. Those that were triggered by any generic framework did not pass the filter. With respect to SQL statements, exactly those that fetch records from the specified CDS views IFIACCDOCJRNL and IFICMPTJRNL were recorded. From here, we can focus on understanding how and where the statements are implemented, how they are called via various involved frameworks, and most importantly, how the database server has processed them. I cover all this in my blog post ST05: Analyze Individual Trace Records.

Figure 6: Complete trace of the start of the Audit Journal Fiori app recorded with the filter conditions shown in Fig. 5. Without filter, the trace for the same user interaction would contain several hundred SQL trace records.

Summary

The option to Activate Trace with Filter significantly enhances the basic trace recording capability of ST05 to capture the communications and data transfers between the ABAP work process and external components. Without it many performance analysis projects would be much more complicated, and others would even be impossible.
The particular choice of filter conditions shall be guided by previous measurements of the application under analysis, e.g., with transactions STATS or STATS_FE based on the statistics records. They identify the critical areas, and with corresponding filter criteria you limit the trace recording to these parts. This facilitates your subsequent investigation.
Trace Scheduling is really convenient for the analysis of applications that run at known after-hours times.

ST05: Technical Background of Trace Recording and Analysis

2023-03-22T06:07:28+01:00

This blog post in my series of posts related to SAP GUI transaction ST05 (introduced in Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources) explains the tool's inner workings and configuration options, and how they affect its behavior. With this knowledge, you can avoid some pitfalls that otherwise would make your performance analyses less convenient or less reliable.
My blog post ST05: Basic Use describes the straightforward work with this analysis tool so that you can record your applications' communications with external resources across the boundaries of the ABAP work process. This current post does not teach you any new tricks, and you may safely ignore it at first—as long as you remember it exists when you feel that ST05 misbehaves. Then the tool's technical background as presented here does explain that ST05 actually works as designed.

Authorizations

Transaction ST05 is protected by authorization object S_ADMI_FCD, which checks accesses to various basis functions. It defines a single authorization field, also called S_ADMI_FCD, for these system administration functions. The relevant values are

ST0M Authorization to activate or deactivate trace recording

ST0R Authorization to analyze traces

There are no further checks. In particular, every user with the authorization to switch tracing on or off can deactivate a trace recording switched on by any other user. Similarly, every user with the authorization to analyze traces may see all available trace records, independent of who activated their recording, or who triggered the recorded event.

Trace Recording

ST05 really is just a display transaction. The only changes it can trigger are via buttons Activate Trace and Deactivate Trace (and the closely related buttons Activate Trace with Filter and Deactivate Trace on All Instances). These actions modify the so-called SWitches Block (SWB) in the Shared Common System Area (SCSA), a small shared memory segment accessed by SAP executables (kernel programs). The SWB content determines whether tracing is on or off, which trace types (i.e., communication interfaces) are to be recorded, and what filter conditions shall be applied. The actual capturing of the communication events according to these settings is done by the ABAP kernel.
Because the SCSA is shared across the entire application server instance, trace recording does not work concurrently, i.e., ST05 cannot capture communication events triggered by distinct users (or by distinct user groups defined with a trailing wildcard *).
With one SCSA per instance, tracing is instance specific so that at the same time it can be active for one user on one instance, and for another user on another instance (and inactive on other instances). This also demands that the application whose communication requests you want to capture runs on the instance where trace recording was switched on. Simple trace evaluation via the Display Trace button is also instance specific, i.e., only works with trace records previously captured on the same application server instance. Altogether, trace recording, application execution, and trace analysis must all be done on the same application server instance.
Trace recording is always cross-client. Active tracing captures communication events related to each and every client in the system.

When trace recording is active, the kernel saves trace records into dedicated (binary) files on the application server instance's file system. The base name of these files (including the path) is fixed by the value of profile parameter rstr/filename (e.g., /usr/sap/YI3/D34/log/TRACE). New trace records are always stored into a file by exactly this name. When it is filled to its capacity (as defined in megabytes by profile parameter rstr/max_filesize_MB; default value 16, maximum value 100), the file is renamed by appending a two-digit number between 00 and 99 to the base filename. If a file by that name already exists, it will be discarded. An empty file with the base filename is recreated, and trace recording into this newly created file continues without any gap. The maximum number of trace files (active plus overflow) is defined by profile parameter rstr/max_files (default value 10, maximum value 99). If this number of files is exceeded, the oldest file is deleted, and recreated to act as the youngest overflow file. As a consequence of this round-robin approach, old trace records are eventually lost, and a trace may overwrite its own beginning if recording is active for too long, or if the traced application triggers a large number of recorded events. Depending on the specified filter conditions for trace recording (cf. my blog post ST05: Activate Trace Recording with Filter), on the traced load on the application server instance, and on the configured number and size of trace files, this may happen within just a few minutes.

To minimize the impact of trace recording on the instance's performance, the records are not appended immediately and individually to the trace file, but first cached in an internal memory buffer within the ABAP work process. This buffer's size (in kilobytes) is given by profile parameter rstr/buffer_size_kB (default value 500). If the size of a single statement (including its variables’ values) exceeds this limit, it will be truncated.

The method used for trace recording, as described above, implies that on the technical level there are only individual trace records, stored one after the other. The concept of a trace as a set of records that belong together has no technical realization. Of course it is still a useful mental notion when you work with ST05 to analyze your application's communication events.

Based on benchmark runs, the average size of a trace record is less than 1kB and the overhead for server response time, as well as ABAP CPU time, is less than 3%.

Additional Trace Types

The SWB in the SCSA supports more trace types than are exposed by ST05. The corresponding interfaces deal with authorization checks done by your applications, or with low-level calls to the kernel's C functions or C modules. You can control and analyze these trace types in transaction ST01, together with the others, which are more frequently used and therefore accessible in ST05.
The inability to trace concurrently also extends to these ST01-only trace types. Thus, in rare cases your attempt to record a trace with ST05 may fail although the Trace State area on its start screen is empty. A popup will then explain the situation with an information message: At least one trace type was already activated for user <NN>. (This may be an ST01 trace.) The trace is not activated. You may want to discuss the situation with the given user.

Stack Trace

As I have described in my blog post ST05: Basic Use, in the Configure Trace area on ST05's start screen you can request that relevant communication activities shall be recorded together with their triggering ABAP call hierarchy (Stack Trace = On). With the setting Stack Trace = Off , which is the default in customer systems, only the immediate caller will be recorded.
The recording of the stack trace uses a delta strategy. Each trace record stores only those parts of the corresponding event's ABAP call stack that have not yet been captured in previous records. The correct and complete ABAP call stack is assembled only at display time, i.e., on demand. This approach minimizes the recording overhead for including the stack trace, and—equally important—the size of the trace records does not grow too much so that precious space in the trace files is saved and available for further trace records. This technique comes with a disadvantage: If you apply a filter to the trace records at display time, parts of the call hierarchy may not pass the filter. Then the ABAP call stack cannot be reassembled. ST05 detects this situation and indicates it with a warning popup: Cannot fully display ABAP call hierarchy for statement of selected trace record. To avoid this type of problem, do not filter records when you display a trace taken with Stack Trace = On. Prefer to set a filter already while tracing, which saves even more space in the trace files.

Capturing call stack information takes ~30µs per trace record, and increases the record size on average by ~50%. Thus trace recording with Stack Trace = On increases the danger that a trace overwrites its own beginning.

Profile Parameters

Table 1 summarizes the profile parameters that govern how the trace recording works.

**Table 1**: Profile parameters to control trace recording.
Name	Explanation	Default Value	Maximum Value
`rstr/accept_remote_trace`	Accept trace switches from remote client?	FALSE	N/A
`rstr/auth_delim_tab`	Use tabulator (instead of '&') as separator in authorization trace	TRUE	N/A
`rstr/buffer_size_kB`	Size of trace buffers (one buffer per work process)	500
`rstr/filename`	Base name of the trace files		N/A
`rstr/max_files`	Maximum number of trace files	10	99
`rstr/max_filesize_MB`	Maximum size of one trace file	20	100
`rstr/send_global_trace`	Send global trace switches (together with user-specific settings) to remote server?	FALSE	N/A

References

SAP note 12715 (Collective note: problems with SCSA) covers potential issues with the SCSA.

ST05: Trace Directory

2023-04-28T06:59:16+02:00

In this blog post of my series Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources, I explain how you can store traces of your applications' communication events as long as required. Beside giving you more time for in-depth analyses, stored traces pave the way to completely new types of investigations. Examples are:

Check the temporal evolution of the applications' use of external resources (before-after comparisons).

Investigate the impact of user input or data volume on the applications' performance or scalability.

Compare the applications' behavior between systems.

Share traces with co-workers who do not have access to the system where the application was executed.

This blog post assumes previously recorded traces. See the descriptions in my posts ST05: Basic Use and ST05: Activate Trace Recording with Filter to refresh your trace recording skills.

Some of your traces may be more important for you than other traces. But for ST05, they are all the same and subject to the file system storage mechanism that cycles through the configured number of trace files. Due to this round-robin retention approach, even your most important trace is eventually lost—and this can happen before you have completed your analysis of this trace. For a more complete explanation of the process, please refer to my blog post ST05: Technical Background of Trace Recording and Analysis. It covers the trace recording in great detail, and gives a comprehensive description of the underlying mechanism. The trace directory integrated into ST05 provides a solution by persisting traces into the database.

Save a Trace into the Trace Directory

Let's begin with the assumption that you have just recorded a trace. On the ST05 start screen, click button Display Trace to go to the Filter Conditions for Trace Records (Fig. 1, left panel). Specify criteria that support your intended analysis. Then, click button Save Trace in Database . This will immediately save the trace records without displaying them.
Alternatively and preferably, take a first quick look at the trace to convince yourself that it is complete and representative for your application's typical execution. Only then can it be a reliable foundation for detailed and conclusive investigations. Next, trigger Save Trace in Database via the appropriate button above the ALV grid control displaying the trace records (Fig. 1, right panel).

Figure 1: Saving trace records into the trace directory on the database can be done from the Filter Conditions for Trace Records (left panel) or from the list of Trace Main Records displayed in an ALV grid (right panel). In both cases, a popup (inset) asks for a Short Description of Trace to identify the resulting trace directory entry.

Either way, a popup prompts you to give a Short Description of Trace (Fig. 1, inset) so that later you can reliably find your trace in the trace directory.
When you continue from this popup, the display formats of the trace records (as created on the fly by ST05 from the raw data in the trace files on the application server instance's file system) are stored in the database. Any restrictions you may have defined for the trace records in the Filter Conditions for Trace Records or in the ALV's filter are respected: Only those trace records that match your selection criteria are stored in the database.

Traces that you save into the trace directory are given an Expiry Date, which by default is 120 days in the future. After this date they will be deleted automatically from the database. This mechanism prevents excessive growth of the underlying database table's size due to old and most likely outdated, therefore useless, traces.

Display the Trace Directory

To navigate to the trace directory, click button Display Trace Directory on ST05's start screen. This takes you to the Directory of Performance Traces (Fig. 2). In its two areas at the top, you can select from predefined filter conditions, or provide your own terms for the stored traces that you want to work with. The bottom panel shows the list of matching traces in an ALV grid control. By default, you see the traces you have created on the current day.

Figure 2: The Directory of Performance Traces is subdivided into three areas. In the upper two panels, Trace Selection and Time Period, you define conditions on the stored traces you want to list. The table in the lower panel then consists of the relevant trace directory entries. The icon at the start of each row distinguishes between entries that contain just metadata pointing to traces only available in the application server instance's file system (indicated by the icon depicting a binary file), and entries where the trace records are saved in the database (represented by the DB-like icon).
Here the table shows two sets of five traces each for consecutive calls to the Audit Journal Fiori application. For the older set, the metadata-only entries are already deleted because the trace records are no longer in the file system. The newer set still includes the metadata-only records next to their counterparts with the complete traces persisted in the database.

**Table 1:** Fields in the list of traces in the trace directory.
Client	Client where directory entry was created.
User	User who saved trace.
Instance Name	Application server where trace was created. (Always empty for traces stored in database.)
Start Date	Start date of trace period.
Start Time	Start time of trace period.
Expiry Date	Date after which trace will be deleted from directory. (Only used for traces stored in database.)
Description	Description of trace.

The characteristic fields of directory entries are listed in Table 1. An additional leading column in front of Client contains one of two icons to distinguish the type of trace directory entry. The kind described so far is identified by the DB-like icon , indicating that the entire trace with all its records is stored in the database. The other icon represents a binary file. It is assigned to trace directory entries that keep only metadata (trace types, trace period) in the database. This type of trace directory entry is initiated automatically whenever trace recording is activated (buttons Activate Trace or Activate Trace with Filter on ST05's start screen), and completed when you switch off tracing. The actual trace records are only in the application server instance's file system, where they are subject to the cyclic overwriting described in my blog post ST05: Technical Background of Trace Recording and Analysis. When the trace records are overwritten, ST05 automatically deletes the corresponding metadata trace directory entries. They are a convenience tool: Imagine that you record a trace, and then you have to logout of the system before you can analyze the trace. When you logon again, the load balancer may put you on another application server instance. Then these trace directory entries save you the effort to manually search for your trace on all of the system's application server instances. Instead, it is very easy to find it—provided it has not been overwritten in the meantime: Open the trace directory, find and double-click the entry. ST05 will go to the adequate application server instance and display the trace there. To enable this cross-instance navigation, these types of trace directory entries have the appropriate value in their Instance Name field. For traces completely stored in the database, this value is not necessary and always empty.
The precise semantics of Start Date and Start Time depend on the type of the trace directory entry. For those representing fully persisted traces, they are the values specified on the Filter Conditions for Trace Records screen. For the metadata-only entries, they correspond to the instant of trace activation.

In the Trace Selection area, the default option My Traces restricts the list of traces to those trace directory entries that you have created. Optionally, you may decide to see traces of any user, or you select traces based on their owner (User), their Description, or their Expiry Date (Deleted Before). The selection criteria for User and Description support patterns with the wildcard characters + (single character) or * (any character string). The Description is case sensitive. When you mark the checkbox All Clients, traces saved from every client in the system will be shown. Please note that the User is the one who has created the trace directory entry. This may be different from the user who has recorded the trace, or from the user whose activities were recorded. A similar remark applies to the Client field.
In the Time Period panel, you filter traces by their Start Date. The default is the current day (Today). Other restrictions are for traces from the current week (This Week) or by a Time Period. The option No Limit covers all traces, independent of their Start Date.
The conditions from the two panels are combined by logical AND.

Work with Saved Traces

For completely saved traces, the trace directory allows you to change their properties Expiry Date (if the default retention period of 120 days is too short) or Description. You can do this either via button Change Trace Property , or through the appropriate entries in the corresponding fields' context menus. The Description can also be set or changed for metadata-only directory entries, but because of their short lifetime this hardly ever makes sense.
When you do not need some previously stored traces any longer, mark them in the trace directory. Then manually delete them with button Delete Trace(s) . This saves space in the underlying database table and gives you a better overview of your still relevant traces.

If you have recorded separate but related traces, e.g., one for each individual user interaction of an application with multiple steps, and also want to analyze the entire application's communication with external resources, you can mark all corresponding traces and merge them into one comprehensive trace through button Merge Traces . Then analyze it as if it was recorded in one go.

To access the true content of a trace directory's entry (as opposed to its characteristics shown in the list and explained in Table 1), double click it. For persisted traces, this takes you immediately to the list of Trace Main Records as covered in my blog post ST05: Basic Use. Alternatively, you right-click an entry's row header and select Display Trace with Filter from the context menu or from button Display Trace to get the Filter Conditions for Trace Records screen where you can restrict to a subset of the stored trace records. This screen is always the entry point for metadata-only trace directory entries.

You may also export traces from the trace directory into the local file system of your front-end computer. Mark the desired trace and select the download format from the context menu of button Export Trace . Available formats are binary, or CSV for the various aggregation levels supported by ST05. Give a file name for each trace you want to export. You can initiate trace downloads also from the Filter Conditions for Trace Records screen or from the list of Trace Main Records.
CSV downloads are typically opened with a spreadsheet application. A binary export can be imported into a trace directory of another system (button Import Trace ). There, its description will be the file name with (Imported from file) appended. Based on such an imported trace, you can continue your analysis of the trace even when you do not have access to the original system any longer. Or you can forward the trace to a colleague who does not have a user in the original system. Finally, with imported traces you can compare the behavior of your application between different systems with distinct configurations, or software releases, or amounts of data. The next section explains the trace comparison in detail.

Trace records contain the name of the user who triggered the associated statement. This may conflict with local data privacy legislation. To remedy such collisions, you can anonymize persisted traces: Mark the relevant stored traces and select Anonymize Trace Records from the context menu (right-click the row header). In the ensuing popup, specify the user name to be anonymized. It will be replaced by a string of 12 alphanumeric characters. If you specify * all user names will be deleted. No other wildcards are supported.

Trace Comparison

For fully persisted traces, the trace directory supports their comparison: Mark exactly two traces, then click button Compare Two Traces (Fig. 3).

Figure 3: From the Directory of Performance Traces you can compare two traces that you have previously saved on the database. Fig. 4 shows the comparison.

This leads to the Trace Comparison of Structure-Identical Statements (Fig. 4). Its upper panel shows the two traces' metadata and overall totals for their KPIs Total Number of Executions, Number of Records, Execution Duration, HANA CPU Time, and Redundant Identical Statements. Additionally, it includes the prevalences of buffer types among the database tables or views recorded in the traces. The lower panel contains the actual comparison. The first trace is considered the baseline against which the second trace is contrasted. Press button Swap Traces to interchange the traces if you want to regard the other trace as the baseline.

Figure 4: With a trace comparison you can investigate variations in your application's runtime behavior related to its communication with external resources. These may be due to unequal amounts of data either in principle available to the application, or actually processed by it. Other reasons for runtime differences may be changes to the application's source code, or its configuration, or the way it is executed by its users.
The upper panel shows the two traces' metadata and their overall KPI totals for Total Number of Executions, Number of Records, Execution Duration, HANA CPU Time, and Redundant Identical Statements.
In the lower panel, the actual comparison examines the KPI's values aggregated according to the selected granularity. The default comparison method is the difference, and the rows are sorted descending by the runtime difference.
In the screenshot, the comparison contrasts two calls to the Audit Journal Fiori application F0997. The first one does not find any data, the second has 71 rows in its result set.

To prepare for the comparison, ST05 summarizes, separately within each trace, the values for Total Number of Executions, Number of Records, Execution Duration, HANA CPU Time, and Redundant Identical Statements of all matching records. The default summarization criterion is the object affected by the statements, independent of the type or the structure of the statements. For finer granularity, distinct operations (SELECT, INSERT, UPDATE, DELETE, ENQUEUE, DEQUEUE, ...) can be resolved, or (for greatest detail) the operations and the statements. After this preparation, the summary records of the two traces are matched (by the combination of object, operation, or statement, corresponding to your chosen level of resolution), and their values are compared. The default comparison (Fig. 4) uses the values' differences (absolute numbers). Optionally, ST05 can express this as the percentages relative to the values of the first trace. Other alternatives show the ratios of the second trace's values divided by those of the first trace, or the relative contributions to the corresponding total values of each trace.

The first column of each summary record in the comparison table encodes the record's occurrence:

T1_ : only in the first trace

T_2 : only in the second trace

T12 : in both traces

For a meaningful trace comparison, T12 shall be the predominant code. Otherwise, the two traces are not really comparable.
Columns Object Name, Operation, and Statement characterize the level of detail on which the comparison is done. For rows dealing with database tables or views, column Buffer Type indicates the buffering type.
The four color-coded groups with three columns each contain the compared KPIs with the measured values for the two traces and their differences.
To streamline the comparison, click button Display Differences Only . This compares only those records that are not in both traces (code either T1_ or T_2) or that have different values for Total Number of Executions or Number of Records. Press the button again to toggle back to Display Full List.
From the dropdown menu of button Set Comparison Method you can pick your preferred perspective: absolute difference, ratio, percentage increase, or percentage of each trace's totals.
Similarly, use the dropdown menu of button Set Comparison Granularity to determine the level of detail for your analysis: just Objects, or Objects and Operations, or Objects, Operations, and Statements.

The example shown in Fig. 4 indicates that the same number of HTTP requests to OData services /sap/opu/odata/sap/FAC_AUDIT_JOURNAL_SRV took significantly longer in the second trace than in the first one. The reasons are different accesses to CDS views IFIACCDOCJRNL (Journal) and IFICMPTJRNL (Compact Journal), leading to longer Execution Duration and higher HANA CPU Time consumption. For an in-depth investigation, you open the second trace (by double-clicking it in the upper panel of the Trace Comparison, or by pressing button Display Trace ). There you can use all the tools offered by ST05 to analyze why the second execution is slower. In this example, the evident reason is that it handles more data, but you still may find an approach to accelerate it. Always question the obvious explanations and try to go beyond the apparent cause.

Summary

The trace directory integrated into ST05 serves two main purposes:

It prevents important traces from being overwritten by subsequently recorded traces.
This gives you more time for comprehensive analyses, e.g., to find optimization approaches.

Two previously stored traces can be compared with each other.
With such a comparison you can investigate your application's scaling behavior (related to its communication with external resources) to verify that its execution time or resource consumption grows at worst proportional to the amount of processed data.
Alternatively, with a trace comparison you may be able to demonstrate that your optimization project has been successful.

Traces exported to your front-end computer's file system and then imported into another SAP system enable the comparison of your application's behavior across systems. Such a comparison may reveal the impacts of differences in system configuration, process customizing, or available data volume on your application's performance.

ST05: Aggregate Trace Records

2023-05-17T11:23:04+02:00

This is a follow-up to my blog post ST05: Basic Use. Here I explain various options for aggregating trace records. They greatly facilitate the analysis of a trace.

If you have not yet done so, please read the introductory post Use ST05 to Monitor the Communication of the ABAP Work Process with External Resources. There I describe how my measurement transactions STATS and STATS_FE help you to detect performance bugs in your applications, and I state that ST05 is an essential analysis tool to find root causes for applications that are too slow or consume too many resources. In particular, ST05 captures requests and data transmissions between the ABAP work process and external components. By default these transmissions are not recorded to not incur the associated overhead. You need to consciously and explicitly activate the recording, and subsequently must switch it off. Once you have a high quality trace of your application's communication with external resources, you want to evaluate it in the most efficient way.

Fig. 1 shows the ALV list of Trace Main Records for the Audit Journal Fiori application F0997, with records for the interface types SQL, Enqueue, HTTP, and RFC. Refer to my blog post ST05: Basic Use for a complete description of the scope and purpose of the Trace Main Records. With its chronological sort order, the list provides a comprehensive picture of the flow of the application's communication with external agents, but it does not draw your attention to the most expensive statements, nor does it indicate events that have been executed multiple times or even redundantly. It also does not distinguish between statements related to the involved technical frameworks and statements that contribute to the business logic.

Figure 1: In the chronologically sorted list of Trace Main Records, each entry corresponds to one execution of an ABAP statement calling an external resource, or to the processing of an incoming request.

To remedy these shortcomings,ST05 offers various options to summarize the list of Trace Main Records, which make your work with previously recorded traces much more efficient. They are all accessible from the drop-down menu of button Aggregate Trace . Table 1 briefly introduces them, and the following sections describe them in detail.

**Table 1:** Aggregation levels supported by `ST05`.
Structure-Identical Statements	Specific values of bind variables are not considered.
Value-Identical Statements	Structure-identical statements where the bind variables have the same values.
Table Accesses	Database accesses from the same category (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, ...) grouped by object and by database connection. Buffer accesses using the same number of key fields. (Only for trace types SQL, BUF.)
Trace Overview	Quality control of the trace, break-down of records by type, and high-level evaluation with hints for potential optimizations.
Sizing Information	Estimates the data volume added by the traced application to the database.
Trace Single Records	Resolves individual low-level operations. (They are recorded only for trace types SQL, BUF.)

The main benefit of trace aggregation is for trace types SQL and BUF. They record concrete values used in individual statements, leading to a large variety of trace records. Aggregation creates a significantly shorter list of records, giving you a much better overview of the various kinds of statements and their durations or resource consumptions.

Structure-Identical Statements

The list of Structure-Identical Statements (Fig. 2) summarizes identical Trace Main Records without considering the actual values for any bind variables. The resulting list is augmented by some statistics on the set of main records contributing to each entry, and (for SQL, BUF, and ENQ trace records) by metadata from the ABAP dictionary on the affected table, view, or lock object. Table 2 covers the columns that are shown in the default ALV grid layout. You may add a few additional fields with more technical content (e.g., from the SAP extended passport) by changing the list's layout.

Figure 2: The list of Structure-Identical Statements summarizes the Trace Main Records for statements with the same structure, but potentially distinct values for bind variables. The list is sorted descending by Duration.

**Table 2:** *Fields in the default layout of the list of **Structure-Identical Statements**.*
Executions	Total number of executions of the structure-identical statements.
Redundancy	Absolute number of redundant value-identical statements.
Identical	Relative number of redundant value-identical statements.
Duration	Total elapsed execution time in µs of the structure-identical statements. (Measured by the ABAP work process.)
CPU Time	Total HANA CPU time in µs used during the structure-identical statements' executions. (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Memory	Maximum memory consumption in kByte during the structure-identical statements' executions. (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Records	Total number of records affected by the structure-identical statements. (SQL, BUF: number of table or view rows; ENQ: number of lock granules) (Not for RFC, HTTP, APC, and AMC trace records.)
Duration / Execution	Average execution time in µs per contributing statement.
Records / Execution	Average number of records affected per contributing statement.
Duration / Record	Average execution time in µs per record. (Records = 0 ⇒ average execution time per contributing statement.)
Min. Duration / Record	Minimum execution time in µs per record. (Records = 0 ⇒ minimum execution time of contributing statements.)
Length	ABAP dictionary record length in Byte of the table or view. (Only for SQL and BUF trace records.)
Buffer Type	Buffer type of the table or view. (Only for SQL and BUF trace records.)
Table Type	Table type of the table or view. (Only for SQL and BUF trace records.)
Data Class	Type of data stored in the table. (Only for SQL and BUF trace records.)
Size	Estimated size category of the table. (Only for SQL and BUF trace records.)
Object Name	Name of the object that was accessed. (SQL, BUF: table or view; ENQ: lock object; RFC: function; HTTP, APC: path; AMC: channel ID)
Statement	Edited statement. (SQL: field list and `FROM` clause removed, variable names as place holders for values; BUF: buffer type, length of key values in characters, buffer operation; ENQ: lock operation, lock mode, table, granule argument; RFC: source, destination, `CLIENT` or `SERVER`, function, sent data in Byte; HTTP: method, status code, status text, scheme, host, port, `CLIENT` or `SERVER`, path; APC: executed action, protocol, host, port, `CLIENT` or `SERVER`, path; AMC: <empty>)

The sort order is descending by Duration, placing the most expensive statements at the top. They have the biggest impact on the application's end-to-end response time and are your first entry point to a detailed analysis, which shall also consider the number of Records affected by the statement:
If Min. Duration / Record exceeds 10,000 µs = 10 ms for SELECTs, or 50,000 µs = 50 ms for INSERTs, UPDATEs, or DELETEs, the statement's execution on the database is slow and shall be investigated in detail. If a statement has been executed multiple times (e.g., Executions ≥ 10), also consider the value of average Duration / Record. With few executions, a single outlier may distort the average, i.e., one slow execution may conceal several fast executions, but with 10 or more executions, it should be reliable and taken seriously. In my blog post Analyze Individual Trace Records, I explain how you can recognize the database's processing of a slow statement and how this may indicate optimization approaches.

Other purely technical indicators for potential improvements related to SQL trace records are

Column Buffer Type
It shall be empty. If a table's technical settings is Buffering switched on, the vast majority of Open SQL statements against this table shall be handled by the table buffer on the application server instance. Then they will not be recorded by an SQL trace. SQL trace records where Buffer Type equals FUL, GEN, or SNG indicate that the table buffer was bypassed, which slows down the application, puts unnecessary load on the database server, and does not take advantage of the application server memory used to buffer the table's rows. Rewrite the corresponding statements so that they can be handled by the table buffer. Values in the Buffer Type column starting with DE represent the technical setting Buffering allowed but switched off. Check whether buffering can be switched on for the affected tables. The Buffer Type CUST identifies accesses to unbuffered customizing tables (Data Class = APPL2). Consider to switch on buffering for the table—this is typically suitable for customizing data. Finally, the entry DDIC in the Buffer Type column indicates direct accesses to tables belonging to the ABAP dictionary. Replace them by using functions DDIF*, in particular DDIF_NAMETAB_GET or DDIF_FIELDINFO_GET.

Columns Redundancy or Identical
They shall contain only zeros. These columns show the absolute number or relative number, respectively, of redundant Value-Identical Statements within the corresponding set of Structure-Identical Statements. These duplicate Value-Identical Statements repeatedly access the same rows on the database, instead of taking their content from the application's state in the application server instance. This is slower than necessary and wastes precious resources on the database server. You need to eliminate them. Refer to the following section on Value-Identical Statements for a complete discussion.

Columns Executions and Records / Execution
Entries with a large number of Executions and Records / Execution ≈ 1 may indicate single row SQL statements nested inside a loop. Consider a mass-enabled approach to reduce the number of data transfers.

With domain knowledge about the application's business logic and its purpose, you may identify further critical statements by looking at

Columns Object Name and Statement
For SQL trace records: Does the application need rows from the table or view? If yes, is the statement's WHERE clause as restrictive as possible, and does the field list request only necessary columns?
For the other trace types, similar questions apply and help to identify potentially unnecessary communication events triggered by your application.

Column Executions
Are all executions required for the application to be functionally correct?

Column Records
Does the number of rows make sense? Are all the records needed by the application? Can you use code push-down (at least in the form of SQL aggregate functions) to calculate on the database instead of transferring many rows to the ABAP work process?

With button Absolute <-> Relative Values you can toggle between these two options for columns Executions, Duration, CPU Time, and Records. The relative values are expressed as percentages of the respective totals.
The drop-down menu of button Break Down Trace Record allows you to view either the contributing Trace Main Records or the associated Value-Identical Statements for selected list entries.

Value-Identical Statements

The list of Value-Identical Statements (Fig. 3) also summarizes identical Trace Main Records, but unlike the Structure-Identical Statements, it takes the concrete values for any bind variables into account. This aggregation supports only SQL, BUF, and ENQ trace records. These are the only trace types where values are recorded. The columns in the list of Value-Identical Statements are mostly like the ones for the Structure-Identical Statements. Table 3 contains the differences.

Figure 3: The list of Value-Identical Statements summarizes Trace Main Records for statements with the same structure, and with identical values for bind variables. The list is sorted descending by Duration.

**Table 3:** *Fields in the default layout of the list of **Value-Identical Statements** that differ from the fields for the **Structure-Identical Statements**.*
Executions	Total number of executions of the value-identical statements.
Redundancy	Not available—covered by Executions.
Identical	Not available—covered by Executions.
Statement	Edited statement. (SQL: field list and `FROM` clause removed, variable values; BUF: buffer type, length of key values in characters, key values; ENQ: lock operation, lock mode, table, granule argument)

The list of Value-Identical Statements is sorted descending by Duration. Records with Execution = 1 are suppressed.

Value-identical SQL statements read the same data multiple times from the database, thus create unnecessary load. Identical data can be read redundantly by statements that are not value-identical―and not even structure-identical. This is not visible in the list of Value-Identical Statements. All unnecessary accesses to the persistence layer must be eliminated. This is a more comprehensive demand than just removing value-identical SQL statements. Unfortunately, there is no support in ST05 (nor in any other tool) to find all superfluous SQL statements.
Click button Break Down Trace Record to see the Trace Main Records contributing to the selected Value-Identical Statements.

Table Accesses

The Table Accesses aggregation (Fig. 4) combines statements by their type, separately for each object. It aggregates all SQL trace records corresponding to statements of the same category (e.g., SELECT, INSERT, UPDATE, DELETE) that access a table or view over the same logical database connection. Similarly, all BUF trace records triggered by statements that were handled by the table buffer or by other buffers on the application server instance, and that access an object with the same number of key fields are aggregated. No other trace types are supported by this aggregation option. Table 4 contains the fields displayed in the Table Accesses list.

Figure 4: The list of Table Accesses summarizes, separately for each table or view, all statements of the same category. The sort order is alphabetically by Object Name, Statement category, and Connection Name.

**Table 4:** *Fields in the default layout of the **Table Accesses**.*
Object Name	Name of the table or view that was accessed.
Length	ABAP dictionary record length in Byte of the table or view.
Buffer Type	Buffer type of the table or view.
Table Type	Table type of the table or view.
Data Class	Type of data stored in the table.
Size	Estimated size category of the table.
Statement	Category of statement SQL: `SELECT`, `INSERT`, `UPDATE`, `DELETE`, ... BUF: buffer type, key length
Connection Name	Name of the logical database connection used by the contributing statements.
Duration	Total elapsed execution time in µs of the contributing statements. (Measured by the ABAP work process.)
CPU Time	Total HANA CPU time in µs used during the contributing statements' executions. (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Memory	Maximum memory consumption in kByte during the contributing statements' executions. (Only for SQL trace records on SAP HANA.) (Measured by the SAP HANA DB server.)
Records	Total number of table or view rows affected by the contributing statements.
Executions	Total number of executions of contributing statements.

The list is sorted ascending by Object Name, category of Statement, and Connection Name.

The Table Accesses display provides a highly condensed survey of the tables with which the traced application works and how the application affects these tables. Use it to verify that there are no expensive accesses to tables that have no relevance to the business process, and to make sure that the required accesses have good performance.

As in the list of Structure-Identical Statements, button Absolute <-> Relative Values toggles between these options for columns Duration, CPU Time, Records, and Executions.

Button Component Hierarchy enhances the list of the Table Accesses (Fig. 5). It groups the accessed tables by their application component (levels 0 and 1 are supported, level 0 is the default, buttons or toggle between the levels), and additionally shows the tables’ short descriptions from the ABAP data dictionary. For each application component, absolute and relative subtotals of the duration, the number of records, and the number of executions are given separately for the statement categories (SELECT, INSERT, UPDATE, DELETE; buffer accesses), and across all categories.

Figure 5: The Application Component Hierarchy view augments the list of Table Accesses (Fig. 4) by including the tables' application components and descriptions. With this additional information you can better understand why the tables are accessed when the application executes. The grouping by application component helps you to judge the run time contributions of the components.

Use this enriched list of Table Accesses to understand the software domains used—either directly or via intermediary frameworks—by your application, and to assess how much they impact your application's performance. It may also uncover unnecessary accesses, which you can eliminate without breaking your application's functional correctness.

Trace Overview

The Trace Overview (Fig. 6) provides a quality control of the trace, and also a high-level analysis to indicate potential problem areas. Use this aggregation option to convince yourself that the trace is reliable and to obtain a quick survey of the variety of recorded statements. This may give you some guidance for more detailed analyses.

Figure 6: The Trace Overview evaluates the trace's technical quality, and identifies performance bugs.

It is divided into several areas:

At the top, the main characteristics of the system are shown. This corresponds to the situation when and where the Trace Overview was called. It is not necessarily related to the circumstances in which the application was executed and traced. This is particularly true for old traces from the trace directory, and even more so for traces imported from other systems.
Then, the time intervals between start and end of the trace analysis period and between the first and the last trace records are compared. A large discrepancy may indicate that the trace is incomplete.

The next part evaluates the trace’s technical quality and reliability. If it contains ROLLBACKs, OPEN operations, or SQL statements that load content into the table buffer on the application server instance, you have not executed enough pre-runs of your application before recording the trace. The Trace Overview then advises you to Consider re-recording the trace. Before doing so, execute the application several times. Typically, you want to analyze and eventually optimize the repeated executions of your applications. For them, there should not be any ROLLBACKs. Also, all SQL statements should already be in the database server’s statement cache so that no OPEN operation is needed, and the table buffer should contain all necessary rows.

The following region condenses the trace by the records’ types (SQL, BUF, ENQ, ...), and within the types resolves the main statement categories. Values for Executions, Duration, and Records are given as absolute numbers and as percentages of the respective totals.

Finally, the trace is analyzed for violations of common best practices to identify problematic or suspicious patterns like unnecessary accesses to the persistence layer, e.g., SELECTs for buffered tables or redundant DB accesses. Additionally, the SQL trace records are checked by Code Inspector to find statements whose access times to the persistence may erroneously depend on the amount of data persisted. This is typically a consequence of inadequate index support.
One section also deals with accesses to CDS views, to ensure that analytical CDS views are not used in transactional applications. If the trace contains SQL statements that access transactional CDS views, this section also suggest that you verify the modelling of these views. It must guarantee that SQL queries using them have response times and CPU consumptions independent of the data volume saved in the underlying tables.

Sizing Information

The Sizing Information (Fig. 7) covers only changing database accesses, specifically INSERTs, DELETEs, and MODIFYs. It estimates the impact of the corresponding SQL statements recorded in the trace on the data volume stored in the database. This provides important input for correctly sizing the storage capacity of the production system landscapes where the application will be executed.
Your storage system sizing must be based on a trace that covers the entire business process with all user interaction steps from end to end. Do not apply any filter conditions while recording the trace.

Figure 7: The Sizing Information calculates rough estimates for the change of the data volume stored in the database caused by the statements recorded in the trace. Uncertainties are produced by not knowing whether MODIFYs have added new rows or changed existing rows. Another source of uncertainty is related to table fields of types whose sizes are not statically predetermined.

The top of the Sizing Information contains the main characteristics of the system, identical to the top row in the Trace Overview (Fig. 6).
Section Number of INSERTs, MODIFYs, and DELETEs per Table alphabetically lists the database tables affected by INSERT, DELETE, or MODIFY statements, and shows the numbers of rows affected by statements in these categories. Because a MODIFY can either insert new rows into a database table or update existing rows, two values per table are calculated for the overall net number of changed rows: The difference of INSERTs and DELETEs considers all MODIFYs as updates of existing rows. This is a lower limit for the net number of rows created by the application. The opposite approach treats all MODIFYs as if they insert new rows: It uses the sum of INSERTs and MODIFYs minus the DELETEs to obtain an upper limit for the net number of new rows in each table. The Sizing Information does not consider UPDATEs because they never add rows to a database table.
The next area, Size of Tables, Primary Keys, and Secondary Indexes, summarizes the static sizes in Bytes of the changed tables, including their indexes. It also indicates whether tables or indexes contain fields of type LRAW (X in column LRAW) or of types STRING, RAWSTRING, or TEXT (X in column VAR). The size of the content of these fields is not known statically and taken to be 0, acting as a lower limit. Finally, this area contains the tables' descriptions from the ABAP data dictionary.
Based on this information, section Estimation of Data Volume Added approximates the change to the database's overall size by the statements recorded in the trace: For each table its length plus the lengths of its indexes are summed up. These table-specific sums are then multiplied by either number of changed rows (as explained above) and the resulting sets of products are totalled and given in Bytes, kBytes, and MBytes. Typically, these values are lower limits for the change in the stored data volume, but in some situations may be overestimates. To guard against negative consequences of storage system under-sizing, you should always consider them lower limits, and try to assess the additional impact of the table fields whose sizes are not statically known. From previous experience, you may know their average size in typical production systems.

Individual Records

So far, this blog post was all about aggregating several Trace Main Records that share some common properties. But you can also move in the opposite direction. For trace types SQL and BUF, the Trace Main Records are already combinations of individual Trace Single Records, which represent low-level operations involved in the processing of the database or buffer accesses. From the list of Trace Main Records, button Display Individual Records takes you to the Trace Single Records (Fig. 8). Table 5 explains the columns that are not in the Trace Main Records.

Figure 8: The chronologically sorted list of Trace Single Records shows the individual trace records without any aggregation. The three selected entries contribute to the processing of one SQL statement, and are combined into one Trace Main Record with the Start Time of the first entry (cf. Fig. 1).

**Table 5:** *Fields in the default layout of the list of **Trace Single Records** that are not also part of the **Trace Main Records**.*
Instance Name	Application server instance where the statement was triggered.
Operation	Operation that was executed. (RFC, HTTP: `Client` or `Server`; APC: executed action, protocol, path, port, sent and received data in Byte)
Return Code	Return code of the executed operation.

The Trace Single Records are sorted chronologically. The main value of this full resolution view on the trace records is mostly as an entry point to a technical analysis of how the database server or the buffers on the application server instances have handled table accesses via (Open) SQL statements. Investigations from a business logic perspective hardly ever work with the Trace Single Records.

Summary

Compared to the default list of chronologically sorted Trace Main Records, the various aggregation levels offered by ST05 greatly improve its capabilities to analyze traces in an efficient way.

The high-level Trace Overview provides a very basic quality control of the trace recording. It can also identify performance bugs of the application, and it signals suspicious database accesses.
The Table Accesses view indicates with which tables your application works and what categories of statements it uses on these tables. Use it with your knowledge of the underlying business logic to detect unnecessary or expensive table accesses.
The list of Structure-Identical Statements focusses on statements with a common form and indicates slow statements or statements that are better handled by the application server instances' buffers. The list of Value-Identical Statements concentrates even stronger on redundant, therefore superfluous, statements with the same bind variable values. You must eliminate them to accelerate your application and to take load away from the central database server.
The Sizing Information has a different objective and supports the forecast of the required storage capacity by estimating how much the changing SQL statements of an application change the persisted data volume.

ST05: Analyze Individual Trace Records

2023-06-30T11:40:01+02:00

This is—for the time being—the last blog post in my series Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources. Previous blog posts in this sequence have mostly dealt with the ST05 tool itself, or with complete traces recorded for an entire end-to-end business process or at least for a whole user interaction. Now it is time to focus on individual trace records, representing statements or communication events triggered by your application. Typically you want to concentrate on those with a long duration or with a high resource consumption. They have the biggest impact on your application's performance and scalability. You should direct your optimization efforts to them.

The best entry point for such a project is the aggregated list of Structure-Identical Statements (Fig. 1). I have covered it in my post ST05: Aggregate Trace Records. This list sorts the statements descending by their Duration. The overall slowest statements (considering all their aggregated executions) are at the top of the list. If you are more concerned about HANA CPU Time or HANA Memory consumption, re-sort the list according to these values. In any case, you want to work on the list's first few records.

Figure 1: The list of Structure-Identical Statements summarizes Trace Main Records for statements with the same structure, but potentially distinct values of bind variables. By default, the list is sorted descending by Duration. This list is the recommended starting point for an optimization project that focuses on the most expensive communication events triggered by your application.
The subsequent discussions use the marked SQL trace record as an example.

Before diving into a thorough analysis of how a statement was processed and how you can optimize it, ask yourself whether it is really required. If the statement contributes neither to your application's business logic nor to the technical frameworks employed by the application, you shall remove it. Elimination of unnecessary statements is always the best optimization strategy—and often the easiest. Statement tuning is meaningful only for essential statements.

Statement Details

The Statement field in all types of trace lists is always strongly abbreviated. For a trace record representing a statement that you have confirmed as essential for your application, you can display all of the statement's details by clicking button Statement , available for the Trace Single Records, the Trace Main Records, the Structure-Identical Statements, and the Value-Identical Statements. Alternatively, double-click the Statement field. Fig. 2 shows the Statement Details of the SQL trace record marked in Fig. 1. (The result depends on the type of trace record.)

Figure 2: Statement Details for the SQL trace record highlighted in Fig. 1. The left panel shows the default option With Variable Names of button Statement. Bind variables are shown as question marks with an orange background. (Other database platforms may use different place holders.) Their types and values are listed separately in section Variables. The right panel displays the alternative With Variable Values, where the bind variables have been replaced by the actual values.
Navigation to the Statement Details from an aggregated trace record list (Structure-Identical Statements or Value-Identical Statements) takes specific values (e.g., for bind variables in the case of an SQL trace record) from the first contributing record.

For an SQL trace record, the Statement Details display the SQL statement as it was sent from the ABAP work process to the database server for execution. The statement's implementation, e.g., as Open SQL, in your ABAP source code may be different (e.g., it will typically not have an explicite WHERE clause condition on the MANDT table field). This is the main benefit of the Statement Details display: It enables you to understand how the database interface in the ABAP work process interpreted your implementation, and what the database really had to do. As the application's developer you are familiar with its statements from the design time and implementation time perspective. The Statement Details give you the run time view which helps you to verify that the statement sent to and processed by the database fully matches the intended purpose.

No measurement values (e.g., for elapsed time, consumed resources, or number of processed table rows) are included in the Statement Details for an SQL trace record. This also holds for BUF trace records, while details for trace records of the other types do include some measurements.

DDIC Information

A full analysis and understanding of statements needs to consider the declarations and technical settings of the objects affected by them. For trace records of types SQL or BUF, a click on button DDIC Information , or a double-click on the Object Name field, produces a summary of the ABAP Dictionary Information for the table or view (Fig. 3). If the trace record corresponds to an SQL statement that joins several tables, an intermediate popup enables you to select an individual table. From the ABAP Dictionary Information, button Display Table Fields takes you to the ABAP Data Dictionary transaction SE11 to look at the complete declaration, including primary key fields, secondary indexes, and buffer settings. Or you jump immediately to either all available indexes, or to a selected index via buttons Display All Index Fields or Display Fields of Selected Index, respectively. For ENQ trace records, the lock object is shown in transaction SE11. DDIC information does not exist for the other trace types.

Figure 3: The ABAP Dictionary Information summarizes a table's or view's declaration and technical settings as maintained in the ABAP Data Dictionary. From this summary, you can navigate into transaction SE11 to see all details.

Based on the DDIC Information, you can judge whether accesses to the object are really required for your application. When you combine the statement (e.g., as seen in the Statement Details) with the available indexes, you may recognize potential mismatches between the WHERE clause and the existing indexes. Such mismatches may cause excessive run time or resource consumption while processing the statement.

ABAP Source Code

As already mentioned, the Statement Details show the SQL statement as executed by the database server. To compare this with how the statement was implemented in your application's source code, click on button ABAP Call Location . This takes you to the correct position in the appropriate tool of the ABAP development workbench (i.e., Class Builder SE24, Function Builder SE37, ABAP Editor SE38). There you can apply optimizations, locally to the source code or—after forward navigation into the ABAP Dictionary SE11—globally to the accessed object.

ABAP Call Hierarchy

In my previous blog post ST05: Basic Use I have indicated that the immediate caller of the traced communication event may not be the ABAP source code location where you as a business application developer can apply optimizations. Rather, it may be nested inside some software framework called directly or indirectly by your application's ABAP code. To see the full ABAP call stack (Fig. 4) click button Call Hierarchy .

Figure 4: The ABAP Call Hierarchy for the SQL statement corresponding to the trace record marked in Fig. 1 indicates that only the ABAP events at levels 26 to 30 are specific to the business application. Everything else is framework coding.

Use the ABAP Call Hierarchy to verify that your application coding invokes the framework which eventually triggers the statement in the best possible way, and to confirm that calls from frameworks to your source code are fully optimized. Click on any entry in this list to navigate to the corresponding source code.

Data Access Strategy

For a deep understanding of an SQL statement's run time behavior—duration as well as resource consumption—you need a very close look at how the database server has executed the statement which you have formulated in a declarative rather than imperative way. Based on a detailed insight you can derive optimization strategies. Ideally, they work locally on the expensive statement so that they cannot have side effects on any other statement. Be very cautious with global changes to the table's or view's declaration, indexes or technical settings. They may improve the performance or resource consumption of the current statement, but may negatively impact other statements which also access the table or view.

Execution Plan

The standard option Execution Plan (button ) is available for all supported database platforms. Its upper area shows the SQL statement with variable names, and the lower area indicates, in a database specific format, how the database server has handled the statement. This part is often fairly terse and technical and already for only moderately complex statements not easy to digest.

Figure 5: The Execution Plan for the SQL statement corresponding to the trace record marked in Fig. 1 displays at the top the statement, similar to the option With Variable Names of the Statement Details (Fig. 2). This part is common to all supported database platforms. It is followed by the database specific representation of how the SQL statement was processed on the database server (here SAP HANA 2.00).

The Execution Plan displays the nested hierarchy of the applied operations with their options and conditions and the objects on which they were used. The buttons offered in the application toolbar depend on the underlying database platform and may reveal some additional facts (e.g., estimates on the sizes of the involved tables and the sizes of the operation's output) which add significant analysis capabilities.

SQL Analyzer Tool for SAP HANA

If your system uses SAP HANA, you are better off with the graphical SQL Analyzer Tool for SAP HANA. This requires that you have Visual Studio Code installed on your local machine and that you have the SQL Analyzer Tool for SAP HANA extension added to it. For a seamless integration with ST05, you also should set parameter HDB_OPEN_STUDIO (Open HANA Studio or SQL Analyzer when a Planviz is created) to X in your user profile, and set Visual Studio Code as the default app for .plv files. Then a click on HANA SQL Analyzer will create an appropriate plan file, download it to your local machine's hard disk (into your SAP GUI's documents folder), and from there open it with the extension SQL Analyzer Tool for SAP HANA (Figs. 6 and 7). Before the plan file creation, a popup asks you to specify (either confirm or overwrite) session variables. These are a kind of global variables set in the ABAP user context and forwarded from there to the database server where they control the execution of SQL statements. When analyzing an SQL statement in ST05, and creating its execution plan, the values for the session variables are retrieved from the SET SESSION VARIABLE statements which have defined the values and which are recorded together with the SQL statement that you are analyzing.

Figure 6: The SAP HANA SQL Analyzer extension for Visual Studio Code is the tool of choice for understanding how the SAP HANA database has executed an SQL statement recorded in a trace. The Overview organizes significant information in dedicated cards.

Figure 7: The Plan Graph in the HANA SQL Analyzer represents the execution of the SQL statement as a network graph of the involved plan operators and their connections.

The SQL Analyzer's main section is the analysis view. Its three tabs support different perspectives on the SQL statement and its execution.

The Overview tab (Fig. 6) summarizes the most important information in several cards. The main cards for performance and scalability investigations are:
- Context:
  Displays the statement with its conditions, variables, and parameters. For more detailed information, go to the SQL tab.
- Execution Summary:
  Provides details such as compilation and execution times, peak memory usage, number of operators, and connections.
- Dominant Plan Operators:
  Lists the top 5 operators by execution time.
- Data Usage:
  Shows the size of the result set and the number of accessed tables.

The Plan Graph tab (Fig. 7) provides a visualization of the SQL query plan. It shows the data flow from bottom to top: Data retrieval from tables is at the bottom, and the query result is at the top. Along the path, plan operators achieve the transformations to obtain the desired output. Operators working concurrently, i.e., in parallel, are positioned next to each other. Nodes in the Plan Graph represent physical or logical operators, and contain associated information, especially a breakdown of the operator's execution time. The edges connecting the nodes indicate the number of rows transferred between the two operators. The critical path is determined by the chain of operators with the biggest impact on the query performance. Its edges are shown in red.

The SQL tab shows the SQL statement with the full session context and the parameters with their types and values. With this information, the SQL query can be re-executed under identical conditions.

The property pane below the analysis view contains 8 tabs where you can focus on small details of the statement's execution to get a very thorough understanding and to discover optimization approaches.

Summary

With a monitoring tool like my transaction STATS (refer to my blog post Measure your Application’s Performance and Resource Consumption: Evaluate Statistics Records with Transaction STATS for an introduction) you may discover that your application's performance is insufficient. Maybe your application is too slow or its resource consumption is too high. In either case, STATS may indicate that the poor performance is caused by communication events triggered by your application. Then you would use my transaction ST05 to record a corresponding trace. Your application's most expensive communication events will be at the top of the appropriately sorted trace list—ideally you work with the list of Structure-Identical Statements. With the various tools offered by ST05 you can

gain insight into the significance of the event for your application

inspect the statement's implementation in the ABAP source code

recognize the ABAP call stack that eventually triggered the event

review the database tables or views on which the statement operates

examine the statement as it was processed

Based on this comprehensive information, you can derive optimization strategies which will improve your application's performance.

References

The official documentation for the SAP HANA SQL Analyzer is available at SQL Analyzer Tool for SAP HANA Guide.
The SAP HANA Troubleshooting and Performance Analysis Guide might also be helpful.
The Open SAP course A First Step Towards SAP HANA Query Optimization covers further techniques for SQL query performance analyses and optimizations.

Azure Monitor for SAP solutions One-stop-shop for SAP technical monitoring

2024-01-04T09:19:34+01:00

Are you an SAP Customer/Partner/ISV seeking a seamless monitoring solution? In this blog, I introduce Microsoft's latest Monitoring solution—Azure Monitor for SAP Solutions—a user-friendly and plug-and-play tool.

What is Azure Monitor for SAP Solutions?

What is the architecture?

What can you monitor?

What are the other key capabilities of AMS?

What are pricing and regions available?

1. What is Azure Monitor for SAP Solutions?

Azure Monitor for SAP solutions is a comprehensive technical monitoring tool tailored for businesses leveraging SAP applications on Microsoft Azure. This turnkey solution facilitates the collection of telemetry data from SAP NetWeaver, databases, Linux Pacemaker clusters, and Linux operating systems. Centralized within the Azure portal, this tool streamlines visual correlation and rapid troubleshooting, requiring just a few minutes for configuration without the need for customer infrastructure deployment or maintenance.

2. What is the architecture?

The following diagram shows, at a high level, how Azure Monitor for SAP solutions.

AMS Architecture

Monitoring SAP Systems: Azure Monitor for SAP solutions allows monitoring of multiple components across various SAP systems in a virtual network.

Configurable Providers: Configure providers for different components (e.g., HANA database, HA cluster, NetWeaver, DB2 and OS) to collect data.

Key Architecture Components:
- Azure Portal: Access point.
- Monitor Resource: Displays monitoring data.
- Managed Resource Group: Automatic deployment with Azure Functions, Key Vault, Log Analytics Workspace, and Storage Account.

Visualization and Querying:
- Azure Monitor workbooks for customizable visualizations. Pin items to the Azure dashboard for auto-refresh (max every 30 minutes).
- Use Kusto Query Language (KQL) for log queries in Log Analytics.
- Detailed Architecture is available at What is Azure Monitor for SAP solutions?

3. What can you monitor?

SAP HANA
- CPU, memory, disk, and network use
- HANA system replication
- HANA backup
- HANA host status
- Index server and name server roles
- Database growth
- Top tables
- File system use

SAP NetWeaver
- SAP system and application server availability, including instance process availability of:
  - Dispatcher
  - ICM
  - Gateway
  - Message server
  - Enqueue server
  - IGS Watchdog
- Work process usage statistics and trends
- Enqueue lock statistics and trends
- Queue usage statistics and trends
- SMON metrics (/SDF/SMON)
- SWNC workload, memory, transaction, user, and RFC usage (St03n)
- Short dumps (ST22)
- Object lock (SM12)
- Failed updates (SM13)
- System log analysis (SM21)
- Batch job statistics (SM37)
- Outbound queues (SMQ1)
- Inbound queues (SMQ2)
- Transactional RFC (SM59)
- STMS change transport system metrics (STMS)

HA Pacemaker Cluster
- Node, resource, and SBD status
- Pacemaker location constraints
- Quorum votes and ring status
- Also see the metrics specification for ha_cluster_exporter.

OS (Linux)
- CPU use, fork count, running processes, and blocked processes.
- Memory use and distribution among used, cached, and buffered.
- Swap use, paging, and swap rate
- File system usage, along with number of bytes read and written per block device.
- Read/write latency per block device.
- Ongoing I/O count and persistent memory read/write bytes.
- Network packets in/out and network bytes in/out

IBM Db2 & Microsoft SQL Server
- Database availability
- Number of connections, logical reads, and physical reads
- Waits and current locks.
- CPU, memory, and disk use
- Host name, SQL instance name, and SAP system ID
- Batch requests, compilations, and page life expectancy over time
- Top 10 most expensive SQL statements over time
- Top 12 largest tables in the SAP system
- Problems recorded in the SQL Server error log.
- Blocking processes and SQL wait statistics over time.

4. What are the other key capabilities of AMS?

The SAP Landscape Monitor is a one-stop destination to learn about the health of your entire SAP landscape. You can create a group of SAP systems based on the environment (e.g., Development, Production, etc.), application (e.g., ERP), or criticality and see the health of these systems based on the alerts that have been raised. You can also check alerts triggered by a group or a specific SAP system.

AMS Landscape Monitor

5. What are Pricing and regions available

There is no licensing fee for Azure Monitor for SAP solutions. Customers pay for the infrastructure components that are deployed as part of the managed Azure Resource Group. Standard Azure prices are applicable for Azure Functions (Premium Linux), Azure Log Analytics, Storage, Key Vault, alerts, and notification groups.

Azure Monitor for SAP solutions is available in East US, East US 2, West US, West US 2, West US 3, Central US, South Central US, North Central US, West Central US, West Europe, North Europe, Australia Central, Australia East, Australia Southeast, South India, Southeast Asia, and East Asia.

Learn more!

To learn more, read documentation or watch overview and demo video.

To get started, head over to Microsoft Azure portal and create your first resource.

Share your thoughts and feedback.

Basis Monitoring & Tcodes with Key notes

2024-02-05T14:25:53.980000+01:00

Hi All,

I am thrilled to have the opportunity to connect with all of you through this blog.

The purpose of this blog is to aid newcomers in Basis in gaining knowledge about Basis-related Tcodes, including key notes and their usage and frequency.

I believe this will be beneficial for those who are beginning their careers in SAP Basis.

I wish you good luck and welcome to SAP Basis Team

The Daily Monitoring Basis-related Tcodes, their uses, and their related Tcodes are utilised for any future investigation.

Tcodes that pertain to operating systems and databases, their usage, and any future process.

The SAP Basis Admin is accountable for tuning performance. These Tcodes are associated with performance analysis at the application level.

Ticketing tools vary widely between organizations, including SAP ITSM (SOLMAN), Non SAP (ServiceNow, Zendesk), and others.

User Management, Role Management, and Transport Management will receive the majority of daily ticketing. Here are the Tcodes that pertain to these areas.

Tcodes that pertain to SAP Software Maintenance and related OS and other tasks.

Programs that are useful for administrative tasks related to Basis.

Thank you for taking the time to read the blog.

#SAPBasis #Basis Basis Technology EWM - Basis NW ABAP Monitoring Tools SAP EarlyWatch Alert SAP HANA Cloud, SAP HANA database SAP Advantage Database Server Oracle Database SAP S/4HANA SAP NetWeaver Application Server for SAP S/4HANA #Dailymonitoring NW Java Security and User Management Defense and Security SOLMAN Setup/Configuration/LMDB NW ABAP Monitoring Tools SOLMAN System Monitoring