SAP Community - SAP HANA

How Data Anonymization in SAP HANA secure Generative AI applications through SAP Datasphere

2024-03-15T20:12:44.703000+01:00

Introduction

As Generative AI continues to evolve, the complexities surrounding the management of Personally Identifiable Information (PII) also increase.AI systems can sometimes deduce PII from seemingly harmless data or unintentionally reveal PII in their results. Therefore, it's essential for AI developers and users to understand these potential risks and take proactive measures to address them.

Personally identifiable information (PII) refers to any information that can be used to identify an individual. This might include, but is not limited to, name, social security number, date and place of birth, mother’s maiden name, or biometric records. In the digital age, it could also include digital identity data such as IP addresses or mobile device IDs.

Handling PII responsibly is critical for many reasons:

Respecting Privacy
Ensuring Security
Building Trust

On the regulatory front, there are laws including the European Union's General Data Protection Regulation (GDPR) and the U.S.'s California Consumer Privacy Act (CCPA) that offer a framework for PII management. These laws empower individuals with rights over their personal data. However, it's worth noting that these regulations are relatively recent and still evolving. There's also ongoing discourse about the optimal way
to regulate AI systems to balance the need for privacy protection and technological innovation. SAP HANA Data Anonymization provides methods in order to overcome all the above potential risks. The methods supported by SAP HANA are :

k-anonymity
l-diversity
Differential privacy

More information on the SAP HANA Documentation :
https://help.sap.com/docs/SAP_HANA_PLATFORM/f88e51df089949b2af06ac891c77abf8/ee693d6584d243e1a0daf7c137f9600c.html

Architecture

On this blog we will explore the various solutions for tackling PII-related challenges through the SAP Business Technology Platform. We'll be focusing on SAP Datasphere's capabilities, which not only enable us to consolidate all necessary data sources, SAP or non-SAP, but also leverage SAP HANA's powerful data anonymization methods. This combination ensures your data's security while maximizing its potential.

Considering the unique capabilities and skills of the personas interacting with SAP Datasphere, we have come to the following options.

We really hope that the first option , direct usage of Data Anonymization annotations from SAP Datasphere interface to be on the roadmap really soon.

A generic architecture of a Generative AI project, where SAP Datasphere takes center stage, can be seen below :

More information regarding the CAP LLM Plugin , please check this link

https://community.sap.com/t5/technology-blogs-by-sap/elevating-data-privacy-in-sap-cap-based-genai-applications-with-cap-llm/ba-p/13585745

Hybrid Approach, focus mainly for Business personas

The starting point for business users will be a local table accessible through the SAP DSP interface (INPUT).
We'll elaborate more on this in the upcoming images. The entire process will be seamlessly managed behind the scenes, offering a user-friendly experience (PROCESS). It will be set up just once on the underlying SAP HANA of SAP DSP. A procedure will generate an Anonymization view, which will be displayed on the SAP Datasphere interface (OUTPUT). This procedure is programmed to run every minute. Only when the user activates the "enable" flag during the initial step will this procedure execute. Otherwise, it simply remains inactive. The final output will be an SAP Datasphere view, tailor-made for business users based on their specific requirements. The anonymized view produced can be distributed by the Data Controller to the necessary DSP spaces, substituting the original non-anonymized dataset. This straightforward step allows us to exert additional control over the access permissions for the Data Consumer.

Hybrid approach

Details regarding the Configuration (SAP Datasphere Local Table)

Information regarding SAP HANA Procedure and Scheduler

Conclusion

In conclusion, the use of SAP Datasphere in a generative AI scenario presents an array of advantages. It significantly enhances data handling and security, ensuring the protection of personally identifiable information through SAP HANA's anonymization methods. The versatility of SAP Datasphere allows it to cater to both IT and Business personas, offering a range of implementation methods including direct, SQL, Python or via a CAP Application. Being a one-stop solution for all generative AI needs, its hybrid approach is highly flexible, capable of extending and adapting to any specific requirement.

This makes SAP Datasphere a highly efficient and user-friendly tool for handling generative AI scenarios.

Taking Data Federation to the Next Level: Accessing Remote ABAP CDS View Entities in SAP HANA Cloud

2024-03-25T13:35:26.317000+01:00

Accessing data in ABAP systems, such as SAP BTP ABAP environment, S/4HANA Cloud, BW/4HANA, and others, from an external SAP HANA database is not merely a new requirement or passing trend; it has been a longstanding and ongoing need to enhance and streamline business processes. This need becomes even more crucial when considering the SAP HANA Cloud, which serves as a single gateway for all your enterprise data and where next-generation Intelligent Data Applications are built and run.

However, there are several reasons why directly accessing the ABAP-managed standard objects in the underlying SAP HANA database of an ABAP system is not recommended. These include the instability of names and internal structures, unexpected typecasts, incorrect session variables, and bypassed ABAP-level security, among other concerns. For more details on these issues, please refer to SAP Note 2511210.

To overcome these challenges, data federation through the Smart Data Integration (SDI) ABAP adapter has been proposed as a potential solution, while data replication is supported by both the SDI ABAP adapter and SAP Landscape Transformation (SLT). It is important to mention, though, that the SDI ABAP adapter, as explained by Zili Zhou in her blog, is primarily designed for data replication and may have limitations when it comes to data federation due to limited pushdown capabilities.

In this context, I'm glad to share that starting with SAP HANA Cloud, SAP HANA Database (SAP HANA database in SAP HANA Cloud) QRC 01/2024, a new Smart Data Access (SDA) ABAP adapter has been released. This adapter allows for accessing ABAP CDS view entities in a remote ABAP system using SQL via Open Database Connectivity (ODBC) from an SAP HANA database in SAP HANA Cloud. Please find a nutshell of this new capability below:

Create a remote source to an ABAP system based on SAP ABAP Platform 2108 or later.
Virtualize remote ABAP CDS view entities by creating read-only virtual tables that point to them, and if needed, also pass parameters to the remote parameterized ABAP CDS view entities.
All limitations from SAP Note 2511210 have been lifted, providing the most optimal pushdown capabilities for data federation.
This capability is only available in SAP HANA database in SAP HANA Cloud.

Prerequisites & considerations

First, please ensure that the versions of the ABAP system and SAP HANA database in SAP HANA Cloud are as mentioned below:

The ABAP system should be based on SAP ABAP Platform 2108 or a later version where SQL service is available. For instance, starting with S/4HANA 2021, SQL service has been available.
SAP HANA Cloud, SAP HANA Database (SAP HANA database in SAP HANA Cloud) should be QRC 01/2024 or a later version where ODBC driver for ABAP is available, and the script server should be enabled and running.

Another aspect that should be considered is whether the ABAP system is protected by a corporate firewall (e.g., on-prem S/4HANA). If it is, then SAP Cloud Connector is required.

If the above conditions are met, the next step is to expose ABAP CDS view entities from the ABAP system side so that they can be accessed from SAP HANA database in SAP HANA Cloud via the SQL service.

Since developing and exposing an SQL service in the ABAP system is not the focus of this blog and has already been well-explained by Frank-Martin Haas in his blog, please refer to his blog and the following links.

SAP Business Technology Platform: Developing and Exposing an SQL Service in the ABAP System
SAP Tutorials: Using ODBC driver for SQL Service (From Step 3 to Step 9)

Getting started with SDA ABAP adapter (abapodbc)

If everything is ready in the ABAP system to expose ABAP CDS view entities, let’s configure the SDA ABAP adapter from the SAP HANA database in SAP HANA Cloud.

The configuration steps are almost the same as using SDA with other remote sources: downloading the DigiCert SSL certificate, importing the certificate for SSL connection, creating a remote source, and finally, creating a virtual table. Please refer to the examples below.

Download the DigiCert SSL certificate:

-- DigiCert Global Root CA
curl -O https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem

-- DigiCert Global Root G2
curl -O https://dl.cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem

Since DigiCert has stopped signing certificates with the root certificate authority (CA) "DigiCert Global Root CA" and is now moving forward with the "DigiCert Global Root G2" certificate, it is recommended to add both certificates during the switching period. However, using only one certificate is basically sufficient.

Import the certificate for SSL connection:

-- import the certificate for SSL connection (CA)
CREATE PSE SSL;
CREATE CERTIFICATE CA_CERT FROM '
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
...
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----
';

-- import the certificate for SSL connection (G2)
CREATE CERTIFICATE G2_CERT FROM '
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
...
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----
';

ALTER PSE SSL ADD CERTIFICATE CA_CERT;
ALTER PSE SSL ADD CERTIFICATE G2_CERT;
SET PSE SSL PURPOSE REMOTE SOURCE;

Create a remote source:

-- create a remote source to the ABAP SQL service endpoint
CREATE REMOTE SOURCE ABAP ADAPTER "abapodbc" 
CONFIGURATION 'uidtype=alias;driver=ODBC_driver_for_ABAP.so;servicepath=/sap/bc/sql/sql1/sap/s_privileged;host=<abap_host_name>;port=<abap_port_number>;language=EN;typemap=semantic;'
WITH CREDENTIAL TYPE 'PASSWORD' USING 'user=<user_name>;password=<password>';

Alternatively, like other SDA adapters, the remote source can be created using the SAP HANA Database Explorer UI. Once the remote source is created, the exposed ABAP CDS View entities can be browsed using the SAP HANA Database Explorer as described below.

Create virtual tables:

-- create virtual tables by pointing to the exposed ABAP CDS view entities
CREATE VIRTUAL TABLE DEMO.V_ORDERS AT "ABAP"."NULL"."ZORDERS"."Orders";
CREATE VIRTUAL TABLE DEMO.V_ORDERITEMS AT "ABAP"."NULL"."ZORDERS"."OrderItems";

Alternatively, virtual tables can also be created using the SAP HANA Database Explorer UI by selecting the remote object and clicking on the 'Create Virtual Object(s)' button in the screenshot above.

Please also refer to the links below for further details.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Import Certificates for SSL Connections to Remote Source
SAP HANA Cloud, SAP HANA Database Data Access Guide: Create an ABAP Cloud Remote Source

Before moving forward, please don’t forget to create statistics on virtual tables. Creating statistics is always strongly recommended to ensure optimal decision-making on the query execution plan by the query optimizer.

Create statistics on a virtual table:

CREATE STATISTICS ON DEMO.V_ORDERS TYPE SIMPLE;
CREATE STATISTICS ON DEMO.V_ORDERITEMS TYPE SIMPLE;

Further details can be found at the link below.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Create Statistics on a Virtual Table or Linked Database

Query execution and execution plan

With above configurations, if the same statement from the blog by Frank-Martin Haas is executed but with virtual tables, the result will be as follows.

SELECT OI."Item", SUM(OI."Amount")
FROM DEMO.V_ORDERS O
INNER JOIN DEMO.V_ORDERITEMS OI
ON O."Id" = OI."Orderid"
GROUP BY "Item";

Now, let’s see the query plan using Analyze - Explain Plan.

As you can see, both aggregation and join are pushed down to the remote ABAP system and executed there. In other words, both aggregation and join are performed on the remote side, and then the result is transferred to the SAP HANA database in SAP HANA Cloud. Since SDA is a part of SAP HANA core like the query optimizer, if statistics are available, the query optimizer will create the most optimal execution plan based on the tight integration with SDA.

Other limitations

The most important limitation to keep in mind is that the SDA ABAP adapter supports only read-only virtual tables at the moment. This leads to the following limitations:

DMLs (e.g., INSERT/UPDATE/DELETE) cannot be executed on virtual tables.
WITH REMOTE is not supported when executing CREATE VIRTUAL TABLE.
Join relocation to a remote ABAP system, which requires temporary table creation, is not supported. However, if another remote source with temporary table creation capability is involved additionally, join relocation to that remote source is possible but it will be decided by the query optimizer.

FAQs

Q: Is this SDA ABAP adapter only available in the SAP HANA database in SAP HANA Cloud? Is there any plan to support this in SAP HANA Platform 2.0?
A: Yes, it is only available in the SAP HANA database in SAP HANA Cloud. There are no concrete plans to support this in SAP HANA Platform 2.0.

Q: Is there any plan to support real-time replication, such as SDA-based remote table replication described in this blog? If not, what is the recommended technology for replicating ABAP CDS view entities?
A: There are no plans to support real-time replication with the SDA ABAP adapter. The SDA ABAP adapter is designed for data federation purposes only, even though toggling to a snapshot replica is supported to improve query performance. For replicating ABAP CDS view entities, it is recommended to use SDI ABAP adapter or SLT.

Q: Is it possible to pass parameters to the remote ABAP CDS view entity through the virtual table?
A: Yes, passing parameters to the remote ABAP CDS view entity is supported. Please refer to the link below.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Virtualizing Parameterized Views

Q: Is it possible to connect to the on-prem ABAP system? If so, how can it be done?
A: Yes, the way of connecting the on-prem ABAP system can be found in the links below.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Create an ABAP On-Premise Remote Source

However, please ensure that the following conditions are met:

The on-prem ABAP system is based on SAP ABAP Platform 2108 or later version.
SAP Cloud Connector is properly configured so that the on-prem ABAP system is ‘reachable’ from the SAP Cloud Connector and SAP BTP Cockpit.
SAP Cloud Connector is enabled, and the corresponding IP addresses are allowed in the SAP HANA database configuration in SAP HANA Cloud.
The CREATE REMOTE SOURCE statement includes the correct specifications for useCloudConnector and ConnectivityLocationId properties.

Q: It seems that only the ‘technical user’ can be used for authentication. Are there any plans to support other authentication methods?
A: Yes, other authentication methods will be considered in the future.

Q: Is it also possible to access remote ABAP CDS (DDIC-based) views or other remote ABAP-managed objects?
A: No, only remote ABAP CDS view entities or remote parameterized ABAP CDS view entities can be accessed. For the differences between ABAP CDS (DDIC-based) views and ABAP CDS view entities, please refer to this blog.

Q: Is it possible to use the linked database feature?
A: The linked database feature with a 3-part name is supported. However, passing parameters through the linked database feature is not supported. Linked database optimized mode is also not supported in SAP HANA Cloud.

Conclusion

To conclude, it is evident that accessing data in ABAP systems plays a vital role in the development of Intelligent Data Applications on the SAP HANA database in SAP HANA Cloud. This also aligns perfectly with the concept of Clean Core, which involves minimizing customizations and rearchitecting them on SAP BTP, particularly on the SAP HANA database in SAP HANA Cloud. By elevating data federation to the next level with SDA ABAP adapter and embracing these practices, organizations can unlock the full potential of their data, leveraging the power of SAP HANA Cloud. So, don't miss out on the opportunity to harness the benefits of Intelligent Data Applications and propel your business towards success with the SAP HANA database in SAP HANA Cloud.

Product Updates for SAP Business Application Studio - March 2024 Edition

2024-03-26T08:39:28.924000+01:00

Dear community,

Welcome to our detailed exploration of some recent enhancements within the SAP Business Application Studio. This post will guide you through key updates, including our streamlined Storyboard, a newly designed 'Welcome' page, the introduction of SAPUI5 adaptation projects, a comprehensive SAPUI5 Walkthrough Tutorial with TypeScript, and the new feature – release of the powerful UI5 linter. Each feature mentioned contributes to overall user productivity, designed to improve your working experience with us.

Enhancing Productivity with our Updated Storyboard

SAP Business Application Studio now provides a streamlined storyboard. The improved interface allows you to readily view which service actions and functions your application includes. This overview aids in the comprehension and management of your tools, ensuring you are using your resources to their full potential.

We've also added the option to trigger the creation of unbound actions and functions directly. This eliminates the need for extra steps and consolidates key tasks in one location, reducing the time required for task navigation and management.

The described enhancements improve the experience in two major ways: by having a more complete, clearer visualization of your app's functions and by simplifying the creation and editing process.

Have a look at some sample of our updated storyboard below:

Updated Storyboard of SAP Business Application Studio

New Welcome Page for SAP HANA Applications

Our new 'Welcome' page, designed with your needs in mind, serves as an efficient gateway to interact with SAP HANA-related features within SAP Business Application Studio. This hub offers enriched documentation, insightful blogs, and direct access to various integral aspects that can facilitate and speed up your SAP HANA development journey.

SAP HANA Welcome Page in SAP Business Application Studio

As soon as you start your SAP Business Application Studio instance for SAP HANA Native Applications, you will find this new page. Some of the hyperlinks point to guides within SAP Business Application Studio, others to outside of it, including blogs on community.sap.com and material on help.sap.com.

The information is segmented into five structured sections. The 'Start' segment provides straightforward access to SAP HANA project creation, while the 'Development' section enlists tools to accelerate initiating a project or service.

The 'Blog posts' segment serves as a crucial knowledge base, offering valuable insights into SAP HANA development, including vital components like calculation view modeling and best practices. For those aspiring to refine their skills, the 'Get Started with SAP HANA' section presents comprehensive learning materials related to SAP HANA development, facilitating continuous skill improvement.

Lastly, the 'Help' section plays the role of an informative guide, providing exhaustive documentation pertaining to SAP HANA development topics. In essence, our 'Welcome' page aims to be a comprehensive, easy-to-navigate resource, dedicated to refining your SAP HANA developmental experience, right from the outset.

SAPUI5 Adaptation Projects for SAP S/4HANA Cloud Public Edition

SAP Business Application Studio now enables a significant feature for enhancing SAP S/4HANA Cloud Public Edition - the ability to employ SAPUI5 adaptation projects. These projects enable the creation of application variants tailored to fulfill the specific business requirements of diverse business roles, organizational units, and countries. They offer the ability to design efficient processes by concealing unnecessary fields, representing data in chart formats, or alternation between responsive and grid tables for enhanced data comprehension.

Creation of an SAPUI5 Adaptation Project

The primary advantage of an adaptation project lies in allowing you to leverage the original SAP Fiori application, its OData service, and application logic, to exclusively govern modifications specific to the application variant. Upon deployment, the source application and the derived application variant co-exist under distinct titles, accessible to corresponding business users via uniquely named application tiles in ‘My Home’, the new SAP S/4HANA Cloud Public Edition’s starting point.

See this blog post from Elisabeth Riemann linking to an extensive, end-2-end tutorial.

Did you know that adaptation projects are a part of ‘SAPUI5 flexibility’? Everything you want to know about this toolset is described in our Help Portal.

SAPUI5 Walkthrough Tutorial with TypeScript

Our UI5 team has created this tutorial to introduce all development paradigms of SAPUI5 and also demonstrate the benefits of TypeScript with SAPUI5.

This tutorial provides extensive guidance on key SAPUI5 development principles and data binding concepts through creating an application from scratch. After this, the tutorial guides you through enhancing the app with advanced features like navigation, control extension, responsiveness, and exploring built-in support tools.

Introducing UI5 linter

We have introduced UI5 linter, a static code analysis tool that enhances code quality, readability, security, and efficiency. Its initial features focus on validating compliance with the Best Practices for UI5 Developers, reporting areas of concern like deprecated APIs, synchronous loading mechanisms, and global variables usage. The UI5 linter can be easily installed and used via .npm package manager and is instrumental in preparing your project for future versions of UI5.

Use the UI5 linter to identify problematic code that does not follow UI5 best practices.

Learn more about UI5 linter in a blog post from Oliver Graeff.

That brings us to the end of this update. We hope you found this insight into our recent enhancements beneficial. As we continue to enhance SAP Business Application Studio, we encourage you to explore and leverage these new features to increase your productivity. Thank you for your ongoing commitment to our products, and we ensure that we remain dedicated to improving your user experience. Stay connected for our next round of updates!

Happy coding!

Safeguarding Data Privacy in KSA - Leveraging SAP to navigate NDMO’s Regulations in Digital Era

2024-03-28T12:05:30.971000+01:00

Blog v1.0 | Published On: 28 March 2024

Authors: @asadkhan02 , @AyeshaSafeer , @Zainab_ASalam

_________________________________________

In today's digital era, data privacy is a crucial issue for both individuals and organizations. The Saudi National Data Management Office (NDMO), in partnership with the Saudi Data and Artificial Intelligence Authority (SDAIA), has introduced stringent Data Governance and Personal Data Protection Standards. These regulations mandate all organizations operating across various industries in the Kingdom of Saudi Arabia comply by September 2024.

Failure to comply with these regulations can result in hefty financial fines reaching SAR 3 million or higher in some cases, reputational damage, legal consequences, and loss of trust among customers and partners.

To ensure compliance with these new regulations, organizations are encouraged to implement processes aligned with the 15 domains outlined by the NDMO for Data Governance and Personal Data Protection Standards. Leveraging technology as an enabler, organizations can implement robust data privacy measures and effectively meet these requirements.

Top 5 Areas part of 15 Domains outlined by NDMO for Data Governance and Personal Data Protection Standards

SAP, a global leader in enterprise software, provides advanced technologies with artificial intelligence (AI) capabilities that provide a solid foundation for organizations to implement data governance and personal data protection processes. SAP solutions enable Saudi organizations to efficiently navigate and fulfill regulatory requirements:

By implementing SAP solutions, Saudi organizations can empower their data privacy practices, mitigate compliance risks, and build trust among stakeholders. SAP technologies uphold data integrity, safeguard personal information, offer the framework for regulatory compliance implementation, and adapt to the demands of the digital age.

As a team at SAP, we are committed to supporting organizations in achieving their regulatory compliance initiatives. We invite you to take the next step by exploring how our technologies and solutions can assist you on this journey.

Contact us today to learn more about how SAP can help you navigate the complex landscape of data privacy regulations in Saudi Arabia and ensure compliance.

Elucidation of Associations in CDs Views

2024-04-02T20:20:00.019000+02:00

ASSOCIATIONS:

ASSOCIATIONS are kind of Joins to fetch data from multiple tables on Join Conditions but these are ‘JOINS ON-DEMAND’ i.e. they will only be triggered when user would access the required data which needs the Association of tables. For example, your CDS view has 4 Associations configured and user is fetching data for only 2 tables, the ASSOICATION on other 2 tables will not be triggered and the system would return the results quickly, so it enables really high turn-around time as compared to regular SQL JOINS.
Associations are defined with ‘Cardinality’. Syntax : association[<cardinality>]
Cardinality concept is not new and holds the same concept with CDS views as well. There are 4 types of Cardinality possible based on the data and relationship in the tables joined;

0..1
0..n or 0..*
1..0
1..n or 1..*

NOTE: If you are confused on what kind of association I should configure in my CDS view then you can apply a rule of thumb: ‘always use this : association[1]. This will always trigger an OUTER join and will work in all the cases.
Let’s create a CDS view with ASSOCIATION concept.

Create View with Association template;

Below is the default structure you will get. As you can see the association cardinality is defined as ‘Association[1]’. You can keep it as it is, if you are not sure what association you have to assign.
Check the default Association name as _association_name. This can be any name of your choice but SAP has created a naming convention standard to start this name with an underscore(_) and it is always good to follow SAP standard best practices.

Add the required data elements to complete the view. Lets take the same example we did with JOIN view.

EXPOSED Association

Right click and select ‘Show SQL CREATE Statement;

As you can see, NO Join is created;

We can add separate fields in the CDS view as well like a join and that is called Ad-Hoc Association Concept. In this scenario a join will be performed beforehand.

AD-HOC Association:

Check the SQL CREATE statement. Note that a ‘LEFT OUTER MANY TO ONE JOIN’ is created. The join created because we selected individual fields instead of exposing the whole Association and it is LEFT OUTER MANY TO ONE i.e. *..1 due to the fact that we configured cardinality like ASSOCIATION[1].

As we saw above the ASSOCIATION concept with EXPOSED and AD-HOC Associations. Now lets activate the view with Exposed Association and see how it performs a join at RUN Time.

Activate this view and hit F8 to execute;

Execute and look at highlighted – CDS View name and CARRID data AA and AZ. Also note that NO data from table SPFLI is displayed as NO join is yet performed.

Now if you want to see further Airport information for CARRID ‘AA’ from table SPFLI. right click on any row with CARRID ‘AA’ and choose ‘Follow Association’. Another pop-up window will opn, click the Association type;

Check data: Join is now performed ON-DEMAND and data only for CARRID ‘AA’ is displayed based on Association Key.

In summary, Association concept is one of the wonderful things I liked about CDS views. So intelligently, SAP developed this to save the effort in reading only the information which is required for the analytics purpose. You can feel the real difference after designing a very complex CDS view with a combination of multiple table and other CDS views. The very complex CDS view still performance great when Associations are used. In this blog post, we saw the Association concept together with Exposed and Ad-Hoc associations.

Workload Analysis for HANA Platform Series - 2. Analyze the CPU, Threads and Numa Utilizations

2024-04-02T22:26:10.409000+02:00

In the previous blog, we had discussed the workload pattern. This blog dissects how the CPU, threads and NUMA related topic will influence the workload, and how to figure out the most useful information through the workload analysis. The conclusion can be used for further tuning the HANA system into a good shape on CPU resources consumption.

How to Begin with CPU Utilization Analysis

The analysis of CPU utilization stands as a key element within the broader scope of workload analysis. Determining the right approach to analyze CPU usage is crucial. It's essential to first identify the type of CPU utilization that may be problematic—be it user CPU utilization within the HANA instance, system CPU utilization governed by the OS, or a combination of both. A clear understanding of this distinction is necessary to properly address the issue.

Should the user CPU utilization be significant, it would be prudent to delve into thread information to gain insights into ongoing processes. Conversely, if the focus is on system CPU utilization from the OS, gathering thread data alongside call stack histories is advisable. Collaboration with the OS Administrator might also be necessary to investigate any scheduled jobs or activities that coincide with peak usage times.

In this blog, our primary focus is on user CPU utilization, which is directly impacted by the workload within the HANA instance. A valuable starting point for gaining deeper insights is to analyze thread samples. The monitoring views pertinent to this are:

Although both views contain very similar data, for the purpose of workload analysis, we particularly concentrate on specific columns within these views:

HOST	VARCHAR(64)	Displays the host name.
TIMESTAMP	TIMESTAMP	Displays the timestamp of the record.
THREAD_TYPE	VARCHAR(128)	Displays the thread type.
THREAD_METHOD	VARCHAR(256)	Displays the thread method.
THREAD_DETAIL	NVARCHAR(256)	Displays the thread detail (truncated).
THREAD_STATE	VARCHAR(32)	Displays the thread state.
STATEMENT_HASH	VARCHAR(32)	Displays the unique identifier for an SQL string.
ROOT_STATEMENT_HASH	VARCHAR(32)	Displays the MD5 hash value for the root statement string.
USER_NAME	NVARCHAR(256)	Displays the SQL user name.
APPLICATION_NAME	NVARCHAR(256)	Displays the name of the application.
APPLICATION_USER_NAME	NVARCHAR(256)	Displays the application user name.
APPLICATION_SOURCE	NVARCHAR(256)	Displays that the application can define which source file SAP HANA is called from. The usage is up to the application. This value is also displayed in M_PREPARED_STATEMENTS.APPLICATION_SOURCE.
STATEMENT_THREAD_LIMIT	INTEGER	Displays the effective statement thread limit.
STATEMENT_MEMORY_LIMIT	INTEGER	Displays the effective statement memory limit.
PASSPORT_COMPONENT_NAME	NVARCHAR(32)	Displays the passport component name.
PASSPORT_ACTION	NVARCHAR(40)	Displays the passport action.
NUMA_NODE_INDEX	SMALLINT	Displays the last known NUMA node that the thread was executed on.
WORKLOAD_CLASS_NAME	NVARCHAR(256)	Displays the name of the workload class.

Please find more information in SAP Note 2114710 - FAQ: SAP HANA Threads and Thread Samples

Considerations for CPU Utilization in HANA Workload Analysis

Now that we have the tools at our disposal, where should we initiate our analysis of CPU utilization? Addressing CPU usage within SAP HANA is multifaceted. A high CPU usage, peaking at 90% of total capacity—for instance, with 8 sockets, 208 CPU cores, and hyper-threading—suggests that approximately 374 threads (90% of 208 x 2 logical CPU cores) are active. This scenario often leads to system hang-ups and prevents new user connections. However, this is not the sole concern.

We also face perplexing situations where a mere 50% CPU utilization can cause similar system hang-ups, or times when the system appears idle yet experiences high system CPU utilization. Moreover, there are instances where a single statement monopolizes all threads despite workload classifications and global concurrency limits being in place.

To unravel these complexities, a thorough review of thread samples, supplemented by other monitoring views, is necessary for deeper insight into the system's behaviour.

Typically, the following factors can lead to CPU-related issues in a HANA system:

OS configurations
Expensive SQL statements alongside impractical concurrency settings
Absence of secondary indexes on column tables
NUMA node contentions
System CPU utilization at the OS level
Concurrency controls implemented by applications
Conflicting schedules of background jobs
Activities of the statistics server
Admission control setting
And more.

It's worth noting that high CPU usage doesn't inherently indicate a problem. At times, it may signify that the HANA system is effectively utilizing its capacity to handle the workload, which is not a concern. Our goal is to identify workload contributors to minimize CPU consumption and safeguard the system against potential future issues.

Please find more information in SAP Note 2100040 - FAQ: SAP HANA CPU

Analyzing CPU Utilization Issues: Two Practical Examples

In this section, we simulate two real-world scenarios to provide deeper insights into CPU utilization analysis in SAP HANA environments.

Before diving into the analysis of expensive statements, it's crucial to establish an understanding of the system's configuration that affects workload management. In the examples that we will explore, the system is equipped with :

8 sockets, and each socket houses 52 logical CPU cores.
global.ini -> [execution] -> default_statement_concurrency_limit set to 12
Admission control set to default (indexserver.ini -> [admission_control] -> queue_cpu_threshold = 90)

Example 1 - CPU Spikes Caused by Expensive Statements

Expensive statements in a database context typically refer to queries that run longer than anticipated or consume a significant amount of memory during execution. In the realm of workload analysis, especially concerning CPU consumption, we focus on those statements that utilize a large number of running threads. In our example, various types of statements have been identified that could potentially cause extremely high CPU utilization. These could be so intensive as to lead to system hang-ups.

This situation might involve complex join operations, extensive data aggregation, or poorly optimized queries that put a heavy load on the CPU. Identifying and optimizing these expensive statements is crucial to prevent CPU spikes and maintain system stability. This process involves analyzing query execution plans, reviewing indexing strategies, and possibly restructuring or simplifying the queries themselves to reduce their resource demands.

In the given example, we observe distinct CPU utilization patterns over a span of two weeks on the HANA platform, with utilization nearing 100%. The first week’s pattern is characterized by abrupt spikes in CPU usage, quickly rising and falling, whereas the second week displays more frequent and enduring peaks.

Peak CPUs Lasting for a Few Minutes: Such short-term CPU spikes are not unusual in database environments. They typically arise from intensive data loading, execution of complex queries, or during routine operations like backups or batch processing. However, the spikes observed here warrant close examination due to their extreme frequency and high utilization rates, which could indicate underlying issues needing immediate attention.
Peak CPUs Lasting for Hours: The prolonged CPU peaks, extending over several hours, present a more critical challenge. They could be symptomatic of deeper issues such as inefficient query design, absence of necessary secondary indexes, or even physical hardware constraints. Persistent high CPU utilization of this nature demands a comprehensive analysis to pinpoint the fundamental causes and to devise appropriate strategies for optimization and adjustment to prevent potential system degradation or failure.

It turns out that three major contributors are responsible for the high CPU utilizations within these two weeks.

Statement 1: Executed by user axxx from application source axxx:1, this statement has been stable in production for years. Following a logical adjustment on the application side intended to enhance capacity during request processing, the number of connections initiated by this statement increased. Despite the implementation of a global statement concurrency parameter and a designated workload class, this statement alone occupied 90% of CPU resources when activated. The immediate solution is to curb the number of concurrent connections either at the application level or by refining the HANA workload class parameters. Specifically, adjusting the workload class "wlc_1" to impose a stricter statement concurrency limit is recommended, especially by employing the Hierarchies of Workload Classes feature available in HANA 2.0 SPS 6 and above.
Statement 2: Run by the application user SYSTEM, this activity is associated with HANA's internal optimize compression function, which is automatically calculated and optimized as part of the delta merge operation. Historically, it has been challenging to restrict the CPU consumption of this process. However, with the introduction in HANA 2.0 SPS03, the parameter indexserver.ini -> [optimize_compression] -> row_order_optimizer_threads can now be set to a specific value to moderate the concurrency of this activity effectively.
Statement 3: A select statement executed by user cxxx and application source axxx:3, it has been noted for its prolonged execution time and exceptionally high CPU usage. Thread sampling revealed that most threads were engaged in "IndirectScanVecOutJob<range>" or "ClusterScanBvOutJob<BV>", indicative of extensive table scans. More information on thread methods related to table scans can be found in SAP Note 2114710 - FAQ: SAP HANA Threads and Thread Samples

When look into the detail, we found that table MATDOC and MATDOC_EXTRACT, had lack of secondary index on the key column "MATNR" or "WERKS". Per seconds, the number of column scan can reach to 10 billions+, which is quite significant. Creating a secondary index on this key column should alleviate the CPU strain by optimizing the data retrieval process.

TABLE_NAME	COLUMN_NAME	SCANNED_RECORDS	SCR_PER_S
MATDOC	MATNR	828290874021485	19304757728
MATDOC_EXTRACT	WERKS	109487724317894	10032855641

Example 2 - System Hanging due to NUMA Contention

Sometimes we suffer system hanging situation, but seeing that the overall CPU consumption is not reach to its maximum capacity, or even lower than 50%. This may indicate that the short board is on the NUMA Node (CPU socket) side. The following example showing a NUMA contention analysis from high level, using thread samples. There are also tools to monitor the NUMA awareness from OS level, which gives more details, but thread sample could tell, in the first picture, which contributor may cause the contention issue.

The chart offers a detailed view of the running threads across NUMA nodes in a HANA system equipped with 8 sockets, each hosting 52 logical CPUs. Here’s a refined analysis of the patterns observed and their potential implications:

NUMA Node 4: The count of running threads frequently exceeds the threshold of 52, occasionally spiking to over 95. This pattern suggests excessive parallelism from specific statements or multiple statements accessing the same table or partition. These occurrences appear to bypass the constraints set by global parameters or workload classes, which either lack a total concurrency limit or have a single statement concurrency limit set too high. Such scenarios necessitate a review of the implicated statements to understand the cause of the surge in connections. Repartitioning tables may be beneficial, distributing data across partitions aligned with different NUMA nodes to balance the load. Moreover, the surges in thread count may be attributed to specific thread types like JobWorkers or SQLExecutors. A detailed statement-level investigation is warranted to identify and implement appropriate solutions.
NUMA Node 4: There’s a significant, consistent surge that extends for 30 minutes, signalling NUMA node contention. This is a critical observation as it suggests that the transactions competing for memory resources within this node are likely causing system performance degradation or even system hangs. This could be due to a variety of factors such as the lack of secondary indexes which forces full table scans, or database operations, e.g. optimize compression, that have not been sufficiently controlled. Implementing limitations on internal job concurrency, as prescribed by HANA's configuration settings, might serve as a remedial measure.
Across the remaining NUMA nodes, exceeding the 52-thread limit occasionally doesn’t necessarily indicate a problem; it may reflect the system making full use of the resources at its disposal, which is typically a positive scenario. Nevertheless, it is crucial to ensure that the workload is evenly distributed to prevent any single NUMA node from being overloaded while others are underutilized. Vigilant monitoring for NUMA contention is essential, particularly when the overall CPU capacity is not fully leveraged, to mitigate any adverse effects on system behaviour.

Reducing CPU Utilization on Primary Site by Using Active/Active Read Enable Features

Utilizing Active/Active Read-Enabled (AARE) in SAP HANA can swiftly reduce CPU and memory usage on the primary site. By rerouting read operations to a secondary site—through hints in SQL, ABAP, Procedure or CDS views—resource consumption can be balanced without altering existing job logic. Monitoring the secondary site is vital to ensure it doesn't affect system replication performance. For details on AARE, refer to the official SAP documentation.

🙉🙈🙊

Workload Analysis for HANA Platform Series

This blog post is part of the 'Workload Analysis for HANA Platform Series'. In upcoming posts, we will demonstrate how to analyze the issue related to CPU, threads and NUMA Node . Here's what you can look forward to in this series:

0. HANA Workload Analysis Overview
1. Define and Understand the Workload Pattern
2. Analyze the CPU, Threads and Numa Utilizations (This blog)
3. Identify the Memory Consumption
4. How the Network, I/O and locks influent the workload
5. Performance Impact by HANA Internal Functions
6. HANA Workload Management Methodology

Stay tuned as we explore these aspects in detail, providing insights and strategies to optimize your HANA environment.

Mastering SAP HANA Cloud Vector Engine for RAG-driven LLM Applications: An Insightful Guide

2024-04-08T18:23:37.926000+02:00

Introduction to Vector Similarity in RAG

Retrieval-Augmented Generation (RAG) techniques significantly improve the contextual relevance of large language model (LLM) outputs by incorporating real-time database lookups. SAP HANA Cloud vector engine plays a key role here, enabling efficient retrieval of pertinent data through vector similarity measures, thereby augmenting the performance of RAG tasks.

This tutorial provides a comprehensive walkthrough, using Python, to showcase the utilization of SAP HANA cloud vector engine in a RAG-based LLM application.

Setting Up: Connecting to SAP HANA

Start by establishing a secure connection to your SAP HANA instance.

from hdbcli import dbapi

# Establish a connection. Replace placeholders with actual credentials.
connection = dbapi.connect(
    address="HANA_DB_ADDRESS",  # Your database address
    port="HANA_DB_PORT",        # Typically 443 for HTTPS
    user="YOUR_USERNAME",       # HANA DB username
    password="YOUR_PASSWORD",   # Corresponding password
    charset="utf-8",            # Ensures correct encoding
    use_unicode=True            # Allows for Unicode characters
)

Creating an Embedding Table

For RAG to work, we need to store document vector representations. This involves creating a table in SAP HANA for embeddings.

cursor = connection.cursor()

# SQL statement to create an embedding table
create_table_sql = """
CREATE COLUMN TABLE embeddings_table (
    id INTEGER PRIMARY KEY,     -- Unique identifier for each document
    document NVARCHAR(5000),    -- Text content of the document
    embedding REAL_VECTOR(768)  -- Vector representation of the document
)
"""

# Executing the table creation SQL command
cursor.execute(create_table_sql)
connection.commit()
cursor.close()

Populating the Table with Document Embeddings

Before retrieval, our table needs populating with document embeddings. This involves defining documents, simulating embedding generation, and batch inserting these into SAP HANA.

Defining Documents

# A curated list of documents for embedding
documents = [
    "What is natural language processing?",
    "How do vector embeddings work?",
    "Examples of machine learning applications.",
    "Understanding deep learning for text analysis.",
    "The impact of artificial intelligence on society."
]

Generating Embeddings

Simulate LLM calls for embedding generation. In practice, replace this with actual model interactions.

def get_embedding(document):
    # Placeholder function to simulate embedding generation
    return [float(i) for i in range(768)]  # Returns a fixed-size (768) dummy vector

Batch Insertion into SAP HANA

Efficiently insert the document embeddings into the database.

cursor = connection.cursor()

# SQL command template for inserting document embeddings
insert_sql = """
INSERT INTO embeddings_table (id, document, embedding) VALUES (?, ?, TO_REAL_VECTOR(?))
"""

# Iteratively inserting each document and its embedding
for i, document in enumerate(documents, start=1):
    embedding_str = str(get_embedding(document)).replace(" ", "")
    # Execute insert command for each document
    cursor.execute(insert_sql, (i, document, embedding_str))

connection.commit()
cursor.close()

Executing Vector Similarity Searches

With the database prepared, perform a similarity search to find relevant documents for a given query.

Generating Query Embedding

First, generate an embedding for the query.

query = "What is an example of vector similarity search?"
query_embedding = get_embedding(query)

Conducting Similarity Searches

Use L2 distance and cosine similarity measures to find the most relevant documents.

cursor = connection.cursor()

# L2 Distance search
l2_query = """SELECT TOP 5 id, document FROM embeddings_table ORDER BY L2DISTANCE(embedding, TO_REAL_VECTOR(?))"""
cursor.execute(l2_query, (str(query_embedding).replace(" ", ""),))
l2_results = cursor.fetchall()

# Cosine Similarity search
cosine_query = """SELECT TOP 5 id, document FROM embeddings_table ORDER BY COSINE_SIMILARITY(embedding, TO_REAL_VECTOR(?)) DESC"""
cursor.execute(cosine_query, (str(query_embedding).replace(" ", ""),))
cosine_results = cursor.fetchall()

cursor.close()

The l2_results and cosine_results variables in the code snippet above contain the top 5 outcomes of similarity searches performed on the database, ordered from the most relevant to the least relevant.

Conclusion

This tutorial demonstrates how SAP HANA cloud's vector engine can be utilized in a RAG-based LLM application. The approach enhances LLM responses by ensuring that the generated outputs are informed by the most relevant data.

Reference

Official SAP HANA Cloud Vector Engine Documentation: https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-vector-engine-guide/sap-hana-cloud-sap-hana-database-vector-engine-guide

SAP Datapshereでの HANA Cloud, Data Lake の利用方法

2024-04-12T11:12:57.669000+02:00

はじめに

SAPは、インメモリ、カラムストアという優位なアーキテクチャを前提として設計された SAP HANA およびSAP HANA Cloud をご提供しています。この高速な SAP HANA Cloud のデータ処理機能を、Data Warehouse（DWH、あるいは情報系システムやOLAPと呼ばれるかもしれません）の用途でお使いいただくために、DWHの利用や運用に適した様々な機能をパッケージ化した製品としてSAP Datasphere を提供しています。SAP HANAのインメモリアーキテクチャはすべてのデータをメモリに配置する事で高速な検索処理能力を発揮しますが、一方で相対的にメモリはストレージ（HDDやSSD）よりも高価であり、大規模なデータを扱う場合にはコストも重要な課題になります。SAP HANA Cloudにおけるこのコスト最適化問題の解決策として、Native Storage Extension と HANA Cloud, Data Lakeという機能を提供していることを、過去のブログでご紹介してきました。今回のブログでは、これらの機能をSAP Datasphereで利用する方法についてご紹介いたします。

Native Storage ExtensionとHANA Cloud, Data Lakeについては、それぞれのブログ（HANA Cloud でデータの階層化を実現するNative Storage Extension（NSE)および HANA Cloud, Data Lake First step）をご参照いただくとして、ここでは概略をご説明します。

いずれの機能もSAP HANA Cloudの階層化データ管理を担うための機能です。データベースに格納されるデータは時間の経過とともに利用頻度が低下する傾向にあり、利用頻度の高いデータと、低いデータに対して同じコストをかけて維持するのは得策ではありません。利用頻度の高いデータは高い処理速度で、利用頻度の低いデータはコストに見合った処理性能の領域に格納する事で、コストとシステム性能のバランスを取る事ができます。

SAP HANA Cloud の Native Storage Extensionは、SAP HANAソフトウエアに組み込まれたデータ階層化機能で、「page loadable」という属性を与えた「データの単位」を対象に、データアクセスがなくなるとメモリから揮発し、次回のデータアクセス要求に応じてHANA Cloudのストレージ領域（永続領域）からメモリ領域に配置され、処理されることでメモリ領域のサイズを抑える事ができます。

SAP HANA Cloud, Data Lake はSAP HANAとは独立したData Lake relational engine にテーブルが作成され、HANA Cloud 上の仮想テーブルとして利用されます。Data Lake relational engineはインメモリアーキテクチャではありませんが、DWH用途には最適なカラムストアアーキテクチャを有しており、ストレージベースでありながら、（HANAには及ばないものの）高速にデータ処理を行えるエンジンとなっています。

図１ : 階層化機能　概要図

SAP DatasphereにおけるNative Storage Extensionの利用

前述のNSEに関するブログにあるように、HANA CloudではNSEを利用する対象テーブルに割り当てられたパーティションに対し、SQLコマンドによる操作で「Page Loadable」属性を与える事で、対象のパーティションのデータが階層化管理の対象として機能しました。

SAP Datasphereでは、この設定方法をより簡易に行う事ができる様、管理画面のGUIを使用して設定できるようになっています。一方で、階層化管理の対象の単位は、パーテイションではなく、テーブル単位となっています。また、デフォルトの設定はNSEの機能が有効なウォームストアへの格納となっています。Databuilder の画面から、該当のテーブルのプロパティを表示させ、「テーブルデータをメモリに保存」のスライドボタンを有効にして、NSEの機能を無効化して、パフォーマンスを上げる事ができます。

図２ : テーブルデータをメモリに保存（NSEを利用しない）設定

このことから、Datasphereのテーブルを設定する際は、２つの点を考慮して行っていただく必要があります。

テーブル単位で、アクセス頻度が高いか、低いか
アクセス頻度が高いテーブルであると判断される場合はデータビルダ画面の「テーブルサービス」の設定を「テーブルデータをメモリに保存」フラグをONにする

HANA Cloudでは、パーティション単位でNSEの機能が設定できるため、一つのテーブル（主に時間の経過とともにデータが増加するトランザクションデータのテーブル）に対して最近のデータと過去のデータを分けて階層化管理が可能でしたが、現時点ではSAP Datasphereではパーティションの作成はできますが、NSE機能の設定の粒度はテーブル単位となっていますので、テーブル単位でアクセス頻度が高いか、低いかを指針にして、NSEの機能を活用するようにして下さい。

SAP DatasphereにおけるData Lakeの利用

一方で、多くのお客様が、一つのテーブルのデータに対して階層化管理を行いたいというニーズをお持ちになると想定されますので、その際は以降でご紹介するData Lakeの機能をご検討下さい。本ブログの以下の手順で設定できる構成を以下に示します。

図３: 本ブログにおける Data Lake 構成例

図3では、SAP DatasphereのスペースAにあるメモリテーブルと、Data Lakeに配したストレージベースのテーブルを活用する場合の構成を示しています。この場合でも、オンメモリほどのパフォーマンスではないものの、常にオンラインで利用する事ができる様にData Lakeのデータへは仮想テーブルを介してアクセスできるようになっています。

Open SQLスキーマの利用

最初にご説明した通り、SAP DatasphereはHANA Cloudのデータベースを利用しており、通常のデータはSAP Datasphereから操作をする事で、テーブルなどのオブジェクトの管理が行われています。一方でData Lakeで利用されるテーブルは現在のところHANA Cloud から直接操作する必要があります。そのためにSAP Datasphereの Open SQL スキーマを利用します。図3で示されているように、アクティブに利用するデータは通常のスペースで管理され、Data Lakeに格納するデータは、Open SQL スキーマを介してHANA Cloud経由で利用する、という点にご注意下さい。Open SQLスキーマについての詳細はこちらをご参照ください。

Data Lakeの利用方法

最初に、Data Lakeの機能を利用するための準備設定を行います。左画面のメニューから「スペース管理」を選択します。Data Lakeの機能を使用するスペースの「編集」ボタンをクリックします。

図4：Data Lakeを使用するスペースの選択

続いてスペース管理の画面から「概要」タブが選択されている事を確認し、「データレイクアクセス」の項で「データレイクにアクセスするには、このスペースを使用します」のチェックボックスをチェックし、保存した後、デプロイをクリックします。

図5：Data Lakeアクセスするための設定

ここでの注意点は、Data Lakeにアクセスするスペースは、１つのインスタンスにおいて１つとなります。複数のスペースでData Lakeにアクセスする場合は、Data Lakeテーブルをそれぞれのスペースで共有するようにして利用します。本ブログでは、Data Lakeアクセス可能なスペースでの手順をご紹介します。

続いてOpen SQLスキーマを利用する準備を行います。

左端のメニューから「スペース管理」を選択し、「データベースアクセス」タブをクリックします。データベースユーザにある「作成」ボタンをクリックし、OPEN SQL スキーマにアクセスするためのデータベースユーザを作成します。これはSAP Datasphereとは異なる、HANA Cloudへアクセスするためのユーザになります。

図6: Open SQL Schemaにアクセスするためのデータベースユーザの作成

データベースユーザの作成画面では、「データベースユーザ名の接尾辞」を設定し、「読み込みアクセスを有効化」、「書き込みアクセスを有効化」をチェックして「作成」をクリックし、データベースユーザを登録します。画面が閉じたら、「保存」をし、続けて「デプロイ」を実行してください。

図7 : データベースユーザの登録

図8 : 保存とデプロイの実行

デプロイが完了し、ユーザが追加されたら、そのユーザのチェックボックスにチェックを入れ、行の右端にあるインフォメーションアイコンをクリックします。

図9 : 追加されたユーザの情報確認

データベースユーザ詳細画面が表示されますので、Open SQL スキーマパスワードを取得します。「新しいパスワードを申請」ボタンをクリックし、可視化ボタン（目玉アイコン）を押してから「パスワードのコピー」をクリックし、作成したユーザ用のパスワードを入手します。以後、この画面からは同じパスワードを入手する事はできませんので、必要があれば記録しておいてください。ただし、本シナリオの場合はログインの度に毎回新しいパスワードを生成しても特に問題はありません（ので記録し忘れても大丈夫）。

図10 : データベースユーザのパスワードの取得方法

取得したパスワードを使って、Open SQL スキーマの領域にアクセスします。前述しましたが、Open SQL スキーマとは、SAP Datasphereの管理下ではないHANA Cloud から操作する必要があります。そのため、SQLコマンドを使用することができる、データベースエクスプローラーというツールを介して操作します。作成したデータベースユーザにチェックを入れ、メニューにある「データベースエクスプローラを開く」をクリックします。

図11 : データベースエクスプローラの起動

ログインプロパティの入力画面に先ほど取得したパスワードを入力し、「OK」すれば、Open SQL スキーマを管理するHANA Cloudへのログインが完了します。

図12 : ログインプロパティの入力

ログインに使用したデータベースエクスプローラは、HANA Cloud用のSQL入力用のツールです。このツールを使用して、SAP DatasphereのOpen SQL スキーマで使用しているデータベース領域に直接SQLコマンドを実行する事ができます。以降の操作では、下の図のData Lakeにあるテーブルを作成し、このテーブルをOpen SQL スキーマにある仮想テーブルで参照する設定を行います。

図３: 本ブログにおける Data Lake 構成例（再掲）

まずData Lakeの物理テーブル（図中 ①　SALES_Datalake）を作成します。HANA Cloudに接続したDatabase ExplorerからData Lakeにテーブルを作成する場合 ”DATA_LAKE_EXECUTE" というシステムプロシージャを使用し、Data Lakeを管理するData Lake relational engineに対してSQL（Create table）文を間接的に実行させます。

call "DWC_GLOBAL"."DATA_LAKE_EXECUTE"('CREATE TABLE  SALES_Datalake(
Sales_ID INTEGER,
ShopID	INTEGER,
Purchase_Num	INTEGER,
Prod_ID	INTEGER,
Price	INTEGER,
Order_date date
)');

上記のSQLプロシージャ文を、Database ExplorerのSQLコンソール画面から実行します。左上にあるSQLコンソール起動アイコンをクリックし、実行してください。

図13 : Data Lake テーブル作成

次に、今作成したData Lake内のテーブルを参照する、Open SQL スキーマ内の仮想テーブル（図中②　V_SALES_Datalake)を作成します。仮想テーブルの作成は ”DATA_LAKE_CREATE_VIRTUAL_TABLE" というシステムプロシージャを使用します。

CALL "DWC_GLOBAL"."DATA_LAKE_CREATE_VIRTUAL_TABLE"
	( VIRTUAL_TABLE_NAME => 'V_SALES_Datalake',
      DATA_LAKE_TABLE_NAME => 'SALES_Datalake'
    );

Database Explorerから上記SQLプロシージャ文を実行すると、Data Lakeにある実テーブルを、SAP Datasphereから参照するための仮想テーブルが作成されます。実行完了を確認したのち、左上のペインにある「catalog」フォルダを展開し、「Table」アイコンをクリックすると、左下のペインに作成された仮想テーブルがリストされることが確認できます（図中にもあるWarningに関しては、本シナリオでは特に影響はありません）。

図14 : Open SQL スキーマ仮想テーブル作成と確認方法

作成した仮想テーブルに対して挿入および更新するための権限を SAP Datasphere のスペースに付与します。これにより、SAP Datasphereのデータフローによってテーブルにデータを挿入する権限を与えます。次の手順で、スペースにすべての権限を付与します（JAPAN_D&Aは実施した環境のスペース名です、適宜ご自身のスペース名に置き換えてください）。

GRANT ALL PRIVILEGES on "JAPAND&A#YAGAWA"."V_SALES_Datalake" 
to JAPAND&A with grant option

図15 : 仮想テーブルに権限付与

以上で、Data Lakeに物理テーブルを作成し、Open SQL スキーマにこれを参照する仮想テーブルが作成されました。それではSAP Datasphereからこの仮想テーブルを操作してみましょう。

Data Lakeに作成したテーブルに、SQL Open スキーマにある仮想テーブル経由でデータをインポートするため、ソースとなるCSVファイルを、AMAZON AWS S3に配置し、これをDatasphereのデータフローを使ってデータを格納します。AWS S3との接続設定に関しては、こちらのブログをご参照ください。このブログに基づき、Import_Location_from_S3 という接続設定を作成しておきます。また、接続先のAWS S3 には、インポートファイルをアップロードしておいてください。サンプルとして SampleData_Datalake2024.csv を添付します。

ブラウザのSAP Datasphereのタブを選択し左のメニューから「データビルダ」を選択、「フロー」タブをクリックし、「新規データフロー」タイルをクリックします。

図16 : AWS S3からCSVファイルをインポートするデータフローの作成

新規データフロー作成画面にて、左のペインから「ソース」タブをクリックし、「接続」フォルダを展開すると、上記で準備した Import_Location_from_S3がリストされます。これを展開すると、アップロード済のファイルがリストされます(下の画面では、AWS S3に作成したフォルダ dspimportlocation の下)。この中からData Lakeに配置するためのデータ SampleData_Datalake2024.csv　を、Drug ＆Dropし、中央のワークスペースに配置します。

図17 : AWS S3に配置したCSVをソースとして定義

つづいて「リポジトリ」タブをクリックし、作成した仮想テーブル「V_SALES_Datalake」をクリックし、ワークスペースにDrug & Dropします。この時同時に表示されるコンテキストメニューから「ターゲット」を選択します。

図18 : 仮想テーブルをターゲットとして定義

ワークスペースのソースオブジェクト（AWS S3ファイル）の黒い　●　を、ターゲット（V_SALES_Datalake）にDrug ＆ Drop で結線し、ターゲットオブジェクトをクリックしたのち右のペインのプロパティ画面を下方にスクロールして、ソースとターゲットの項目をマッピングします。こちらも左の項目をDrug ＆Dropすることでマッピングする事ができます。なお、ソースのCSVファイルの１行目にヘッダー行としてターゲットテーブルのカラム名を記入しておくと、自動マッピングする事も可能です。

図19 : ソースとターゲットの関連付け

設定したデータフローを左上のアイコン「保存」し（データフロー名を指定します）続いて「デプロイ」します。ワークスペースの中央下に「Deployed...」のメッセージが表示されたらデータフローの準備は完了です。このメッセージは自動的に消えますのでご注意下さい（しばらくすると右ペインのプロパティの表示が「未デプロイ」から「デプロイ済み」に更新されるので、こちらで確認可能です）。

図20 : データフローの保存とデプロイ

このデータフローにより、AWS S3にアップロードされたCSVファイルを、Data Lakeにインポートできるようになりました。このワークフローをスケジュールに基づき実行させる場合は、右ペインのプロパティの「実行ステータス」にある「スケジュール」アイコンをクリックし、スケジュールを作成する事ができます。ここでは、すぐにデータフローを実行したいので、左上にある「実行」アイコンをクリックし、データフローを実行します。

図21 : データフローのスケジュール実行と直接実行方法

データフロー完了のメッセージ（DIRECT run of DATA_FLOW .... completed このメッセージは自動で消えます）を確認するか、右ペインのプロパティにある実行ステイタスで「完了」を確認したら、ターゲットオブジェクトをクリックし、表示されるコンテキストアイコンから「データのプレビュー」をクリックすると、下方にこのオブジェクトのデータを確認する事ができます。このオブジェクトは仮想テーブルであり、その実態はData Lakeを参照しているので、これでData Lakeにデータをインポートしたことが確認できました。

図22 : データフローの実行確認と、その実行結果の確認

SAP Datasphereにおける Data Lakeの利用方法については以上となります。

SAP DatasphereにてData Lakeを利用したときのパフォーマンスに関する情報として、以下のブログが参考になりますので、リンクをご紹介しておきます。

Working with the Embedded Data Lake in SAP Datasphere

Why Don't We Use Data and Analytics Group on SAP Community?

2024-04-18T18:28:14.375000+02:00

Hey Data and Analytics Folks! Why Don't We Use Data and Analytics Group on SAP Community instead of Technology?

Of course, it does not matter at the end. As long as if you tag your posts with SAP Managed tags everybody can find your posts. But there is a really one huge difference for usability that here in Interests Groups we can reply-the-reply but under Products and Technology we cannot! Check out this blog post SAP Datasphere is ready to take over the role of SAP BW by @kenneth_dalvik and all responses that are very all boring sequential --there is no reply-the-reply option! 🙃

So let's start using this group Here is already available Data and Analytics. The group will be definitely a perfect fit for SAP Analytics Cloud SAP Datasphere SAP Analytics Cloud, hybrid analytics SAP HANA Cloud, SAP HANA database SAP Analytics Cloud, connectivity SAP Analytics Cloud, analytics designer SAP S/4HANA Cloud, extended edition SAP S/4HANA Embedded Analytics SAP Business Planning and Consolidation, version for SAP NetWeaver SAP Business Warehouse Accelerator BW (SAP Business Warehouse) SAP BW/4HANA BW SAP BEx Analyzer BW SAP BEx Web SAP ASE - BW Enablement BW SAP HANA Data Warehousing SAP Data Services SAP Data Intelligence SAP Data Custodian SAP Big Data Services SAP BusinessObjects - Platform Administration SAP BusinessObjects - Semantic Layer SAP BusinessObjects Mobile SAP BusinessObjects - Semantic Layer - SDK SAP BusinessObjects - Web Intelligence (WebI)

These are welcome too SAP Master Data Governance SAP NetWeaver Master Data Management SAP Data Quality Management for SAP NetWeaver MDM

One thing I've noticed we cannot submit the blog posts directly. We need to "Submit for Review" only. You can still post discussions aka questions directly.

Thanks owners @ColinC, @meganhoy, @moshenaveh, @caroleighdeneen, @craigcmehil for their support!

Regards,
Tuncay Karaca

Understanding Data Modeling Tools in SAP

2024-05-02T16:25:43.459000+02:00

Choosing the right SAP data modeling solution can be a challenging task, as there are multiple options available, including cloud-based or on-premise solutions with or without data visualization capabilities etc..so to select the best tool that can meet the project requirements and specifications, several factors need to be considered including, reviewing the raw data source and understanding how it will be used, assessing the need for data warehousing capabilities, evaluating the team's skill sets, and deciding whether a cloud or on-premise solution is preferred.

During the last few years, I've worked with almost all the SAP Data Modeling tools that are available, and In this blog article, I would like to briefly highlight the strength of each tool and where would be the best fit for it 🤗.

Understanding Data Modeling

so what it's data modeling in SAP and why it's important? 🤔

when we talk about Data modeling we are defining the process of creating diagrams that show how data flows in and out of a database. These diagrams define the data formats, structures, and functions required to support the data flow. Once the database is built, the data model becomes documentation for its existence and design. Data models provide a framework of relationships between data elements within a database. They enable different applications to share the same data and create a simple and logical database that eliminates redundancy, reduces storage requirements, and enables efficient retrieval.

SAP HANA Calculation Views

SAP HANA calculation views are the key data modeling object on HANA, it's a graphical object that describes all the data processing steps from raw data to the final, consumable output.

Common data processing steps include filtering data, combining data, aggregating data, calculating new values, converting values, and ranking data. Calculation views are built in layers so that each calculation view can consume other calculation views to form a stack of super-models. This approach promotes the reuse of calculation views at the lower levels.

Strengths:

In-Memory Processing: Data processing occurs directly in memory, ensuring high performance.
Graphical Interface: Intuitive tools make it easy to create and visualize models.
Custom Logic: Ability to integrate custom SQL for tailored data processing.
Advanced Analytics: Supports spatial, graph, text analytics, and predictive/machine learning.

Best Fit: Developing flexible data marts with custom logic requirements.

SAP BW/4HANA

SAP BW/4HANA is an on-premise data warehouse. The main purpose of a data warehouse is to extract data from any data source, integrate the data, store the cleansed data, and provide modeling tools so that data models can be created ready for use by any analytical tool. This is exactly what SAP BW/4HANA can do.

Strengths:

Ready-Made Models: Comprehensive data models are provided for virtual and physical storage.
Data Warehousing Features: Supports dimensional modeling, hierarchies, and efficient data storage.
Graphical Tool: User-friendly interface for efficient model visualization.
On-Premise Deployment: Ideal for industries requiring high security and data control.

Best Fit: Establishing on-premise data warehouses with strong data integration needs.

ABAP CDS Views

ABAP CDS Views enable developers to create data models that are semantically rich. These models are then exposed to UI clients by the application services. CDS Views is the central pillar of S/4HANA development and is used as the core technology in most of SAP's programming models.

Strengths:

High Performance: Data processing is pushed down to the database level, ensuring efficiency.
Ready-Made Models for SAP S/4HANA: Simplifies operational reporting on S/4HANA.
Flexible Development: Allows for code-based customization leveraging SQL skills.

Best Fit: Operational reporting on SAP S/4HANA with ready-made content.

SAP Analytics Cloud

SAP Analytics Cloud is an end-to-end cloud solution that brings together business intelligence, augmented analytics, predictive analytics, and enterprise planning in a single system. It is a solution aimed at business users for self-service analytics.

Strengths:

Integrated Modeling: Modeling seamlessly integrated with reporting tools for streamlined analytics.
Cloud-Based: No need for software installation; data modeling is conducted in the cloud.
Self-Service Analytics: Empowers business users to create their own models.

Best Fit: Organizations seeking self-service analytics with integrated modeling and reporting.

SAP Datasphere

which is so far My Absolute favorite tool 😁! It is a public cloud software as a service (SaaS) data warehouse that supports the business data warehouse architecture. This tool effortlessly harmonizes data across the enterprise, making it a must-have for any business. It includes services for data modeling, data integration, cataloging, and data warehousing across SAP and non-SAP data, making it a one-stop-shop for all data management needs

Strengths:

Comprehensive Data Warehouse Features: Supports dimensional modeling, data history management, and hierarchy modeling.
Scalability: Easily scale resources in the cloud, supporting new innovations.
Cloud Service: Simplifies software management through cloud consumption.

Best Fit: Cloud-based data warehousing with advanced features and scalability.

SAP BusinessObjects BI Semantic Layer (Universes)

An organized collection of objects that called a Universe, which consists of dimensions, measures, and attributes grouped by business topics. The skill of a Universe developer is to create objects that resemble business terms by mapping them to the columns of the database tables. In some cases, SQL coding may be required to generate the value of the objects.

Strengths:

User-Friendly Interface: Graphical tool simplifies data modeling tasks.
SQL-Based Development: Easy to understand for database developers.
Mature Tool: Offers a wide range of features for diverse modeling needs.

Best Fit: Supporting SAP BusinessObjects reporting tools with Universes.

for more information, I would recommend this Learning journey from SAP which will give you an overview of all the SAP data modeling solutions 😊

Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive

2024-05-08T17:00:00.007000+02:00

Introduction

This blog post is part of a series describing the usage of MLflow with HANA Machine Learning co-authored by @stojanm and @martinboeckling. In this blog post we provide a more technical deep dive on the setup of a MLFlow instance and provide a general introduction how Machine Learning models trained with HANA ML can be logged with MLflow. The first blog post of the blog series is called Tracking HANA Machine Learning experiments with MLflow: A Conceptual Guide for MLOps and gives an introduction to the topic of MLOps with MLflow.

Starting with the python HANA ML package version 2.13, HANA Machine Learning added support for tracking of experiments with the MLflow package, which makes the incorporation of models developed using HANA Machine Learning into a comprehensive MLOps pipeline easy to achieve.

In this blog post we will provide an overview how MLflow can be used together with HANA ML. MLflow, which manages the experiment tracking and artefact management can run as a managed service at a hyperscaler platform, deployed locally or on a remote infrastructure. In the following we describe how to deploy MLflow on SAP Business Technology Platform and how to track your HANA machine learning experiments with MLFlow. In addition, we present which methods and algorithms in the hana ml package currently support the experiment tracking feature. Finally, we touch on the possibility to use logged models in MLflow for prediction.

Prerequisites

In this blog post we solely focus on the technical integration of HANA ML and MLflow as a logging platform. Generally, we assume that Python is already installed together with an already established development environment. Furthermore, we will not completely explain all details of docker and Cloud Foundry, but simply focus on the essential parts for HANA ML and MLFlow within this blog post.

Set up MLFlow on BTP

MLFlow is leveraged and integrated in different solutions. For example Databricks as well Machine Learning in Microsoft Fabric integrate or provide a managed MLflow instance. In case MLFlow is not yet provided, we outline in this section a possibility to deploy MLflow in SAP BTP. We focus for simplicity reasons on the SQLite based deployment of MLflow. However, for productive environments it is recommended to separate the storage from the runtime of the MLFlow instance. A detailed explanation for setting up your own MLFlow server with alternatives to SQLite can be found under the following link: https://mlflow.org/docs/latest/tracking/server.html. In the following paragraphs, we focus on a step by step overview to set up your own MLFlow instance using SQLite in BTP.

As a first step we want to create a local docker file which can be used to upload it to our BTP environment. In the following code snippet, we provide the coding used to construct your own docker container locally. For that, paste the following code into a file called Dockerfile within your desired local folder.

# Retrieve a python version as a base runtime for our docker container
FROM python:3.10-slim
# Run the pip install command for the package of mlflow
RUN pip install mlflow
# Create a temporary folder within our docker container to store our artifact
RUN mkdir -p /mlflow
# Expose the port 7000 to make our application which runs within docker accessable over the defined port
EXPOSE 7000
# Define environment variables BACKEND_URI and ARTIFACT_ROOT to define the backend uri as also the 
# artifact root
ENV BACKEND_URI sqlite:///mlflow//mlflow.db
ENV ARTIFACT_ROOT /mlflow/artifacts
# run the shell comand to setup our mlflow server within our docker container
CMD mlflow server --backend-store-uri ${BACKEND_URI} --host 0.0.0.0 --port 7000

After creating successfully the Docker object, you can run docker build -t {tagname}. to construct your docker container. Afterwards, the local docker image is locally built. To expose the docker image, we push the image to a docker registry. In our example, we assume that you already have a docker registry set up where you can push your image to. For that step, you can run the following commands: docker tag {tagname} {dockerhub repository tag}, docker push {dockerhub repository tag }. After the successful run of the command, you see within your private docker hub the newly published docker container, which contains MLflow and all its dependencies inside of it.

After the successful publishing of your docker image to your registry, we can run the following command to create a BTP app based on the published docker image:

cf push APP-NAME --docker-image REPO/IMAGE:TAG

After successfully publishing the docker image on our BTP Cloud Foundry environment, we can find our published app within our BTP account and are able to access it under the published URL.

Set up tracking for MLflow

With MLflow users have the possibility to track their trained HANA ML models. In the following paragraph, we introduce the aspects that are needed to be able to log HANA ML models into MLflow itself.

To be able to use MLFlow together with HANA ML, we need to first install besides the HANA ML package also the MLFlow package. Therefore, you need to run the following command in your virtual environment, to be able to run the following scripts.

pip install mlflow hana-ml

As a general setup, we first need to run the following command to set up our tracking with MLFlow to our available MLFlow instance. Therefore, place into the following two lines first your personal MLFlow tracking URI and your own custom experiment. In case you do not want to create a separate experiment, the different runs together with the MLFlow model are stored under the default experiment.

The method that allows us to track HANA ML models is implemented in the HANA ML package and is called enable_mlflow_autologging(schema=None, meta=None, is_exported=False, registered_model_name=None). This method can be used for initialised HANA ML models that are under the following methods:

Within the method enable_mlflow_autologging the user has different keywords that can be filled that allows us to influence the behaviour of our MLFlow autologging in HANA ML.

schema: Defines the HANA database schema for MLFlow autologging where the MLflow logging table is stored
meta: Defines the name of the model storage table in HANA database
is_exported: Determines if the hana model binaries should be exported to MLflow
registered_model_name: Name of the model stored in MLflow

In the following section we provide an overview for the Unified Interface how the logging of MLflow can be used.

Run HANA ML Algorithms with MLflow

As we have explained and outlined in the sections above, we have created a MLFlow instance and have introduced the syntax that is needed for the logging of HANA ML models in MLflow. In the following sections we will provide based on an example how the logging of HANA ML models on MLflow is done.

Model training of HANA ML with MLFlow

For the training of HANA ML in combination with MLflow, we focus in this blog post on the Unified Method. We apply for the respective elements a Classification on the sample bank dataset which can be found under the HANA ML sample dataset folder on GitHub.

The dataset can either be uploaded directly to the SAP HANA database or you could also use SAP Datasphere as your starting point. Generally, to use HANA ML directly you would need to store the dataset in a HANA database. However, HANA ML also provides methods to integrate third party files/ data structures. This involves Pandas, Spark as also shapefiles. In addition also HANA Data Lake file tables can be integrated with HANA ML functionalities. An overview of the different methods can be found under the following page. In the following paragraphs, we will go through the sample code that we have created to combine HANA ML and MLFlow.

Connect to HANA database (Deployed under SAP Datasphere)

To be able to connect to the HANA database instance, we first need to build up a connection to the HANA database. In our example, we load the data from the data samples provided by HANA ML. During the time of this blog post, the OpenSQL schema of Datasphere only supports Basic Authentication. Therefore, in this blog post we only elaborate how the connection is done over basic authentication. SAP HANA standalone supports however non-basic authentication, which are also supported in the HANA ML package to connect certificate based to the SAP HANA instance.

To establish the connection to the HANA database, we make use of the implemented HANA ML dataframe class and call the method ConnectionContext. We store the instance of the connection in the variable conn. To now be able to establish the connection to the HANA database view or table, we will need to specify over the method table the connection. The beautiful aspect is that overall, the dataset is not going to be loaded to the Python runtime, but will only be represented with a proxy to the actual table in the HANA database. All transformations, if done over the methods of HANA ML, are then pushed down to the database itself and executed there if the training is executed. In our case, we load the sample dataset into our database by making use of the provided methods of the HANA ML package.

"""
This script provides a short example how HANA ML and MLFlow can be integrated together.
The credentials to the database are following the currently supported authentification (Basic) of the 
SAP Datasphere OpenSQL schema. Overall, HANA Cloud standalone is also able to support multiple other authentification
methods. We have used an abstraction python file (constants) where we retrieve the securely stored 
authentification properties. To get more details about the exact method structure needed, please
have a look at the documentation:
https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.07/en-US/hana_ml.dataframe.html#hana_ml.dataframe.ConnectionContext
"""
from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.algorithms.pal.auto_ml import AutomaticClassification
import mlflow
from hana_ml.algorithms.pal.auto_ml import Preprocessing
from hana_ml.algorithms.pal.partition import train_test_val_split
from constants import db_url, db_user, db_password
# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url, port=443, user=db_user, password=db_password)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

After connecting to the database, the user is able to use the preprocessing methods implemented in HANA ML. Generally, the different changes are pushed down to the HANA database and are not executed within the Python runtime. In our use case, we do not need to use the data preprocessing as we directly retrieve a sample dataset which we can directly use for our ML training.

After finishing the potentially needed transformations, we are now able to implement the tracking of our HANA ML runs with the possibility of MLFlow. Similar to the normal usage of MLFlow, we set up first our tracking uri under which we want to store our HANA ML runs and models. In your case you would need to change the keyword mlflow_tracking_uri with your respective MLflow tracking URL. Furthermore, we then are able to specify the experiment name under which the runs are tracked. If we do not specify a specific experiment, the runs are tracked under the Default experiment.

# set up MLFlow
mlflow.set_tracking_uri(mlflow_tracking_uri)
mlflow.set_experiment("HANA ML Experiment")

In the following chapters, we will provide an outline how the exact training is performed and what components are logged to MLflow.

Unified Method

For the example we use the implemented Hybrid Gradient Boosting Tree as a classification algorithm for our Classification. In order to perform the classification, we use the Unified Classification in order to be able to run our algorithm. On the defined variable, we then use the implemented enable_mlflow_autologging method. This allows us to directly log the model using implemented auto logging behaviour.

uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging() uc.fit(training_data, key="ID", label="responded")

We call the fit method once we have initiated the HANA ML model variable and the associated autologging for MLflow. For the fit method, we have in total two different options. Firstly, the non-partitioned training dataset where we only use the training dataset. If we decide to partition our training dataset, we allow to create a validation dataset for which we can log metrics automatically during training.

If we do not define for our fit function the partitioning, we will not log metrics within MLFlow. In the following image, you can see how a potential HANA ML tracked run looks like in MLFlow together with the stored HANA ML model in MLflow.

If we decide to partition our dataset, here for instance to partition the dataset along the defined primary key, we are able to directly log evaluation metrics relevant for the Classification we have used. This includes the following metrics: AUC, Recall, Precision, F1 Score, Accuracy, Kappa coefficient and the Mathews Correlation Coefficient (MCC). This would directly allow us to compare multiple runs within our MLFlow project to one another and measure the different performances.

In addition to the general run, HANA ML also logs the model to MLflow. What is logged to MLflow depends on the parameters set for the method enable_mlflow_autologging. If for instance everything is set to the default settings, we will see the following yaml file to be logged to MLflow.

If within the method enable_mlflow_autologging the parameter is set to is_exported, the model binaries stored in the model storage on HANA are exported to MLflow. This setting would allow us to retrieve the trained model from MLflow and use it in a different HANA database for prediction purposes. In addition to the yaml file containing the metadata we now can see a created subfolder called models which contains the necessary model artefacts normally stored in the HANA database now in MLflow.

After the training is finished, we have besides the auto logging capabilities of HANA ML for MLflow the possibility to track further artefacts in MLflow. In the following section we will outline a few possibilities that exist with the additional tracking.

Additional logging possibilities

Besides the outlined auto logging capabilities, we can track with MLFlow additional artefacts to the respective run. In the following chapters, we outline selected possibilities to further enrich the auto logging for HANA ML runs tracked in MLFlow.

Adding run and experiment description

The description in the experiment section can be handy once the number of your experiments grows in the repository. In addition, mlflow allows to also add individual description to each run of an experiment. Using the following methods you can set up both:

from mlflow.tracking import MlflowClient
current_experiment=dict(mlflow.get_experiment_by_name("HANA ML Experiment"))
experiment_id=current_experiment['experiment_id']
    
run = mlflow.active_run()
MlflowClient().set_experiment_tag(experiment_id,"mlflow.note.content",
"This experiment shows the automated methods of HANA machine learning and how to track them with MLFLOW")
MlflowClient().set_tag(run.info.run_id, "mlflow.note.content", "This is a run tracked with Unified Classification from HANA Machine Learning")

Logging input datasets

Sometimes it is important to keep the input dataset also as part of the tracking with MLflow. Since HANA machine learning datasets are located in HANA, they need to be converted to pandas DataFrames to be tracked as shown in the following code:

# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

This results in the change, that the respective state of the training data is logged to the current run. The logged dataset can be found in the associated MLflow run, where the schema of the dataset is provided together with some metadata information about the number of rows and number of elements. In addition, also the provided context is marked in the UI of MLflow.

Logging a model report

In addition to the logging of the dataset, it might also be important to add a model report to MLFlow. HANA ML generally provides different interactive visualisations for the trained model artefact, which can be stored as an HTML file. After the storing of the model report to your local repository, we can log the input of the model report to our current run. This allows us to interactively explore the model report automatically generated by HANA ML and make it accessible in MLFlow. To log the HANA ML model report, you can use the following code snippet.

# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")

After the Model report is stored successfully under the current run, we can see in the artefact tab in MLFlow the interactive model report:

The complete script used for this section can be found in the following code snippet:

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLflow
mlflow.set_tracking_uri(tracking_uri)
mlflow.set_experiment("HANA ML Experiment")
# set up classification
uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging(is_exported=True)
# train model
uc.fit(training_data, key="ID", label="LABEL", partition_method="stratified", stratified_column="ID", partition_random_state=43, build_report=True)
# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")
# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

Apply of trained model

After we have finished our training, we are able with HANA ML to retrieve the model from MLFLow and use it for our prediction purposes. For this purpose, we will create a separate Python script where we will provide an overview to retrieve the trained MLflow model.

Similar to our training script, we first set up our connection to the HANA database and establish the connection to our table. In our case, we simply use the sample dataset provided by HANA ML.

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

Similar to our training script, we need to set the tracking url for MLflow and need to initiate the model storage of HANA. If we have decided to not export the HANA ML model to MLflow, we need to specify the same schema for the model storage where our HANA ML model is stored after the successful run. In case we have exported our model, we are able to specify a different schema. In the following, you can see the necessary script in order to retrieve the logged HANA ML model from MLflow.

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_url)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)

After the model storage has been initiated, we are able to retrieve the stored HANA ML model from MLflow. In order to select the correct model, you need to extract the correct run id associated to the model you would like to apply for your prediction dataset. In our case, this is the test dataset we have received from the sample dataset method. The model_uri needed for the model retrieval is consisting of the following pattern 'runs:/{run id}/model', in which you would need to exchange the run id with your respective run. For the actual retrieval of the model, we use the initiated model storage, in our case called model_storage and call the method load_mlflow_model to load the MLflow model to our HANA database and assign the respective proxy to our variable mymodel. The variable mymodel is then used to call the predict method in order to apply our model to our dataset. In the end we transform our prediction dataset into a Pandas DataFrame to look at the content of the created DataFrame. Normally, we could directly persist the created temporary table with the save method and therefore make the dataset available for further processing.

# load logged run from MLflow to HANA ML
logged_model = 'runs:/d8a763b7b81940598633605e447cd880/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

After running the script, you should be able to see the following terminal output, for which we can see the download of the artefact stored in MLflow and the created prediction dataset, which consists in our case of 4 columns: ID (primary key), SCORE (predicted label), CONFIDENCE (prediction confidence for applied row) and REASON_CODE (influence of individual variables to prediction output).

In case we have exported our model, the output of our terminal look slightly different indicating that we also download the respective model artefacts stored additionally to the yaml file. In the following you see the complete script used for applying the model to a new dataset.

from hana_ml import dataframe
from hana_ml.model_storage import ModelStorage
from hana_ml.algorithms.pal.utility import DataSets
import mlflow
from constants import db_url, db_user, db_password

conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

# full_set, diabetes_train, diabetes_test, _ = DataSets.load_diabetes_data(conn)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_uri)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)
# load logged run from MLflow to HANA ML
logged_model = 'runs:/ed7b8d4734cb42ca90c417f932957b40/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

Key take aways

In this blog post we have showcased an end to end example how MLflow can be integrated in your HANA ML workload by providing the possibility to share and compare multiple tracked runs in MLflow. If the data is already stored in HANA, this allows you to directly interact with MLflow while being able to run your Machine Learning algorithms on data stored in the HANA database without the need to transfer your data between multiple systems. This blog covered an essential part of the automated logging capabilities of HANA ML models into MLflow.

We highly appreciate your thoughts, comments and questions under this blog post. In case you want to reach out for general questions around HANA, or specifically HANA ML, don't hesitate to use the Q&A tool with the respective tags that describe your question.

Application Development using React and CAP – Deployed in BTP

2024-05-23T09:02:28.846000+02:00

Objective: Develop a gamification application for the SAP Customer Success event held on April 24 & 25.

Key Features: Event agenda for two days of the event; Two types of quizzes - one consisting of general SAP questions, second consisting of Partner Question; Leaderboard consisting of top 15 people of the quiz; QR code scanner for session codes and points; User onboarding via IAS self-registration; My Points session consisting of all the points secured during the games.

Technologies Used: Frontend: React; Backend: CAP; Data Storage: S4; Deployment: SAP Business Technology Platform (BTP); User Management: SAP Identity Authentication Service (IAS)

Development Process:

Initial Setup:

BAS Studio: BAS

Database Explorer: SAP HANA

BTP Subaccount : Training Subaccount

1. Frontend Development with React:

- Developed the user interface using React.

- Designed mock UI screens using Figma Click to view

- Followed the SAP community blog for React application development Click to see the blog

- Learn more from React Tutorial with UI5 Webcomponents

- Created routing in the React application using SAP's tutorial Routing Guide

2. Backend Development with CAP:

- Decided on CAP for backend development after initial research.

- Set up the database and CDS (Core Data Services) in CAP.

- Deployed the backend on BTP following the CAP jumpstart guide CAP documentation, youTube Tutorial

3. Data Storage Integration:

- Integrated data storage using S4 Refer this article

- Faced challenges integrating the front and backend, which were resolved through research and community support.

User Onboarding:

- Implemented user onboarding with self-registration through IAS.

- Followed the SAP Help Portal guide for IAS setup IAS User Guide

Final Integration and Deployment:

- Successfully deployed a mock version of the application to a BTP subaccount.

- Completed the initial development of the React application within 18 hours & deployed it.

Testing and Launch:

- Delivered the completed application to the team for a demo to the RVP.

- Addressed minor bugs during testing - fixed them.

- Launched the application successfully a week before the event day.

Outcome:

- Over 200 customers participated in the event.

- The application got positive feedback from the end-users(Customers) and was well-received during the SAP BTP Customer Success Day event.

- The top three participants were awarded prizes from SAP Partners.

Ultimately, working on this application was a great experience. As the lead developer for the first time, I experienced various ups and downs. I strongly believe that it's okay to make mistakes as long as we are ready to analyze and fix them.

APP DEMO:

App DemoQR Code Scan

End-To-End performance optimization in SAP Analytics Cloud with SAP Enterprise Support Services

2024-05-24T11:48:31.997000+02:00

Recently I posted a blog about SAP Analytics Cloud: Continuous Quality Checks (CQC services).

The overall performance of SAP Analytics Cloud (SAC) is mainly determined by the three components:

Client time - frontend
Network time
SAP Analytics Cloud service time - backend

It is important to regularly monitor and analyze the performance of SAP Analytics Cloud to identify and address any bottlenecks or inefficiencies.
The CQC services in my blog above offer great value in assisting you to analyze your system and/or situation based on real life data from your SAP Analytics Cloud Stories or Analytic Applications etc.
They do primarily focus on the frontend and network aspects.

In many cases the frontend can be the smaller portion of the total build time of a Story in SAP Analytics Cloud. When connecting to live SAP data sources you must ensure that these have been optimized for reporting usage. This can be achieved by optimizing database queries, tuning configurations and parameters, and scaling resources to handle increasing user loads. Additionally, implementing caching mechanisms, utilizing efficient data storage and retrieval techniques etc.

Getting to the Root Cause

To begin with you should perform some root cause analysis to narrow down where time is being spent.
Take a look at these useful resources and tooling within SAC

2511489 - Troubleshooting performance issues in SAP Analytics Cloud
SAP Analytics Cloud Performance Decision Tree
SAP Analytics Cloud – Administration Cockpit
SAP Analytics Cloud Performance Statistics and Analysis
SAP Analytics Cloud Data Action Performance Statistics and Analysis
SAP Analytics Cloud - Performance Measurement

Once you establish the total build time and how much time is spent in the three components, you can follow the guidance in the links above to work on improving and optimizing each part.

SAP Enterprise Support Services - Backend

If you encounter a situation where you have performance impact coming from an SAP source system, such as SAP BW or SAP HANA, one option available to you is to engage with us and consume your entitlements from SAP Enterprise Support. We deliver specific recommendations via our CQC checks to help you fine tune these data sources.
Especially relevant for SAP BW and SAP HANA DB are:

Technical Performance Optimization (TPO)

The TPO service is designed to be an overall health check focusing on throughput of your database.
Some of the checks included in the service include:

Checks for major misconfigurations and hardware bottlenecks
Parameter settings, table space, expensive SQL statements
Database management (CBO statistics, Index and table storage management, Online log configuration)

For more details read the infosheet.

Business Process Performance Optimization (BPPO)

The CQC for SAP Business Process Performance Optimization is designed to ensure best performance for business-critical steps in your solution. It is more relevant for specific issues like modeling.
Some of the checks included in the service include:

Detailed technical analysis of critical process steps, evaluates and identifies potential improvement areas.
Investigation of long running process chains / failed process chains.
Investigation of long running BW queries
Parameter checks of the system for both hardware/software and suggestions for performance improvement

For more details read the infosheet

Further Support and Empowerment via SAP Enterprise Support Value Maps

Value Maps are a support and empowerment program from SAP Enterprise Support and are available to you via the SAP Learning Hub. A one-time sign in with your S User is required, and everything thereafter is free as part of your entitlements. We provide:

Prescriptive Guidance
Empowerment
Social Collaboration
Expert Access

For Data and Analytics we provide all of the services and support outlined above, and much more, as part of our Learning Journeys. This includes SAP Analytics Cloud, SAP Datasphere, and SAP BW topics.
We have dedicated Learning Journeys for SAP Analytics Cloud, SAP Datasphere, and SAP BW topics.

Check out the SAP Enterprise Support value maps landing page for more and start engaging today!

Quicklinks

SAP HANA - Fiori: Add Apps to Business Roles which are not in a Business Catalog.

2024-05-24T15:00:04.449000+02:00

Hi All.

This blog is for newcomers to SAP HANA and SAP Fiori . This blog shows the basic steps to add Fiori apps which are not available in a business catalog to a business role.

Please refer to the below SAP pages for understanding business catalogs and technical catalogs.

https://help.sap.com/docs/SAP_S4HANA_ON-PREMISE/22bbe89ef68b4d0e98d05f0d56a7f6c8/71eaa68c02fe4f778d803ac603279184.html

https://learning.sap.com/learning-journeys/learn-the-basics-of-sap-fiori/creating-technical-catalogs_aaedcee4-2e13-4ab5-b2f3-21402b9fdf87

https://help.sap.com/docs/ABAP_PLATFORM_NEW/a7b390faab1140c087b8926571e942b7/f9b0ccce0766442ca8237eee23494f17.html

Most apps in the Fiori apps library come with a business catalog which can be added to the business roles. This will allow the user to access the app. But this is not the case for some apps. Let us see how we can access the app.

Assumptions: You have access to SAP Fiori Launchpad App Manager, FLP Content Manager and PFCG role maintenance.

Referred System: S4HANA On Premise Release 2022 ,SP - Initial Shipment Stack

Let us take an example of the below app:

TINCOV - Display Service Request

App Library URL: https://fioriappslibrary.hana.ondemand.com/sap/fix/externalViewer/#/detail/Apps('TINCOV')/S24OP

In the configuration section of this app, it says that the app is not assigned to a business catalog.

This app is generally accessed via navigation from the 'Search Service Requests' app. But there might be scenarios where we have to access the app directly via an URL. For e.g. the service request link should be embedded in an URL to be sent to an agent or service manager for a quick review.

I will try to access a service request directly via an URL.

URL: https://server:host/sap/bc/ui2/flp?sap-client=033&sap-language=EN#ServiceRequest-display?ServiceRequest=8000000029

I get the below error:

To resolve this, we will add the app to a new business catalog and a business role.

Step 1: SAP Fiori Launchpad App Manager

T-Code: /UI2/FLPAM

The t-code will launch the web browser.

Get the technical catalog id from the app reference library page.

The technical catalog will open showing all the assigned semantic objects and their actions. Scroll down and select semantic object 'Service Request' with action 'display'.

Now we have the details for the app.

Step 2: FLP Content Manager

T-codes:

/UI2/FLPCM_CONF - To make change at the system level. Affects all clients.

/UI2/FLPCM_CUST - Client specific changes.

Here I will be using client specific t-code.

We will be creating anew business catalog. Click on the 'Create' button.

It will ask for a transport request. Provide the same and the new business catalog will be created.

Now click on 'Add Tiles/Target Mappings'. The service request display has only a target mapping.

In the next page you can search for the service request either by using the technical catalog name or the semantic object name.

Select the required line and click on 'Add TM Reference' since this has only a target mapping.

The required target mapping is added to the newly created business catalog.

Step 3: Role Maintenance

In this step we will add the business catalog to a new business role. This step is usually done by the BASIS team or the team responsible for roles and users.

T-code: PFCG

Create a new role.

After clicking on create single role, click on save. In the 'Menu' tab, click on 'Transaction'->SAP Fiori Launchpad->Launchpad Catalog.

Fill in the newly created business catalog and press Enter.

Save the role and generate the profile.

Step 3 Alternative:

There is another method to add the business catalog to the business role. This is in launchpad content manager.

In the content manager, click on the roles tab. In the bottom screen, click on add catalog.

This can be cross referenced in the catalogs tab by clicking on the 'Show Usage in Roles' button.

Step 4: Add role to user.

Add the newly created role to the user.

T-code: SU01

Provide the user id and click on edit.

In the roles tab, add the newly created role. Click on save.

Now open the link to service request. It will open without errors.

If there is a tile associated with the app, the tile will appear in the fiori home screen.

This method can also be used to create new business catalogs with your own list of apps based on business needs.

Hope you found this blog useful. Please comment on anything that could be improved.

New SAP Analytics Cloud Performance Workshop

2024-06-06T10:49:41.167000+02:00

Attention, SAP Analytics Cloud professionals!

Get ready for a new SAP Analytics Cloud performance training course.

Boost your skills in SAP backend modeling and SAP Analytics Cloud performance story design with our specialized training course, “Backend Modeling with SAP Solutions and SAP Analytics Cloud Story Performance Optimization. “

This one-day training course is designed for seasoned SAP Analytics Cloud users who want to explore SAP`s strategic data modeling options and discover the relationship between story design and performance in SAP Analytics Cloud.

You can expect to acquire basic knowledge of data modeling in SAP's central solutions, such as SAP BW/4HANA, SAP Datasphere, or SAP HANA. You will also apply story design best practices and performance analysis techniques to improve your story performance in SAP Analytics Cloud.

Are you curious? Book your seats for US or EMEA timezone here

Carbon Footprint Calculation with IDM from Implico

2024-06-10T08:08:57.471000+02:00

Integrating with SAP's S4SCSD for Enhanced Sustainability

Carbon Footprint Calculation with the IDM from Implico

As industries worldwide prioritize sustainability and environmental responsibility, managing and reducing carbon emissions has become a critical aspect of logistics and transportation operations. Implico's IDM provides a robust solution designed to enhance these capabilities within SAP S4SCSD. This advanced functionality is tailored for industries such as oil, gas and chemicals, where efficient logistics and environmental compliance are paramount.

Support for Sustainability Goals

Aligning logistics operations with broader corporate sustainability goals is a key benefit of the IDM. By providing actionable insights and recommendations, the module can support companies in achieving significant reductions in carbon emissions. This alignment with sustainability objectives underscores the importance of integrating environmental considerations into logistics management within SAP S4SCSD.

Here are some of the factors that make the calculation and management of carbon footprints easier with Implico's IDM.

IDM as a Core Component of SAP S4SCSD

With its functionality, Implico's IDM is an integral component of SAP's S4SCSD (Secondary Distribution Management for Oil & Gas) and boosts the users' requirements for a more detailed carbon footprint calculation. This integration ensures that secondary distribution operations - which include both bulk and bottled products - are not only efficient, but also environmentally responsible.

Support for Emissions Calculation

By using available data such as fuel consumption, fuel type or distances traveled, Implico's IDM functionality amplifies CO2 emission calculations for each trip or delivery within the SAP S4SCSD framework. In a highly competitive environment where companies are striving to understand and mitigate their environmental impact better, the level of accuracy provided by IDM is essential.

Data Integration and Analysis

IDM has the ability to collect data (e.g. telematics, GPS, fleet management) and seamlessly integrate with SAP S4SCSD. This integration can be used, for example, for historical analysis of carbon emissions, providing valuable insights for better decision making and strategic planning. The ability to analyze data from multiple sources within a cohesive system is a significant advantage for companies aiming to optimize their operations.

Route Optimization

One of the key features of IDM is its ability to optimize routes and loads, thereby minimizing travel distances and fuel consumption. By suggesting the shortest routes, the module helps reduce the carbon footprint of logistics operations. This functionality, integrated within SAP S4SCSD, ensures that logistics processes are both operationally efficient and environmentally sustainable.

Data for detailed Reporting and Documentation

IDM facilitates transparency and accountability in environmental reporting. By providing important data that can be used for documentation, audits and certifications, IDM supports SAP S4SCSD from the inside with its detailed reporting capability. This ensures transparency and accountability in environmental performance and supports companies to meet environmental regulations and standards.

User-Friendly Interface

Designed with a user-friendly interface, the IDM tool makes it easy for operators to enter data, generate reports, and access insights. Its seamless integration with SAP S4SCSD ensures a consistent and efficient user experience, minimizing the learning curve for operators and improving the overall user experience.

Conclusion

By incorporating these features, Implico's IDM not only improves logistics management, but also enables companies to take significant steps to reduce their environmental impact. This makes it an invaluable add-on for all organizations, especially those that are seriously committed to sustainability and responsible logistics operations.

With IDM, Implico has developed a product that is natively integrated into SAP S4SCSD and closes the gap between sales and returns. Most previously manual processes in the reconciliation are now automated, allowing back-office operations to shift gradually from manual to automated processes. Since IDM has been used successfully for years within the supply chain of energy and natural resources, Implico is currently working intensively on a new, fully automated version which is in heavy development and ReadyToShip in 2024 already.

The new version is scalable for every size of organization and provides fully automated trip and route planning.

Implico's IDM provides a significant opportunity for market expansion and revenue growth while enhancing compliance requirements for environmental standards.

Let's shape a sustainable future, together.

SAP S/4HANA SDM: This tool offers a cohesive approach to the trade and transport of liquids and half-solids, integrating seamlessly with the SAP system to streamline processes from demand forecasting to invoicing, thereby facilitating smoother operations in hydrocarbon product trading.

SAP S/4HANA RFNO: A dedicated solution for managing service station networks, it caters to various service station types, offering functionalities like wet goods management and loyalty card management, aiming to enhance the fuel retail experience.

Implico’s Quality Assurance Program for SAP solutions: Implico wants to ensure the implementation of SAP SDM/RFNO solutions is professional and error-free – not only when doing so ourselves but also when external companies are contracted to oversee implementation. This is why we also offer two partner services that are designed to supplement the implementation projects based on a methodological approach and with highly skilled consultants, both of which help to achieve the defined targets: the Partner Governance Service and the Partner Advisor Service.

Release Navigator for SAP BTP - What's New in Q2 2024

2024-06-12T13:36:41.584000+02:00

Check out the Release Navigator.

The Release Navigator for SAP BTP continues to evolve and improve to meet the needs of our customers. With the latest update in Q2 2024 in addition to the quarterly maintenance of resources, users can access new product pages and a new consolidated release schedule in a simplified and streamlined interface.
These updates reflect our commitment to continuously improve content coverage and user experience based on your valuable feedback. In this blog post, we will highlight the key enhancements and additions made in Q2 2024.

We collaborated closely with SAP BTP experts to add new pages for the following products and services in the Release Navigator that are available with the Q2 2024 update:

The new product pages include Foundation and Cross Services and more. Check them out now.

Check out the new consolidated release schedule in the release calendar.

Release schedules are now being communicated more transparently thanks to close collaboration with various SAP BTP teams. Compared to 2023, more than 40 new products and services now announce their release dates via SAP Notes in SAP for Me.

These notes are accessible through the Consolidated Release Schedules for SAP BTP, which can be found in the Release Calendar section of the navigator.

Please note: To access SAP Notes you need to log in to SAP for Me using your Universal ID or S-user ID. Read more about the required authorizations here.

The SAP Note Consolidated Release Schedules for SAP BTP

We have published a quarterly maintenance update with the Q2 release cycle. Links and resources have been updated according to the latest feature releases. Moving forward, new and retiring content will be marked with a green or red badge.

Additionally, in Q1 all release navigators received streamlined and consistent UX across solution areas for improved ease of use.

The release navigator after the Q2 update

*Disclaimer: The schedule is subject to change without prior notice including, when necessary, delays or incorporating changes in subsequent update windows and not a commitment, promise, or legal obligation to deliver any material, code, or functionality.

Next updates are scheduled on a quarterly basis in alignment with the Harmonized Release Calendar for SAP Cloud Products.

Looking ahead, here's a sneak peek at some of the roadmap items planned for the release navigator:

Q3: Additional UX elements will be added and aligned across solution areas.
Q4: More products and services will be included in the Release Navigator.

The Readiness@Scale team at SAP Enterprise Support is dedicated to ensuring release readiness across the SAP Enterprise Support portfolio. Our mission is to empower SAP customers to be prepared and ready to adopt and utilize innovations.

If you have any feedback or specific requests for content in the release navigator, please don't hesitate to reach out to us.

And don't forget to check out our team's blogs tagged with Readiness@Scale.

Happy navigating!

Readiness@Scale, SAP Enterprise Support

Update Reference Field Of Payment Document Posted Using F110 By Substitution / BTE - SAP S4/HANA

2024-06-16T06:00:26.552000+02:00

Introduction:

This blog will guide you to updating the reference field of the payment document, even if a single payment is not activated and multiple invoices have a single payment document. Using Substitution we can update the reference fields.

Substitution:

Substitution rules are stored in the Rule Manager. When data is entered in the system, it is substituted by the Integration Manager. Substitution occurs before the data is added to the FI-SL summary tables.

In FI-SL substitution, the values entered into the SAP system are validated against a prerequisite defined by the user. When the prerequisite is met, the system replaces the values entered with substitute values, and the substitute values are transferred into the FI-SL application component.

A substitution process can contain up to 999 steps. Before posting, therefore, you can substitute the values using any number of Boolean statements.

A substitution step contains the following statements:

Prerequisite statement

The prerequisite statement establishes which conditions must be fulfilled before the substitution can be performed. If the prerequisite statement is false, the transaction is continued without substitution. If the prerequisite statement is true, the transaction is continued with the substituted values.

Substitution value(s)

The substitution value is a numerical value or a string of letters that replace the value entered. A single substitution process can replace more than one value.

Substitution exit

You can specify that substitution is to be performed through a substitution exit. The substitution exit number directs the system to an ABAP program defined by the user. Substitution exits enable you to define more complex substitutions and replace more than one value in a substitution. For more information, see User Exits in Validations/Substitutions/Rules .

When you define substitutions, you use the same syntax that you use with validations. The prerequisite statement used in a substitution can consist of a simple statement or a complex combination of statements composed of logical statements, rules, and sets.

Application Areas:

The application area specifies the general application area where the validation, substitution, or rule occurs (for example, GL for FI-SL postings).

Application Areas Used in Validations/Substitutions/Rules

Application area	Validations	Substitutions	Rules
AM Asset Management	X	X	X
CO Controlling	X	X	X
CS Consolidation	X		X
FI Financial Accounting	X	X	X
GL Special Purpose Ledger	X	X	X
GR Report Writer			X
GS Ledger Selection			X
GU Rollups			X
KC Enterprise Controlling: SAP -EIS	X	X	X
LC Legal Consolidation	X		X
PC Profit Center Accounting		X	X
PS Project System	X	X	X

User Exits in Validations/Substitutions/Rules

User exits are user-defined FORM routines that are used to calculate and/or replace values within a validation, substitution, or rule.

User exits have the following format:

U(for a user-defined user exit)
S(for a standard user exit)

You can configure the form pool name of the user exits and must store it in the table for client-dependent user exits (T80D) in Customizing . Table T80D contains the form pool names for the user exits used in validations, substitutions, and rules. Each validation/substitution form pool is client-dependent.

The following table shows the types of user exits that can be used in validations, substitutions, and rules.

User exit type	Description	Application	Example
1	No parameters are defined for the user exit.	Rules, validations, and substitutions (prerequisite)	See form pool RGGBR000, parameter type C_EXIT_PARAM_NONE
2	Same as user exit type 1, except one parameter (the field to be substituted) is defined in the user exit. For example, you can create a substitution routine that analyzes the cost center irrespective of the field used.	Substitution	See form pool RGGBS000, parameter type C_EXIT_PARAM_FIELD
3	All data is passed as one parameter; this exit type can only be used in matrix validations and substitutions.	Rules, validations, and substitutions (prerequisite)	See form pool RGGBR000, parameter type C_EXIT_PARAM_CLASS

Formula Builder:

In the Formula Builder, all elements that you can enter in a logical statement (for example, operators, operands, sets, constant values, and so on) are displayed as push buttons. You enter a logical statement by choosing the elements that you want to include in the statement.

You can work in the Formula Builder using three different settings:

The default setting when you call up the Formula Builder is Short descriptions . All operands (such as table fields, sets, exits, and so on) are displayed using their respective descriptions.

If you want to enter statements using the technical names of the operands rather than the descriptions, you change the setting by choosing Settings Technical names .

The Expert mode setting displays the technical names of the operands and allows you to enter a statement directly. In this case, the statement field is ready for input.

Updating Invoice Text As Payment Document Reference:

Prerequisites:

Go to Transaction : FBZP

Transaction Code

FBZP

Click Payment method in the Country

Double click on Payment method

Posting Details Tab:

Document type for payment "ZP"

The Payment Document Will be posting in document type ZP.

Configuring the FI Substitution :

Go to Transaction : GGB1

Transaction Code

GGB1

In this activity, you can create and maintain substitutions.

Actions

Expand the hierarchy to the desired application area and event
Select the substitution that you can to change or use as a template
Proceed according the the instructions for creating and maintaining substitutions

Create substitution

To create a substitution without a template, select an event and choose the Create substitution button
To create a substitution using a template, choose the Copy substitution button.

Maintain substitution

Create a substitution step and select a substitution step to be edited.
If you want to insert a step, you have to select the field that is to be changed. Enter the prerequisite and define how the substitution is to be carried out.

In Financial accounting, we have 5 Part

Document Header
Line Item
Completed Document
Cost of Sales Accounting
Cost of Sales Accounting (New)

The reference field is in the header part, we need to create a substitution in the header part.

Now create a step Inside the substitution.

Select BKPF-XBLNR in Steps.

Here you can define the prerequisite based upon your requirements.

Document Type = 'ZP'
Company Code = "XXXX'

In the Substitution step, define the Substitution based on Exits. We have to define our custom Exit in the EXIT-Formpool provided by SAP or we can copy the standard and create our own Module Pool for the same. In our case, we have copied RGGBS000 to ZRGGBS000.

Take a Z copy of RGGBS000

In the Module pool, define the exit name in the header in FORM GET_EXIT_TITLES.

Define user Exist Called Z300.

  exits-name = 'Z300'.
  exits-param = c_exit_param_none.
  exits-title = TEXT-z08.
  APPEND exits.

Down below in code, we define our code based on our logic as follows:

*&---------------------------------------------------------------------*
*& Form Z300
*&---------------------------------------------------------------------*
*& text
*&---------------------------------------------------------------------*
*& -->  p1        text
*& <--  p2        text
*&---------------------------------------------------------------------*
FORM z300 .

  DATA: lv_date  TYPE regup-laufd,
        lv_runid TYPE regup-laufi.

  TYPES: BEGIN OF ty_regup,
           lifnr TYPE regup-lifnr,
           belnr TYPE regup-belnr,
           gjahr TYPE regup-gjahr,
           blart TYPE regup-blart,
         END OF ty_regup.

  DATA: lt_regup_final TYPE TABLE OF ty_regup.

  REFRESH: lt_regup_final.

  IF bkpf IS NOT INITIAL AND bkpf-tcode = 'F110' AND bkpf-bukrs = '7180' AND bkpf-blart = 'ZP'.

    SPLIT bkpf-bktxt AT '-' INTO lv_date lv_runid.

    SELECT lifnr, belnr, gjahr, blart, xvorl FROM regup INTO TABLE (lt_regup)
    WHERE laufd = _date AND laufi = _runid AND zbukr = '7180'.

    LOOP AT lt_regup INTO DATA(lwa_regup).
      READ TABLE lt_regup INTO DATA(lwa_regup_temp) WITH KEY belnr = lwa_regup-belnr xvorl = ''.
      IF sy-subrc NE 0.
        APPEND lwa_regup to lt_regup_final.
        CLEAR: lwa_regup, lwa_regup_temp.
      ENDIF.
    ENDLOOP.

    IF lt_regup_final IS NOT INITIAL.

      SELECT belnr, lifnr, sgtxt FROM bseg FOR ALL ENTRIES
      IN _regup_final
      WHERE bukrs = '7180' AND belnr = _regup_final-belnr AND lifnr = _regup_final-lifnr
      AND gjahr = _regup_final-gjahr AND koart = 'K'
      INTO TABLE (lt_bseg).

      SORT lt_bseg ASCENDING BY lifnr.

      IF lt_bseg IS NOT INITIAL.
        READ TABLE lt_bseg INTO DATA(lwa_bseg_1) INDEX 1.
        IF sy-subrc = 0.
          bkpf-xblnr = lwa_bseg_1-sgtxt.
        ENDIF.
      ENDIF.


    ENDIF.

  ENDIF.
ENDFORM.

Assign the same in view V_T80D with Application Area as GBLS (Val/sub:Exits for substitution) and Ex.prog. as ZRGGBS000.

Assign Z300 in substitution step.

In our last step, we have to activate this Substitution.

Transaction Code

OBBH

There are times when we need to use the substitutions for particular fields but they are unavailable in Tcode GGB1. This requires a small effort to include the requisite field for Substitution.

1. Check if the field that needs to be substituted is already present in Tcode GGB1 or not.

2. If it is not present, go to SM30 -> table VWTYGB01. Click Maintain.

3. Uncheck the “Exclude” checkbox for the requisite field.

4. Run the program RGUGBR00 via SE38 and click execute.

5. The field will get added to the Substitution Rules.

Testing Scenarios:

Review the Invoice:

Open the vendor invoice which you want to make payment and review it.

Text : F110 Substitution Test

Manage Automatic Payments:

Payment run process is an accounts payable process that is used to handle payments to vendors. The process is called automatic payment program.

With the transactional app Manage Automatic Payments, you can schedule payment proposals or schedule payments directly and get an overview of the proposal or payment status. The app identifies the overdue invoices and checks whether all the required payment information is complete.

The Payment Process includes the Following Steps

Invoices are Entered

Pending Invoices are analysed for due date

Invoices due for payment are prepared for review

Payments are approved or modified

Invoices are paid

Start fiori app “Manage Automatic Payments” and Click Create Parameter.

On the first screen, you need to enter run date and identification. In the identification field, give a five-characters alphanumeric code that will be used to identify the payment program. A combination of the run date and identification is used to identify the program in SAP system.

Fill the Run date and identification and click Create:

In the Parameters Tab, We have to define the following and Click Save

What is to be paid – Docs. Entered Up to

What payment methods will be used – Payment Methods

When will the payments be made – Posting Date?

Which company codes will be considered – Company Codes?

How are they going to be paid – The payment Method Sequence decides the Priority of the Payment Method

Enable Additional log – To get some additional log information.

You can see the payment parameter created :

Create Payment Proposal:

Selected the Parameter and click schedule Proposal:

When you click on Proposal the system is going to ask whether you want to create it immediately or want to schedule it for a time in the future. For this demo, we are going to run it immediately. You should tick the checkbox for creation of payment medium.

Make sure you select “Start Immediately” and then click on Schedule.

You can see the payment proposal in the respective tab.

A payment proposal is generated based on the parameters.

Review Payment Proposal:

Click on the arrow button to review the document selected in this run.

The system will list all the vendors that have been included in the payment proposal.

We can view the Proposal Log for possible errors by Pressing the Proposal Log Button.

We can edit the proposal to block the some payments if we want, Press the Edit Proposal Button.

Payment Run:

Go to Proposal and click on Schedule Payment to run Payment.

Check “Start Immediately” to start the payment run instantly and then click on Schedule.

Go to “Payments Processed” and You can see the payment posted.

We can check the status of the Payment run on the Status Tab.

You can also select the posted payment to see the log:

Review The Post Payment Document:

Invoice Document Text: F110 Substitution Test

Payment Document Text : F110 Substitutio

Since the length of BKPF-XBLNR is 16, It allows only 16 characters.

Transport of Substitutions:

You transport substitutions manually into the productive system using the function "Substitution" -> "Transport".

You can also transport substitutions using a specific Customizing transaction. For more information, see the "Transport Substitutions" activity.

Updating Reference Fields for single Payment:

When you clear items using the payment program, the system does not transfer the reference document number to the payment document, even though the reference document number in the items to be cleared is unique to each payment document. The requirement is not programmed, since the reference document number is generally not unique to each payment document.

Implement the attached program corrections.

A reference document number unique in all cleared items is transferred to the payment document.

SAP Note Reference : https://me.sap.com/notes/213551

Hope this helps you in applying Substitution successfully in SAP.

SAP Data and Analytics: Join our Performance focused Live Sessions Series

2024-07-12T08:47:47.532000+02:00

Performance optimization is a central task for every developer, data modeler, data engineer or data analyst since in addition to quality, speed is a key requirement for modern reporting. With this in mind, we have created a sophisticated series of expert-guided Live Sessions to address this important challenge.

Boost your skills in SAP Analytics Cloud performance and explore key features and optimizations that enhance data analytics efficiency and effectiveness in SAP Analytics Cloud, in our specialized expert-guided Live Session events, “SAP Analytics Cloud Performance overview”. Register here.

Understand the performance implications of story design and scripting in SAP Analytics Cloud, in our expert-guided Live Session events "Understanding the relationship between story design and performance in SAP Analytics Cloud". Register here.

Discover best practices in data modeling in SAP HANA to ensure your models run at the highest possible speed and are built for sustainability, in our expert-guided Live Session events “Best practices when developing calculation views in SAP HANA”. Register here.

Maximize the performance of accessing data in SAP BW/4HANA. Master the techniques for increasing efficiency and speeding up the retrieval of critical data in your organization, in our expert-guided Live Session events “Performance aspects of Advanced DataStore Objects and CompositeProviders in SAP BW/4HANA.” Register here.

Explore advanced SAP Analytics Cloud features like custom filtering or display logic and review the scripting power of SAP Analytics Cloud having best practices for a clean code approach in mind, in our expert-guided Live Session events "SAP Analytics Cloud – intermediate reporting". Register here.

The Live Session events mentioned above are linked to the Learning Hub and can only be attended with a valid subscription.

Product Learning CoE Analytics Team

How to migrate Convergent Charging from Oracle DB to HANA DB.

2024-07-16T18:44:59.972000+02:00

Hello SAP Community!

I am coming to you with kind of niche topic about niche product which is a part of the SAP portfolio - Convergent Charging. It was a part of the Hybris solution which evolved into BRIM (SAP Billing and Revenue Innovation Management). There are many descriptions of Convergent Charging on the Internet. Just to make it short: CC is a solution which offer companies a simplified way to charge customers on a pre- and post-paid basis. You may create many different types of subscriptions to offer variety of services for different type of customers and all can be done really quick and on the easy way.

Some basic information about CC

The first thing we should know, is that CC is officially Netweaver based product. It doesn't mean that it is built like ABAP- or JAVA-stack. It may be, however, closer to JAVA-stack, since it is developed in JAVA. There are common parts with Netweaver stacks, like:

profiles: has its own specific parameters, but some are taken from ABAP- or JAVA-stacks,
kernel,
installer: I mean SWPM here, which is used for installations, updates, upgrades and uninstallations, however for the CC you have to use a special version which is delivered exclusively for it,
there may be some more similarities too, but I cannot think of any of them right now.

On the other hand, this product is not that complicated as known NW-stacks. It has a documentation and support above the satisfying level. We never had a big issues with it, I can count on the fingers of one hand major issues and on both hands minor ones. Since we started few years ago with implementation of this product, we have a positive thoughts about it. Updates/upgrades are really fast, administration is simple, support is responding very fast. I may write another blog entry about technical side of CC, but I will try to focus on the main topic - migration from Oracle to HANA.

Challenges

First of all, required OS or DB versions are well described in the PAM: Convergent Charging PAM. According to it, SAP HANA 1.0 (and higher) is supported until CC 5.0 and from 2020 only SAP HANA 2.0. Very good, we can migrate to 2.0. We have been searching for the generic migration Guides, but could not find any. You may find on the Web products, which will help with Oracle to HANA conversion, but you need additional license and probably some trainings. After discussion with SAP they informed us, that there is no official or even unofficial guide how to do it and such procedure is not supported ☹️. At first were disappointed and didn't know how to proceed, because we wanted to move to HANA without a big problems and this type of steering like for ABAP- or JAVA-stack was missing here. Luckily for us, they pointed some HANA tools used by the other clients like: SDA (Smart Data Access) and SDI (Smart Data Integration). Additionally the CC itself is really small system. It has around 700 tables, so it is not that overwhelming. The next thing they pointed was a guideline Copying an SAP CC system | SAP Help Portal. It describes migration of the CC system to any database, from the higher level, so without specific OS, DB, etc. Just pure, consistent strategy. Not that bad! After some brainstorming and meetings regarding available tools, different approaches, application support, we decided to make it as simple as possible and to use SDA.

Oracle-export approach

Before I go to the substance, let me present our road to the goal. The first, and seems to be a natural, step was to export content of the tables from Oracle in CSV-format and import them to HANA. What we noticed, was the date format. In Oracle with our default environment variables we had DD.MM.YY while in HANA we needed YYYY-MM-DD HH:MM:SS. Change of the NLS did the job and we could extract the tables. From this point we could choose to have the tables in CSV or in the text files as INSERTs.

The first approach seems to be very complicated and time consuming, because we will have to import the files via SAP HANA Studio only one at a time. Imagine doing it for ca. 700 tables. The whole procedure is described here: Import CSV\Flat file to Hana Database. There is no bulk option built-in. OK, there is one, if the files are already stored on the HANA host. IMPORT FROM Statement (Data Import Export) can be used here and in an easy way duplicated for each table. Additionally it may be necessary to modify manually each file, because Oracle export may not be suitable 1:1 for import into HANA.

The second approach, with generated INSERTs from Oracle, sounds better. The text files can be modified in an easy way, but the biggest problem is the number of line which can be inserted into SQL Statement in SAP HANA Studio. We tried with small numbers like 50.000 or even 10.000 and every time JAVA memory exception was raised.

There might be other approaches, different methods used, some automation and someone may have also other issues. We gave up here and focused on a tool delivered with SAP HANA - SDA

Smart Data Access

We have started with the basic questions:

What about the license? "Because smart data access is part of the SAP HANA core system, no additional licensing is required to use it." ✔️
Is our database version supported? 2600176 - SAP HANA Smart Data Access Supported Remote Sources ✔️
Is it out-of-the-box tool? Unfortunately no, but preparation steps are quite simple and have already been known to our HANA Team. Just for the record - here they are: Oracle Database ODBC Driver ✔️

After all was set we could finally start. We will follow here the general guideline for copying CC, which I mentioned earlier.

Preparing the destination system

I assume that the installation has already been done on your side and we can move on. If not, please refer to SAP's Installation Guides. Additionally, user created in HANA for CC installation, should have one more Role under System Privileges: "CREATE REMOTE SOURCE".

Backing-up the tables of the destination Core Database and Session Database, according to the copy scenario

I suggest now to perform full backup of the destination system, just to be sure, that we can revert our CC to the first working version. Below are the steps we performed to create backup.

Mark all tables under relevant tenant and CC HANA Admin user, click right mouse button and choose export:

In the next step, all irrelevant tables can be selected and removed from being backed-up as on the screenshot below. Just for the information: the small icon indicates, that they are virtual tables. We do not want to back them up, because it does not make any sense.

Following options must be used from “Export Selection” screen. Export catalog will be stored on a local storage. Number of Parallel Threads can be i.e. 5. Click on Finish will start exporting the tables:

Backing-up the source Core Database and Session Database

This step is not necessary here, because from SAP HANA Studio we are unable to change anything on the source Oracle database. At least I have not found any option to do this.

Transferring the adequate data from the source Core Database and Session Database to the destination ones
Restoring the adequate data within the destination Core Database and Session Database

Congratulations, you have finally reached your destination. Thank you for being patient 😊

At first, we have to create External Source which allows us to connect to the Oracle source database:

Then change Adapter to ORACLE, fill in the Data Source Name from tnsnames.ora, enter username and password for access to Oracle DB:

Now let's create virtual tables with links to the content of the Oracle source system. Expand previously created External Source until you reach username provided above. Mark all tables (I assume there are only CC tables there) and click on "Add as Virtual Table". Provide prefix for the tables and wait. Progress can be monitored in the "Progress" tab:

For the further processing, we have to collect the table names from the target system. To do this, we have to execute a SQL command on the database/tenant where the CC has been installed. As a schema user, insert the CC HANA Admin user used for installation:

select TABLE_NAME from SYS.M_TABLES where SCHEMA_NAME ='<cc_hana_admin_user>';

If you want to list the virtual tables, other statement must be used:

select TABLE_NAME from SYS.VIRTUAL_TABLES where SCHEMA_NAME ='<cc_hana_admin_user>';

The table names are required to build INSERT and SELECT statements, which we used to migrate the tables' content from the Oracle to HANA. The fastest way, for me, was to generate everything in the Excel sheet. So I paste already listed tables into new Excel file.

First step was to delete (actually to truncate) content of the tables in target system. Under documentation Restoring the Destination Core Database SAP describes which tables should be restored with which action. My approach is, that the ones with "Overwrite only", must not be deleted as they already have required content. The other tables can be dropped. Here following statement was used:

TRUNCATE TABLE <cc_hana_admin_user>.<table_name>;

So basically if there is a list of the tables in Excel, the statement for each table can be created easily.

The next and last step is to insert data from the source into target database. It is done by using following statement:

INSERT INTO <cc_hana_admin_user>.<table_name> SELECT * FROM <cc_hana_admin_user>.<virtual_table_name>;

Last word for the end

It is actually all. Of course you may struggle with performance issues as we did. Just to avoid any, the SQL statements for small tables can be executed in one SQL Console and INSERTs for the big tables in other, running parallel. Additional thing, which may speed up the process, is to perform Housekeeping according to the original documentation: Purging and Archiving | SAP Help Portal. We could delete a pretty number of outdated logs.

Hope you find this blog entry interesting. Any comments or questions are always welcome.