SAP Community - SAP HANA Cloud

Webinar Replay: Better Business Outcomes with the SAP HANA Cloud Vector Engine

2024-04-09T18:40:22.053000+02:00

In case you missed it ... I highly recommend reserving some time in the calendar to watch the webinar replay from April 4. Experts from SAP and PwC did a great job introducing the vector engine and its role within the retrieval-augmented generation (RAG) process.

SAP delivered the vector engine as part of the Q1 2024 release of SAP HANA Cloud. Some noteworthy benefits include:

Unite all types of business data within a single database using multi-tier data storage, multi-model engines (now including vector), JSON document store, and machine learning advanced processing
Incorporate semantic and similarity search into intelligent data applications
Enhance the user experience with more natural capabilities
Augment and improve the output of large language models (LLM) with contextual business data

Also, checkout these informative links for additional vector-related resources:

Discovery Center Hands-on Exercise: Explore the capabilities of SAP HANA Cloud, SAP Generative AI Hub, Azure OpenAI and the LangChain Framework
Blogpost: Vectorize your Data: SAP HANA Cloud's Vector Engine for Unified Data Excellence
Blogpost: From Developer's Desk: SAP HANA Cloud Vector Engine
Blogpost: What's New in SAP HANA Cloud - March 2024

Connection restrictions and their relation to user groups in SAP HANA Cloud, SAP HANA Database

2024-04-10T16:48:15.194000+02:00

Connect restrictions are access policies and essentially enforcing the limited access to SAP HANA Database for users which belong to a specific user group*. For example, they can be used to restrict access for administrator users to a given IP range, or for technical users to specific applications.

They enable customers the flexibility to limit access for all users or parts of their user base, for instance, to enable only administrators to log in from the customer's IP range and application range based on the application session variable.

Benefits:

Provides more fine-granular control over security configuration settings
Allows customers to further secure their systems against attacks from the Internet

More about connect restrictions you can read in SAP HANA Cloud, SAP HANA Database Security Guide: User Management -> User Groups -> Connect Restrictions.

Setting up the connect restrictions by using SAP HANA Cloud Central

For a demonstration of how you can make use of connect restrictions, I will use HANA tooling, and more specifically, SAP HANA Cloud Central**.

A typical customer scenario that is pursued with the usage of this functionality is to limit the access of the users from specific applications they use to access HANA. Also, it is a case when technical users of HANA DB should not be allowed to connect or log into specific apps, with the help of which you can manipulate data or settings in HANA, like HDBSQL or HANA Cockpit/DB Explorer.

To create connect restriction policies, you need to access SAP HANA Cloud -> section “User & Authorization Management” -> subsection “User groups”.

If you cannot see this tile or access SAP HANA Cloud Central, please check SAP HANA Cloud Administration Guide documentation for needed tools subscription and missing authorizations for user.

Creating a User Group-Specific Connect Restriction

A single connect restriction can contain multiple applications and IP conditions.

To configure those conditions, a user requires an OPERATOR object privilege on the given user group. This can be done by a creator of the given user group or another user who has GRANTOR privilege on this given user group.

1. If not yet created, in the “User & Authorization Management” section, subsection “User groups,” set up a user group, for which you will subsequently create required restrictions. For us, it will be Administrators1. Move required users to this Group.

2. Scroll down and create a connect restriction NetworkRestrictions1 with conditions for IP addresses in CIDR notation and application. The restriction allows connections for hdbsql application from the mentioned IP addresses and IP ranges.

You can define multiple connect restrictions in a single statement. The following example shows how to define connect restrictions for IPS_V4, IPS_V6, and HDBSQL:

3. The definition of connect restrictions and a list of connection restrictions by user group can also be viewed in the system view USERGROUP_CONNECT_RESTRICTIONS.

An example for the USERGROUP_CONNECT_RESTRICTIONS table would be as follows in the SQL console of SAP HANA Cloud Central if you execute the steps above:

Additional syntax for user group creation statements with parameters of connect restrictions can be found in SAP HANA Cloud, SAP HANA Database SQL Reference Guide -> CREATE USERGROUP Statement (Access Control).

Evaluation of Connect Restrictions

After the user has successfully authenticated, connect restrictions are evaluated as part of the user authentication process. If the user group has connect restrictions, members of that group can only connect if at least one of the defined connect restrictions allows access.

Note: A single connect restriction can consist of multiple conditions, and all must be met to allow access for the user.

Deactivating connection for all Users of a User Group

Use the ENABLE | DISABLE CLIENT CONNECT option to control whether or not the users in a user group can connect to the SAP HANA database, for example, to stop users temporarily from connecting during updates or troubleshooting activities.

Sample Code:

ALTER USERGROUP MyUserGroup DISABLE CLIENT CONNECT

If you disable client connects for a user group using this feature, no user of that user group can connect, even if connect restrictions would allow it.

Validating a User Group-Specific Connect Restriction

In the connect restriction bar on the top right, you also can validate whether a connect attempt would be possible given the active connect restrictions of a user group:

The window will pop up where you can test different options and get the results:

Also, you can use the table function PUBLIC.VALIDATE_USERGROUP_CONNECT_RESTRICTION to do that and use the table function PUBLIC.VALIDATE_USERGROUP_CONNECT_RESTRICTION_DETAILS to display details of each connect restriction if a login was allowed for a specific condition.

Some things to keep in mind about validation views:

This table functions validates all connect restrictions of a user group
Returns a detailed result of each connect restriction if a login is allowed
The check is done for the active and inactive restrictions of a user group:
- Client IP Address
- Application Name

NULL values will be returned if a condition doesn’t exist in a connect restriction.
The input parameters can be omitted as they are supplemented by default values.

Authentication/Connect Failures due to Connect Restrictions

If a user connection cannot be established due to a connect restriction, you can find details about why the connection was rejected in the “Failed Authentications” page of “User & Authorization Management” tile of SAP HANA Cloud Central. We will talk more about these features in our future blog post.

Additional resources to check:

*User groups support a separation of user management tasks, allowing you to manage related users together and efficiently. A user can belong to only one user group. You can create and manage users in user groups in the SAP HANA Cloud Central and SAP HANA Cockpit.

More about user groups you can read in SAP HANA Cloud, SAP HANA Database Security Guide: User Management -> User Groups, and for more information on how to create, see the SAP HANA Cloud, SAP HANA Database SQL Reference Guide:

**With the migration of SAP HANA Cockpit functionality in SAP HANA Cloud Central, all of the documentation for these features has also been migrated to the SAP HANA Cloud docu set.

Moreover, SAP HANA Cloud Central has in-app help that can be accessed by clicking the “?” icon in the shellbar. This in-app help is extensive and contains links to relevant content in the cloud docu set.

The following links may also be helpful:

For SAP data lake (SAP HANA Cloud) administration: SAP HANA Cloud, Data Lake Administration Guide for Data Lake Relational Engine

For SAP HANA Cloud administration: SAP HANA Cloud Administration Guide

For SAP HANA database (SAP HANA Cloud) administration: SAP HANA Cloud, SAP HANA Database Administration Guide

Also, the SAP HANA Cloud Database Administration with SAP HANA Cockpit still exists but will not be updated or maintained.

Nested JSON to SAP HANA Tables with SAP Integration Suite

2024-04-11T12:51:56.541000+02:00

In this blog post, I will demonstrate how to send data to SAP HANA Cloud using the Integration Suite. Additionally, I will explain how to handle nested JSON data and distribute it across multiple tables utilizing parallel multicast and mapping functions.

Problem Statement:

We have exposed an API endpoint through which we push data in JSON format and in response we get the insert count in particular tables. The input data contains user details and role details in nested form. We are supposed to insert the user details in User Table whereas in the User-Role mapping table for each role associated with a user, we ensure the creation of a corresponding entry, linking the user's details with their roles. Our requirement is to process the JSON data via CPI and populate these two tables.

Architecture:

Adding JDBC Data Source in Integration Suite

We need JDBC URL, User ID and Password to connect to the SAP HANA Cloud Database. We can get it from instance credentials.

Here we get the JDBC URL for our database which will be like “jdbc:sap:// <instanceID>.hana.trial-us10.hanacloud.ondemand.com:443/?encrypt=true”

We also get the Runtime User ID and Password for the Schema in the same key.

Go to your integration suite tenant and click on JDBC material under Manage security section.

Add the instance details we got from credentials and deploy.

In case of 3rd party database such as Microsoft SQL Server, Oracle and DB2 you need to upload the JDBC driver jar file under JDBC driver tab. In case of HANA, it is not required.

Integration Flow Design:

I have created this integration flow where I have exposed one endpoint to get the JSON data. Firstly, I have converted the JSON data to XML format because we will be sending data through JDBC Adapter in standard XML format. Below is the standard format for the XML which is used to insert data into database.

<root>
   <StatementName1>
       <dbTableName action="INSERT">
           <table>SCHEMA.”TABLENAME”</table>
           <access>
               <col1>val1</col1>
               <col2>val2</col2>
               <col3>val3</col3>
           </access>
       </dbTableName>
   </StatementName1>
</root>

I have used parallel multicast to route the message simultaneously in two different branches since we have to insert data in two different tables.

Let’s look at message mappings for both the routes.

For User Table we are using simple one to one mapping and passing "INSERT" in the action attribute.

This is the mapping for the mapping table where we map users and their roles. As you can see the userID node has single occurrence per record, but we need it to occur multiple times as per the number of roles that particular user has. So, we will be using the useOneAsMany Node function.

We then gather the messages from both the routes and pass them together through the JDBC Adapter using Batch processing.

Let’s look at the configuration of JDBC Adapter.

Here is the input payload content I am passing from the Postman.

{
  "root": {
    "UserDetails": [
		{
			"userID": 6666,
			"username": "harshal",
			"gender": "male",
			"email": "harshal@test.com",
			"roles": [
				{ "roleID": 1 },
				{ "roleID": 2 }
			]
		}
	]
  }
}

Here is the XML payload content that goes through JDBC Adapter.

<root>
   <StatementName>
       <dbTableName action="INSERT">
           <table>TEST2_HDI_DB_1.USER</table>
           <access>
               <user_ID>6666</user_ID>
               <username>harshal</username>
               <gender>male</gender>
               <email_ID>harshal@test.com</email_ID>
           </access>
       </dbTableName>
   </StatementName>>
   <StatementName>
       <dbTableName action="INSERT">
           <table>TEST2_HDI_DB_1.MAPPING</table>
           <access>
               <user_ID>6666</user_ID>
               <role_ID>1</role_ID>
           </access>
           <access>
               <user_ID>6666</user_ID>
               <role_ID>2</role_ID>
           </access>
       </dbTableName>
   </StatementName>
</root>

Response in Postman:

Let's check our entries in HANA Database.

Thanks for reading this blog. I hope it was helpful.

Value Unlocked! How customers benefit from SAP HANA Cloud

2024-04-16T10:26:48.188000+02:00

It's a wrap! The Unlock the value of SAP HANA Cloud webinar series has come to an end. Over the past couple of weeks our dear customers, Black Horse One, Lufthansa Technik, Żabka Polska and Lexmark shared their respective SAP HANA Cloud use cases with you. They range from intelligent data applications and application development to integration and migration.

If you had no chance to join them live, I have good news for you, the recordings are now available on demand! Register for free, to access all four recordings and the respective resources shared during the webinars. If you are not sure whether the series or single webinars are a fit for you, then have a look at the below summaries to get a first idea of the contents.

Webinar Series Visual

🏇 Building a powerful, reliable, high-performance data solution with Black Horse One

Today businesses face numerous challenges, that make essential cloud qualities such as performance, reliability, and trustworthiness necessary. Explore how Black Horse One leverages SAP HANA Cloud for their high-performance data infrastructure in this session. Witness the shift from manual to digital processes, enabling real-time access for fans to live video streams and competition stats, ensuring stakeholders are consistently satisfied with smooth performance. Additionally, dive deeper into the capabilities SAP HANA Cloud offers to kickstart the development of your intelligent data app.

✈️ Developing a strong and trusted data application for price management, purchasing and supply chain of spare parts for Lufthansa Technik

Data management needs to be flexible and efficient alike. Discover SAP HANA Cloud, the cloud-based database solution crafted to meet these criteria. Gain insight into how Lufthansa Technik is running custom, crucial applications on vast amounts of data, offering a dependable and economical base for modern businesses. Understand how you can enhance your enterprise with a high-performing, cloud-based structure. Participate in this session to understand the advantages Lufthansa Technik gains from SAP HANA Cloud. Moreover, grasp the application development capabilities of SAP HANA Cloud and how they can assist you in achieving a competitive edge. Moreover, don't miss out on the live demo of SAP HANA Cloud that has been shown during the webinar.

🛒 Efficient data access for apps and services in Żabka Polska's hybrid landscape

Get rid of duplication, ensure streamlined access to vital data, allocate space for reports, and replicate on-premise data utilizing SAP HANA Cloud. Manage, regulate, and enhance business processes by conveniently accessing information in a dynamic, hybrid setting. Explore the advantages of employing an in-memory cloud database in Żabka Polska's integration scenario. Furthermore, get insights into offloading workload and accelerating performance when replicating data from SAP HANA to SAP HANA Cloud.

🖨 Lexmark maximizes technology investments and improves cost-of-ownership with a seamless migration to the cloud

Migrating your on-premise solution to the cloud poses considerable challenges. Participate in this session to learn more about the factors driving Lexmark's migration. Gain insights into the smooth transition from SAP HANA on-premise to SAP HANA Cloud, that helped to optimize technological investments and enhanced the total-cost-of-ownership. Additionally, get more information on how SAP can support your migration journey towards SAP HANA Cloud.

Did this make you curious? Then don't miss out and access the replays on demand. Register for free!

Are you looking for more information on SAP HANA Cloud?

Check out these links:

SAP HANA cloud and on-premise databases Community page. Your one-stop-shop for news, learnings and more regrading our products.
whatsnewinsaphanacloud Community tag. Don't miss new blogposts for SAP HANA Cloud.
SAP HANA Cloud playlist on YouTube. Check out our videos about the most recent innovations in out database-as-a-service offering.

Consuming CAPM Application's OData service into SAP Fiori Application in Business Application Studio

2024-04-16T12:50:31.978000+02:00

OBJECTIVE-:

Consuming OData Service of CAPM Application into SAP Fiori Application in Business Application Studio.

CAPM (Cloud Application Programming Model)-:

It is an open-source framework that uses tools, libraries, and languages to create applications that can be used across multiple SAP products. CAPM is a multi-target application that runs at different times, with the persistence layer running in the HANA DB run time, the server layer running in the Node.js run time, and the UI/UX layer running in the browser run time.

CAPM is one of the approaches to developing cloud applications using the BTP platform. CAPM offers a more structured and seamless framework for data modelling and enhancing integration with services.

Advantage of CAPM over other approaches-:

It enables full-stack development in the same environment. (BAS), eliminating the need to switch between different environments for the backend and frontend development.
It offers flexibility in language selection, allowing developers to work with multiple languages according to their needs. BAS provides easy connectivity to required extensions for this purpose.
Seamless integration with git and other applications via APIs facilitating extension or development of the application.
Automatically handling dependencies to a certain extent, reducing frustration, and improving development speed. For example, handling the OData after exposing all entities and making it easier to consume in the UI.

Prerequisites-:

Create a CAPM project.
Log in to Cloud Foundry Credentials.
Ensure the HANA Database is created and running (I Used it for storage and data creation).

Set up a Dev Space.
Create a SAP Build Work Zone instance to act as a Fiori Launchpad for the CAPM Application

You Have to create a new Destination in BTP (Connectivity -> Destination)-:

This destination will be used for passing system information when creating a Fiori Application.

For the URL, DEPLOY YOUR APPLICATION IN CLOUD FOUNDRY (Cloud Foundry ->Spaces ->Dev (Space Name)->Your Service (here it is CAPMChetan-srv) and copy Application Routes).

This URL will be shown only when you deploy your CAPM Application in Cloud Foundry-> A new blog will be there to deploy the CAPM Application.

For Authorization, use the same authorization as created in the package.json -> a new blog will create a security configuration.

Now that the destination for the service is configured, proceed to create a FIORI APP: -

Open SAP BAS (Business Application Studio).
Open DEV Space where you want to create a Fiori Application.
Go to File -> New Project From template.
Choose a project from the Template (SAP Fiori Application).

Choose any Template of your choice.

In Data Source choose to connect to a system.

Specify the Destination created earlier.

In the service path, specify the service path of the OData service that can be obtained by running the command “cds watch –profile hybrid” in your CAPM Project.

Click on next.

Select any entity from the given list of entities you created in your project.

Customize the project attributes.

Choose the target as Cloud Foundry and select the Destination name among the list of destination names (You can go with either Local CAPM Project API and the name of your destination too).

Now, your FIORI project is created and is ready to use.

Go to run configuration and start the project as usual.

You can now run this Fiori Application which is the result of an Application created from CAPM.

Top Picks: Innovations Highlights from SAP Business Technology Platform (Q1/2024)

2024-04-17T20:29:36.355000+02:00

As spring season arrived, we find ourselves already immersed in Q2 2024. So let´s pause for a moment to explore the SAP BTP (Business Technology Platform) innovations we had released in the first quarter of 2024.

Take a break and enjoy reading the blog for a closer look on the Q1 2024 highlights of our recent deliveries and an outlook of what´s up next on the SAP BTP product roadmap.

SAP BTP Innovation Highlights Q1 2024

Application Development and Automation:

At TechEd 2023 in November we announced SAP Build Code. Excited to share that SAP Build Code is generally available to all developers. With the help of our copilot Joule, SAP Build Code provides AI-based code generation, optimized for Java and JavaScript application development. It is providing a turn-key environment for coding, testing, integrations, and application lifecycle management. Tailored for SAP development, SAP Build Code utilizes the favorite SAP programming models and frameworks and enables to seamlessly connect to S/4HANA and other applications with pre-built integrations. The guided development experience helps as well using best practices. In addition SAP Build Code encourages fusion development to use a mix of traditional coding and low-code development. Excited to try out the SAP Build Code offering? Check out the blog post of @bhagat_nainani to get more information on features, test drive tutorial, developer quotes and many more.
SAP Build Process Automation got a new governance capability within the Control Tower: Environments. An Environment, is a virtual space within your SAP Build Process Automation subscription where you can run and deploy projects. It includes triggers and workflow definitions, as well as resources such as Agents producing automation jobs and workflow instances. In the default environment, users can access artifacts based on their security level. Creating shared environments to increase the security of deployed projects and the corresponding artifacts and resources is also possible. Check out more on Environments in the blog post of @Alex-Fortin.
And one more highlight on the datacenter side for SAP Build Process Automation: Switzerland (Zurich) Azure EU Access is available since mid of Q1 as a new region running on Azure for use with enterprise accounts.

Integration:

With Q1 the Advanced Event Mesh adapter with Cloud Integration capability within SAP Integration Suite got available. It enables developers to exchange events and messages with SAP Integration Suite, advanced event mesh brokers. Receive messages from queues or topic subscriptions in brokers of SAP Integration Suite, advanced event mesh through the AdvancedEventMesh receiver adapter or send messages to queues or topics in brokers of SAP Integration Suite, advanced event mesh through the AdvancedEventMesh sender adapter is now possible. Check out the new sender and receiver adapter.
With Q1 the creation of custom service-oriented architecture (SOA) messages using WSDL got released. It is an extension of mapping functionalities within the Integration Advisor capability of SAP Integration Suite and offers supports the WSDL format as input for custom message definitions and is an option for use in customer-proprietary SOA messages and customer-extended SOA messages.

Data and Analytics:

Combining the power of large language models (LLMs) with company-specific, real-time data and business process know-how, all integrated in the multi-model database? This is now possible with the SAP HANA Cloud vector engine. The vector engine includes key capabilities like similarity search, complex query support, readiness for real-time analysis, vector storage scalability. Adding the vector engine you can recognize that SAP HANA Cloud acts as the default database in SAP’s generative AI solution strategy. Learn more on capabilities, practical use cases, etc. on the announcement post, the blog post of @shabana and don´t miss to watch the video.
And the next availability of some announcements from TechEd 2023: Just Ask - our next-generation natural language query (NLQ) powered by AI. The user simply types the question in natural language into the search field and receives the required insights. The system generates charts that show the data you asked for, so you can easily understand the data. Just Ask supports SAP Analytics Cloud acquired data models and SAP Datasphere models in its first release. Want to see how simple it is? Watch the video and don´t miss the blog post of @francois_imberton.

Artificial Intelligence:

Also Artificial Intelligence (AI), one of the hottest topics at the moment provides innovation highlights in Q1. Starting with new features for Document Information Extraction. Users can now combine header fields with different setup types in the same schema, get an improved version of the conversion of country specific unit of measure values into ISO format for invoice documents, extract purchase order numbers that are available on-line item field level from invoice documents and get better models for the extraction of standard document types.

Cross SAP BTP:

Maybe you have already seen the recent enhancements on contract-to-billing traceability to the SAP BTP Cockpit? The SAP BTP Cockpit comes up with a new "Usage Analytics" page introducing an entirely new user interface and a brand-new look and feel. In addition the "Entitlements" page got enhanced. Incorporating new commercial information and additional capabilities for increased transparency to improve the understanding of entitlments and allowing to take cost-cutting measures, where needed, are just two of them. Check out the blog post of @keren_rotenberg to get more information on the new usage analytics and entitlements pages.
Customer have already been waiting for it in the area of Identity and Access Management and in Q1 it got released: The Custom Certificate Authority support for SAP Secure Login Service on AWS. Integration of own Public Key Infrastructure (PKI) by connecting to a cloud-based private Custom Certificate Authority hosted on AWS to the SAP Secure Login Service is now possible. After successful authentication of the end user, your private CA issues an X.509 certificate. And the SAP Secure Login Service then returns this X.509 certificate to the Secure Login Client on the end user desktop. Read more in the blog post of @Martina_K.
Last but not least: Enabling the configuration and security analysis (CSA) functionality of SAP Cloud Application Lifecycle Management for operations to collect a comprehensive set of technical configuration data from Identity Authentication has been relased in Q1. It reduces manual administration efforts for user management and identity provider settings and increase transparency for Identity Authentication security configuration in a central landscape management tool. Read more.

SAP BTP Innovation Highlights - outlook:

Please find here my personal innovation highlights outlook coming up:

Application Development and Automation: Generating a business service of ABAP RESTful application programming model
Application Development and Automation: Enhanced integration of HTML5 application from other SAP BTP subaccounts
Integration: Release of pending adapters for use in Edge Integration Cell
Integration: Event Mesh as new capability for SAP Integration Suite
Data and Analytics: Repointing a model in a story for SAP Analytics Cloud
Data and Analytics: Support for large instances of SAP HANA Cloud, SAP HANA database
Artificial Intelligence: Generative AI command generation
Artificial Intelligence: Automatic refinement of prompts using generative AI
Cross SAP BTP: SAP Cloud Transport Management service: transport of changes from SAP Build Apps
Cross SAP BTP: Identity and Access Management: Support for application specific groups in Identity Directory and Identity Provisioning

Check out all upcoming innovations in our SAP Road Map Explorer to learn more.

Check out more on innovation highlights:

Follow the Roadmap Explorer, which is updated on a regular basis to always stay on top of the latest innovations
Subscribe to our monthly newsletter to get all SAP BTP product news
Check out the next blog post beginning of Q3/2024 summarizing the highlights of the previous quarter
If you want to be up to date on the latest SAP BTP news and innovations check out the new monthly SAP Innobytes video series. Also want to get a recap on what´s new in SAP BTP then check out the What´s New in SAP BTP quarterly series
Mark your calendar for the latest SAP events coming up

Read other SAP Business Technology Platform blog posts and check the SAP Business Technology Platform Topic Page where you can also ask questions. Feel free to follow my profile stephanie.stang to not miss my next personal view of innovation highlights for SAP BTP for Q2 2024.

New Machine Learning features in SAP HANA Cloud 2024 Q1

2024-04-18T08:14:50.590000+02:00

With the 2024 Q1 database release, several new features have been released the SAP HANA Cloud Predictive Analysis Library (PAL), an enhancement summary is available in the What’s new document for SAP HANA Cloud database 2024.02 (QRC 1/2024).

The feature highlights for the current release are described in more detail below

Classification and Regression enhancements

Unified Regression along with Unified Classification and Time Series now supports permutation feature importance, a new and trending method in global explain-ability to evaluate the contribution of individual features to the overall predictive power of a model. This is achieved by measuring the decrease of a model’s performance when a feature‘s values are being shuffled around. A detailed explanation and examples are also given in this blog Global Explanation Capabilities in SAP HANA Machine Learning.

Classic feature importance vs permutation feature importance reports (see blog for details)

The Hybrid Gradient Boosting Tree (HGBT) now supports F1-scores, recall and precision as cross validation metric for improved, more targeted classification models. Furthermore, weight scaling of target values in classification is now supported to address imbalanced classes or weight scale target values in relation for example to different costs associated to the different class values.
A new and trending regression model objective function “reweighted square” has been introduced, aiding to achieve more robust and regularized regression models.
For improved early stopping during model optimization, the validation metric for early stopping can now be explicitly set.

The recently introduced multi-layer perceptron MLP recommender function, now supports multiclass classification and regression recommender scenarios. This allows to reformulate the recommendation task as a classification or regression problem. The implementation employs a dual-stream framework where two sets of features representing for example user – and items features, respectively, are fed into a feature selection module. The outputs are streamed into MLP-neural networks and combined in a bilinear aggregation layer. This new and trending neural network framework can handle large-scale data volumes in recommendation scenarios very effectively.

The K-Nearest Neighbor (KNN) classification and regression functions has been enhanced with a new similarity search method, in addition to brute force and KD-tree searching a matrix enabled search-method has been introduced, allowing for much faster similarity search results especially with high-dimensional numeric feature data.

Auto-ML and ML pipeline function improvements

The Auto-ML functions for the Predictive Analysis Library (PAL) have been enhanced with

a new option to trigger deeper finetuning of the best pipeline found
the genetic algorithm-based Auto-ML optimization has been enriched with a RANDOM SEARCH-based optimization, suited especially for smaller configurations (e.g. simple time series) and yielding with faster results
new method to clear and initialize the Auto-ML log
Auto-ML and pipeline model explain-ability enhancement with a SHAP Global surrogate light-weight model for faster global explanation model calculation and faster local prediction interpretability results

Text Processing

The Text Mining related document and term analysis function do now support massive parallel invocation, allowing for multiple input text to be analyzed in parallel.

Multiple documents (here IDs 0 and 5) are searched in parallel for related documents

New financial data analysis functions

The newly implemented single-factor Hull-White procedure , can be used to model the time evolution of interest rates, which are required for price estimation of financial instruments based on interest rate derivatives.

To apply the Hull-White model it first needs to be adopted to match existing market conditions (interest rates). This is achieved by providing the values of the drift term of the Hull-White model as a time series as input table. The simulation will then provide the mean value for a given number of simulation paths (also specified as an input parameter), their variance, as well as the upper and lower bounds.

The chart above depicts the initial dataset used to calibrate the mode, mean and confidence interval of the Hull-White simulation.

New Benford’s Law function in PAL, a trending algorithm used to detect anomalies in numerical datasets like e.g. financial transactions.

One of the (not so) well-known statistical observations is the fact that in many datasets the leading significant digits are not equally distributed. If all digits were represented equally, then they would appear 11.1 percent (1/9TH) of the time. However, when analyzing real-world datasets, e.g. the population totals of the US census data, it is revealed that the distribution of the leading digits follows the Bedford’s law, also known as the first-digit law.

P(d) = log10 (1+ 1/d), where P(d) is the probability of the leading digit {1,2,....9} to occur.

With the help of PAL’s new BENFORD analysis function it is now very easy to validate if a dataset obeys Bedford’s law or not. A first step means very commonly used in financial applications to detect unexpected value distribution and e.g. potential fraudulent transaction data.

Python ML client (Hana-ML) enhancements

The full list of new methods and enhancements with Hana-ML 2.20 is summarized in the changelog for Hana-ml 2.20.240319 as part of the documentation. The key enhancements in this release include

Time series analysis and forecasting methods

Time series permutation feature importance analysis
Time series outlier detection with voting
Segmented (massive) online Bayesian Change Point Detection

Auto-ML configuration and methods enhancements

Updated Auto-ML configuration dictionary-templates with new operators and random search optimization support for e.g. small time series configurations
Enhanced Auto-ML configuration option for setting connection constraints during optimization of multi-operator pipelines and visualization of pipeline connection scores between operators
Support algorithm-specific parameters with Auto-ML predict-calls, relevant for both pipeline predict and Auto-ML methods.
Enhanced progress monitor for Auto-ML to display at anytime and log management methods, allowing to set log levels, persist progress logs clean up logs and more.

Exploratory data analysis and visualization enhancements

New Bubble Plot and Parallel Co-ordinate Plot

You can find an examples notebook illustrating the highlighted feature enhancements here 24QRC01_2.20.ipynb.

SAP HANA Cloud, SAP HANA database Python Machine Learning

SAP Offers Maximum Attention for Customers’ High Standards

2024-04-18T09:14:46.671000+02:00

In today's rapidly evolving digital landscape it is large enterprises and corporations with complex IT landscapes, that find themselves particularly reliant on robust support systems.

In order to ensure maximum success, SAP MaxAttention service engineers work closely with their customers to identify their specific needs and goals. By leveraging the expertise and proven methodologies of the SAP MaxAttention program, customers can confidently navigate their cloud journey with world-class guidance and support.

To learn more about how SAP provides maximum attention to meet customers' high standards, check out this blog post: SAP MaxAttention for Customers’ Standards | SAP News Center

New Machine Learning features in SAP HANA Cloud

2024-04-19T14:39:25.280000+02:00

SAP HANA Cloud provides out of the box a rich set of embedded AI functions enabling SAP customers and partners to build Intelligent Data Applications, applications embedding AI scenarios like time series forecasting, classification, or regression and more, with applying the AI functions in the database directly on the application data.

Each new release of SAP HANA Cloud includes a lot of exciting new embedded AI features, hence see the list of links below to get an overview about the continuous improvements in the Predictive Analysis Library (PAL), Automated Predictive Library (APL) and the Python Machine Learning client (hana-ml) across the past release versions:

Review all 81 newly released Machine Learning enhancements since the 2022 Q1-release on the What’s New feature page for SAP HANA Cloud
Review planned feature on the SAP HANA Cloud roadmap page
Review the Machine Learning enhancement details in SAP HANA Cloud releases
- 2024_QRC1 (PAL, APL) 2.20 (hana-ml) blog
- 2023_QRC4 (PAL, APL) 2.19 (hana-ml)
- 2023_QRC3 (PAL, APL) 2.18 (hana-ml)
- 2023_QRC2 (PAL, APL) 2.17 (hana-ml)
- 2023_QRC1 (PAL, APL) 2.16 (hana-ml)
- 2022_QRC4 (PAL, APL) 2.15 (hana-ml)
- 2022_QRC3 (PAL, APL) 2.14 (hana-ml)
- 2022_QRC2 (PAL, APL) 2.13 (hana-ml)
- 2022_QRC1 (PAL, APL) 2.12 (hana-ml)

Here are some additional links to reference information introducing to and tutoring about how to get started using SAP HANA Cloud’s Machine Learning capabilities

Featured SAP HANA AI/ML capabilities are highlighted in the following resources:

Leverage data-parallel ML/AI, segmented forecasting from within SAP HANA Cloud utilizing
- APL Time Series Forecast using a Segmented Measure
- Segmented (massive) Forecasting applying PAL Additive Model Analysis (aka prophet) hands-on example here SAP-samples/teched2022/Getting Started with Multi-Model Capabilities in SAP HANA Cloud/Forecasting
Leverage AutoML from the Predictive Analysis Library (PAL)
- Hands-On Tutorial: Leverage AutoML in SAP HANA Cloud with the Predictive Analysis Library
- SAP Community call | Accelerate your Machine Learning efforts - benefit from SAP HANA Cloud AutoML
Integrating SAP HANA ML into your Application Code
- re>≡CAP 2023 conference presentation: "Data Science handshake with CAP developer building an intelligent data app“, Recordings of re>≡CAP 2023 session
- Auto-generating HANA ML CAP Artifacts from Python

Blog collections and sample repository
- Rich collection of references in this blogs post SAP HANA Machine Learning Resources
- SAP HANA ML sample repository https://github.com/SAP-samples/hana-ml-samples see Python samples.

Setting up your Python environment for SAP HANA ML

There are certainly numerous flavors of how to setup a local Python environment for Data Scientists to work with respective Python packages, in this case leveraging hana-ml and hdbcli to work with SAP HANA. The use of Miniconda or venv with Jupyter Notebooks is a common practice. Moreover the following instructions give guidance on how to enrich your SAP Development tools with a Python environment

Moreover, the following blog posts give indications how to leverage SAP HANA Machine Learning and hana-ml from within 3rd party Data Science, Python expert environments

Finally, reference information about the Python native SAP HANA database client can be found in SAP Note 2939501 and the SAP HANA Client documentation. The database client software is available from tools.hana.ondemand.com/#hanatools, while the Python client is also available via the pypi repository at https://pypi.org/project/hdbcli/. Further installation instructions are documented in the section Machine Learning APIs in SAP HANA Client Interface Programming Reference.

The following SAP Note 3220622 indicates which SAP HANA Cloud release is available with SAP Datasphere.

SAP Partners unleash Business AI potential at global Hack2Build

2024-04-19T17:19:54.457000+02:00

The competitiveness in the digital landscape makes the need for innovation and collaboration more critical than ever. In this context, the transformational impact of Generative AI (GenAI) almost inevitably comes onto the agenda of most business conversations these days. This was the motivation behind our recent global Hack2Build, aimed to recruit partners worldwide to explore and adopt the latest technologies and services SAP has to offer in the realm of Business AI. The objective was not just to create a platform for technological exploration, but to foster a global community of partners and innovators who create valuable solutions to customers leveraging Business AI.

Hack2Build is a rapid prototyping initiative for SAP partners to start their innovation journey with a tangible prototype, and then build a full-fledged business solution that creates business value for customers leveraging SAP Business Technology Platform (BTP). During a Hack2Build, partners validate their use cases with SAP experts, and get coaching and feedback from SAP during the build of the prototype on SAP BTP. After the hackathon, they continue developing their solutions and get them ready for go-to-market.

This edition of the Hack2Build was designed with a global outreach in mind, welcoming partners from all corners of the world. It saw a diverse pool of forward-thinking use cases, reflecting different market needs.

Our journey began with the Partner Innovation Day on January 23, dedicated to introducing our partners to the newest topics and technologies in SAP Business AI. The objective was to inspire them to develop their own use cases, harnessing the power of generative AI to enhance business outcomes.

The response was overwhelmingly positive. Numerous partners reached out, eager to submit their use cases and join the Hack2Build.

A team of experts from various fields then embarked on the challenging task of selecting the best use cases. After a rigorous analysis and evaluation process, 33 diverse partner use cases were selected.

The Hackathon Week

Hack2build participant partners

The event then moved on to the next phase and we were glad to welcome the participant partners: Accenture, AG Consultancy, Avvale, Bosch, Capgemini, Celebal Technologies, Cognizant, Crave Infotech, Deloitte, EY, IBsolution, IBM, Incture, Inetum, Inflexion, Infosys, Kinexon, LTIMindtree, Mindset Consulting, Mouritech, msg global, Netizen, NTT Data Business Solutions, Numen IT, OpenText, PwC, Reply, Scoutbee, Seidor, STMS, TCS, Tech Mahindra and Wipro.

At the Kick-off on February 28, SAP provided all participants with valuable guidance on how to leverage Generative AI capabilities within the SAP Business Technology Platform. The participant teams utilized SAP Generative AI Hub on SAP BTP, which gives instant access to a broad range of large language models (LLMs) from different providers, enabling seamless integration and usage of GenAI to enhance applications. Participants also gained early access to SAP HANA Cloud Vector Engine, which boosts GenAI scenarios with capabilities such as Retrieval Augmented Generation (RAG) to obtain better results from large language models.

Post kick-off, partners embarked on a week-long hackathon to develop their prototypes. They were supported in daily scrum sessions by SAP experts, ensuring continuous remote support throughout the week. There was little time to waste and much to prepare for the demo jam.

Here some impressions of the participants about their experience during the hackathon week:

Showtime: Demo Jam

After a week of relentless effort and prototyping, the partners had the opportunity to present their use case and live demo at the demo jam! Three demo jams were organized worldwide across different time zones within two days, on March 06 and 07.

Virtual selfie with participants

The demo jams were more than just a platform to present their work; they were a space for partners to interact, engage, and receive feedback with experts and leaders from SAP. Each participant had 10 minutes to present their use case and operational prototype. In this little time, the partners shone with business acumen, technical skills, and innovation power in front of the panel of judges, composed of senior SAP managers and experts.

Judges at the demo jam sessions

Highlighted use cases

After intense deliberation and discussion, the judges highlighted use cases for each demo jam.

In this blog you can find further details on the highlighted use cases.

Beyond a Hackathon

This global event was a testimonial to the power of collaboration and innovation. It showcased the immense potential that lies in the intersection of a cutting-edge technology like GenAI, diverse ideas, and global partnerships. As we look forward to upcoming Hack2Builds, we remain committed to support partners to drive successful outcome with their innovations.

Acknowledgment

Congratulations and a big thanks to all participating teams for their passion and tireless commitment. the use cases they prototyped during this Hack2Build. All the participants have received digital badges by Credly.com from SAP, as an acknowledgement of the excellent use case prototypes presented.

We also would like to thank all the SAP teams and colleagues for the support they offered. Their dedication, leadership and domain expertise plays a crucial role in advancing the participants beyond the prototype phase. This underscores SAP’s commitment to our partner ecosystem.

Hack2Build Team

Hack2Build on Business AI – Highlighted Use Cases

2024-04-19T17:26:32.837000+02:00

At the global Hack2Builld on Business AI in Q1/2024, teams from SAP partners dedicated themselves to developing prototypes for their use cases on SAP Business Technology Platform (BTP).

During the demo jam each team had 10 minutes to shine with their use case and operational prototype.
And shine everyone did!
No surprise that the juries -composed of senior SAP managers and experts- had a hard time picking the ones to highlight. Only after intense discussions and a holistic analysis of all presented use cases, the judges eventually reached a joint conclusion. Check out their stories below!

Deloitte

Automated Accounting Advisor

The 'Automated Accounting Advisor' solution utilizes company-specific accounting guidelines and the latest GenAI technology to provide fast and high-quality proposals for posting business transactions into the ERP system. This enables accounting departments and the CFO to deliver accurate financial statements faster and more efficiently, while also providing better decision support for FP&A/Controlling departments based on higher quality accounting data. Solution is built on the SAP Business Technology Platform, using SAP's Generative AI Hub, SAP HANA Cloud Vector Engine and SAP Build Apps for the user interface. It’s integrated with SAP S/4HANA system, to perform the document posting.

Automated Accounting Advisor by Deloitte

IBsolution

brAIn.decision_excellence

IBsolutions brAIn.decision_excellence solution identifies deviations, creates action plans and shifts the tasks of a controller towards future-oriented analysis, especially in the current solution scope for P&L analysis. Through the deep integration of GenAI into the application and extensive prompt engineering, the brAIn.decision_excellence takes into account contextual factors such as its role, task details, data of different granularities, ethical constraints and business strategies. This solution significantly reduces the time spent on repetitive controller analysis tasks and creates action plans that are tailored to your organization's unique profile and add significant business value.

brAIn.decision_excellence by IBsolution

Incture

Intelligent On-boarding Assistant

An onboarding assistant, capable of automated processing of documents and conducting an in-depth analysis and assessment for required compliance areas as per checklist. It incorporates an intelligent content search mechanism using similarity search algorithms, to retrieve pertinent clauses from documents relevant to each assessment area. The solution generates a summary of responses for each assessment area/criteria and provides an evaluation rating powered by a GenAI engine. Additionally, a dashboard interface is provided to users, offering a consolidated compliance rating of the document(s), enabling swift actions such as approval or rejection based on the displayed information. The solution leverages SAP HANA Cloud Vector Engine for storing processed information from documents. SAP Business Technology Platform Generative AI Hub & SAP AI Core and prompt engineering techniques (for LLMs used) for the assessment framework.

Intelligent On-boarding Assistant by Incture

Infosys

Voice enabled real-time safety observations reporting & analysis powered by Generative AI & Vector engine

A real-time voice-enabled system for reporting and analyzing SAP EHSM safety incidents. Users can report incidents via voice input in their native language, receive immediate safety guidance, alert & help SAP EHS incident managers in analyzing the root cause. The system considers local regulations, existing solutions, and weather conditions also in consideration.

Voice enabled real-time safety observations reporting & analysis by Infosys

LTIMindtree

Intelligent Master Data Curator

LTIMindtree Intelligent Master Data Curator leverages SAP MDG and General AI capabilities to standardize global data, enforce governance policies, reduce manual efforts, and enhance data quality, achieving over 95% accuracy in master data and improving user productivity by 30%, thereby significantly reducing technical debt and turnaround time for master data requests.

Intelligent Master Data Curator by LTIMindtree

Numen IT

Intelligent Clipping

Intelligent Clipping is a platform which helps companies to tackle Supply Chain Management increasing complexity: Diverse array of raw materials; Broad spectrum of suppliers; Regulatory changes; Unexpected events; Fluctuations in consumer trends. Utilizing Traditional and Generative AI to analyze and synthesize information from various curated sources, including both structured and unstructured data, with the objective of identifying potential risks affecting availability, costs, or ESG, aspects of materials that are critical to production processes.

Intelligent Clipping by Numen IT

OpenText

BTP Tax Compliance Service

Businesses face challenges in ensuring compliance with VAT regulations and corporate spending rules, especially concerning new product categories and suppliers. To address this, automated VAT calculations and real-time compliance monitoring offer a strategic opportunity. The solution involves implementing a Tax Compliance Service on SAP BTP, utilizing SAP BTP Generative AI Hub and SAP HANA Cloud Vector Engine. This enables tax rate determination of digitized line item information from SAP IM by OpenText. SAP Generative AI is used for generating accurate product description and similarity search then provides accurate VAT rates back to SAP IM by OpenText. The benefits include reduced invoice processing costs and increased automation levels, ensuring high levels of compliance with invoice posting standards. This integrated approach optimizes operational efficiency and aids businesses in navigating complex VAT regulations effectively.

BTP Tax Compliance Service by OpenText

STMS

Smart Hazmat shipping - Hazardous Materials Compliance

Regulated shipments face risks from complex and ever-changing regulations, leading to time-consuming operations. Our solution leverages Generative AI and Vector embeddings to streamline and accelerate these processes, reducing errors for supply management, which cuts costs, prevents late payments, and boosts customer satisfaction. For end-users in operations, it simplifies tasks and provides AI-driven support and suggestions, enhancing efficiency. Overall, our approach will increase productivity, safeguard the economic model, and improve working conditions. This innovation not only addresses current compliance challenges but also adapts to future regulatory changes, ensuring long-term resilience and efficiency for our clients.

Smart Hazmat shipping - Hazardous Materials Compliance by STMS

Way forward

As the intent of Hack2Build event is to help our partners to build their solution, the move to the BUILD phase is critical for the partners to drive successful outcome with their innovations. Participant partners plan to continue with their development and bring their solutions to customers as soon as possible.
Stay tuned for updates!

Acknowledgment

Improving Time Management in SAP S/4HANA Cloud: A GenAI Solution

2024-04-22T11:08:14.435000+02:00

Preface/Introduction - Solving Timesheet Management Problem for Beyond Technologies and AlixPartners

Collaborating closely with two distinguished firms, Beyond Technologies and AlixPartners, to streamline their time management practices has presented a considerable endeavor. Both firms are recognized for their contribution to the services industry, wherein they are often required to engage with SAP S/4HANA Cloud for project time management. Despite the advanced features this system offers, they found themselves struggling with the management of their timesheet tasks.

We developed a GenAI-powered SAP BTP application for real-time tracking of project tasks, seamlessly integrating with SAP S/4HANA Cloud, public edition, for automated timesheet entries and billing. This system allows users to immediately log tasks upon completion, which are then processed in natural language and drafted into their timesheet via SAP S/4HANA Cloud APIs and the GenAI Hub on SAP BTP. Our application operates independently alongside any SAP S/4HANA timesheet app, ensuring compatibility with custom apps as well as existing and future standard versions. Users can easily submit their time records for approval within their company's timesheet application.

Business Scenario/Challenge – Addressing Timesheet Record Management in SAP S/4HANA Cloud

The critical problem both companies faced was associated with their timesheet record task in the SAP S/4HANA Cloud system. Delayed and after-the-fact time entries resulting from manual user operation led to inaccurate entries. It was time-consuming and negatively impacted work efficiency, delaying billing and payment processes as well. It was crucial to resolve this challenge as it directly linked with their business' operational efficiency, and therefore, profitability.

Co-Innovation/Solution – Creating an GenAI powered application with SAP AI Core

Generative AI significantly enables our solution by empowering users to log their project time effortlessly in their own words, without strict syntax. It intelligently interprets natural language inputs, crafting accurate payloads for S/4HANA system API calls. This flexibility ensures users can continue their varied task tracking practices across multiple languages, maintaining efficiency without the need for predefined structures. Generative AI thus enables an easy way to track consultant’s own time and adding it to the S/4HANA system already. It preserves the intuitiveness and accessibility of our task tracking, transforming user experience and data entry reliability.

Understanding the customers' need for efficient time management and with the power of Generative AI, we set out to create an application powered by the robust capabilities of SAP Cloud Application Programming Model (CAP) and SAP UI5. This application integrates seamlessly with the SAP Generative AI Hub in SAP AI Core via Destination Service and embeds with CAP LLM plugin(A free and open-source tool created by our team. Click on below drop-down for more information). Meanwhile all conversation data is managed by SAP HANA Cloud.

Spoiler

Introducing the CAP LLM Plugin:

An indispensable solution empowering developers with the essential tools to effortlessly craft robust AI-driven CAP applications. Seamlessly integrating the LLM capabilities of SAP Generative AI Hub with the vector engine and anonymization features of SAP HANA Cloud, this plugin revolutionizes the development journey.

The CAP LLM Plugin is a powerful tool that empowers developers to craft customized Generative AI-based CAP applications seamlessly. It ensures that sensitive data remains confidential by utilizing SAP HANA Cloud Data Anonymization, thereby safeguarding privacy while leveraging the capabilities of LLM. With this plugin, developers can confidently create innovative applications without compromising data security or integrity.

https://github.com/SAP-samples/cap-llm-plugin-samples/blob/main/README.md

https://community.sap.com/t5/technology-blogs-by-sap/cap-llm-plugin-empowering-developers-for-rapid-gen-ai-cap-app-development/ba-p/13667606

Introducing the CAP LLM Plugin: An indispensable solution empowering developers with the essential tools to effortlessly craft robust AI-driven CAP applications. Seamlessly integrating the LLM capabilities of SAP Generative AI Hub with the vector engine and anonymization features of SAP HANA Cloud, this plugin revolutionizes the development journey. The CAP LLM Plugin is a powerful tool that empowers developers to craft customized Generative AI-based CAP applications seamlessly. It ensures that sensitive data remains confidential by utilizing SAP HANA Cloud Data Anonymization, thereby safeguarding privacy while leveraging the capabilities of LLM. With this plugin, developers can confidently create innovative applications without compromising data security or integrity.https://github.com/SAP-samples/cap-llm-plugin-samples/blob/main/README.md https://community.sap.com/t5/technology-blogs-by-sap/cap-llm-plugin-empowering-developers-for-rapid-gen-ai-cap-app-development/ba-p/13667606

As soon as a user logs in, the application collects all the projects to which the user is staffed. Then the user can simply report his or her time on those projects via natural language. The application will process the user’s input, returning a valid JSON payload. This payload is then used to post a time record in the SAP S/4HANA Cloud system effectively bypassing the manual input.

Main features of the application:

Embedded analytics that display the completeness of the recorded hours for the day and week
User can view the history of time recording conversations, including their status. If any time recordings are incomplete, you can complete them in the chat or in the S/4HANA timesheet.
You will see a main window where you can provide all the necessary information for time-recording by typing or using natural language. The required information includes the project, the work package (in case of multiple assignments within one project), the date of service, and the hours worked.

An example interaction between user and application.

User clicks on the project name button for one of the projects the user is staffed on.
User inputs their intention for booking time for the project with natural language.
Interact with application to ensure all necessary information is provided.
The application creates the time record on SAP S/4HANA Cloud via API.

Conclusion/Summary - Efficient Time Management Achieved Through Co-Innovation

By creating a natural language interface application to do time management in SAP S/4HANA Cloud system, we have enabled Beyond Technologies and AlixPartners to manage their timesheet tasks more efficiently. The application delivers accurate time records through a simplified and quick process, significantly reducing the time spent on manual data entry.

I welcome more inquiries on this solution and the benefits it can bring to your business operations. Feel free to reach out to paa@SAP.com for further information or any questions.

SAP Datasphere's updated Pricing & Packaging: Lower Costs & More Flexibility

2024-04-26T13:59:19.616000+02:00

The power of SAP Datasphere is growing every day. Bringing together data warehousing, data integration, data catalog, and semantic modeling it perfectly fits the definition of a modern business data fabric.

SAP's most recent commercial update announced a commercial reset of how SAP Datasphere Tenant Configuration works. Additionally, they announced the BTPEA (=BTP Enterprise Agreement) model as a replacement for CPEA.

This blog provides information on the updated packaging & licensing for SAP Datasphere. The main focus lies on the improved flexibility in configuration and pricing depending on your SAP Datasphere use case. Moreover, the blog also covers the freed-up Capacity Units (CU) for existing SAP Datasphere customers.

Introduction of a smaller minimum configuration

In the past, an SAP Datasphere license started at 4300 CU / month. Furthermore, the extension of the tenant was processed in tiers, meaning you could not configure components individually. As TCO is a crucial component in the decision-making process in an IT Transformation, this created a barrier for a lot of stakeholders.

Starting at 1532 CU / month, the minimum capacity for a SAP Datasphere Tenant has decreased considerably as of April 15. This configuration by default provides you

Storage: 8 blocks of 16 GB disk storage
Compute: 2 blocks of 16 GB of RAM
Data Integration: 200hours/month in 2 parallel jobs

In the next section, you will learn how you can dynamically extend this package considering every component separately.

Flexible Tenant Configuration

SAP Datasphere offers high flexibility to scale up / scale down your tenant (=FTC). Depending on your use case you can configure:

Every component can be extended based on the below Consumption Blocks:

Start configuring your tenant now through SAP Discovery Center

Introduction of Performance Classes

In the compute section of the tenant configuration of SAP Datasphere, you can now choose the performance class you wish to implement:

Memory (default ratio - 1:16): Suitable for most basic workloads (2 vCPU & 2 blocks of 16 GB of RAM)
High Memory (1:32): Optimized to support big data sets that require a lot of memory
Compute (1:8): Optimized to support compute-intensive workloads
High Compute (1:4): Optimized to support compute-intensive workloads that require fewer memory resources

Visual representation of how the Default Performance Class looks:

This is a critical change that sets the future for enabling AI & LLM use cases alongside SAP Datasphere. You can set the correct infrastructure to work out your AI use cases. It provides the necessary flexibility to start small with the potential scalability in the future. This change goes hand in hand with the newly released SAP HANA Cloud Vector Engine. (More details on this in a separate blog).

What with existing tenants?

For existing SAP Datasphere customers, it is highly recommended to check out your tenant configuration through system -> configuration -> tenant configuration. You will most probably be leveraging a lower percentage of your current Capacity Units than before. This means that you can boost the power of your SAP Datasphere tenant without exceeding the limits of your current CU / Month license. More info can be found through SAP Note.

An example:

Conclusion

The pricing program for SAP Datasphere has substantially become more compelling for organizations to take the first step into a business fabric. The reduced minimum configuration removes entry barriers but preserves upscale opportunities.

Thanks to my colleagues simon.vandelanotte@delaware.pro & arne.feys@delaware.pro for the co-creation of this blog.

Some useful links:

Free Basic Trial

SAP Datasphere Roadmap

SAP Datasphere Product Page

SAP Datasphere SAP Business Technology Platform SAP HANA Cloud

HDI Artifact Recovery Wizard

2024-04-27T17:59:57.243000+02:00

Potential use cases are:

Overwrite corrupt, broken or otherwise unusable HDI design time files in your project with their last successfully deployed versions.
Populate a project's database module with HDI design time files as they are currently successfully deployed in the HDI container.
Understand the content of an HDI Container without any source code (design time) be stored in GIT

Prerequisites:

Business Application Studio project with a HANA database module.
HDI container available via Cloudfoundry service of type "hdi-shared".
Database module bound to the "hdi-shared" service.

Detailed instructions on how to use the Artifact Recovery Wizard when working in Business Application Studio can be found in SAP Note 3269751 and the SAP Help Portal ("Restore Design-time Artifacts from the Database").

The Artifact Recovery Wizard is considered an expert tool and should be used carefully:

The Artifact Recovery Wizard reads and, if selected, restores the HDI designtime files in their last successfully deployed version. It cannot restore versions of files for which the deployment failed, as they are not part of the corresponding container filesystem.
Status "modified" means that a file exists in both BAS project and HDI container and the file versions differ. As there is no further information provided about which version is newer the decision which files with this status are eventually restored should be taken with care.
The Artifact Recovery Wizard was not designed to synchronize changes across multiple projects deploying into the same HDI container. Such a scenario carries an increased risk of unintentional object corruption and data loss and is therefore not recommended.

How to use Artifact Recovery?

Using BAS/HANATools:

1.) Create a new project (in my case "LostWorkspace") and bind it to the HDI-Container your want to recover.

"SFlight" in my case.

Notice that there are no sources (src folder)

Now press F1 or SHIFT+OPTION+P (on a MAC)

and type now "Recover"

and select SAP HANA: Start Database Artifact Recovery:

Choose the directory where you have your .env file located

The wizzard will warn you in case that directory does not contain the .env file or
fill immediately the list with all objects in that container

select all file
hdiconfig is by default in your project. By default overwrite the existing one with the older one from the project that was deployed before. There are reasons to keep the file. This is another discussion. Anyhow nothing onw can mend later as well.
the list shows the "lost" missing object in your project.
Check all the get the project back.

at the end of the list you can find the "Continue" bottom

The result:

A backup of the existing content (nicely done by the developer not to overwrite existing version)
All the objects that are inside the container. In my case I recovered a CAP project. That is why we have a "gen" folder.
1. This was intentional to show: A reverse of the HDI does not give you back all your source code. Just the bit that is used in the database to generate the DB artifacts.
2. The CAP CDS has some smart generation concepts. Is is a logical data model. You will not get this back by recovery. THIS METHOD is not a replacement for a proper GIT repository!
3. It is very good for your own HDBTABLE, HDBVIEW, HDBCALCULATIONVIEW etc.

Do not forget to UNBIND the HDI Container now unless you are sure you want to overwrite the content. Recommendation to create a new HDI-Container now to continue.

Summary:

If for any case you want to look into an HDI Container and get back the source objects

ARTIFACT RECOVERY is your friend. It does not replace by any means a proper version system like GIT.

Hope you find this option helpful. Give it a try, so you known how to use this in emergency situations.

Volker and Bastian

Explore the SAP HANA Cloud vector engine with a free learning experience

2024-04-29T22:40:00.482000+02:00

The next version of the popular 30-day basic trial is now available for anyone wanting to build a modern data foundation for AI-powered solutions.

Know, learn, and experience SAP HANA Cloud

With the release of technologies like the SAP HANA Cloud vector engine, SAP BTP solutions are equipping customers and partners to build the next generation of intelligent data applications. Simply put, these types of applications fully utilize the database to enhance the user’s experience as well as execution of the business process. Example database features include the integration with all types of business data, multi-tiered data storage, multi-model processing, and machine learning.

During this experience, a step-by-step tutorial provides guided hands-on learning for topics such as:

Intelligent Data Applications
In-Memory Column Tables
Data Tiering
Multi-Model: Document Store
Multi-Model: Spatial Engine
Multi-Model: Graph Engine
Multi-Model: Vector Engine*
Machine Learning

* The vector engine topic includes an introductory exercise as well as an example that combines the use of more than one multi-model engine. For a fantastic introduction to using vectors, check out this blog.

As part of the experience, each registered user receives access to a private schema on a shared cloud database along with all the data needed to complete the exercises. Once finished, the user can then continue to access the schema for the remainder of 30 days. This extra time provides a great opportunity to use the assigned schema for independent study. For those who like a challenge, try to discover how many of the advanced multi-model engines can be included in a single SQL statement.

This interactive opportunity is ideal for any database enthusiast whether currently enrolled in introductory database courses or a seasoned professional. Note: some SQL familiarity is recommended as many exercises make use of this industry standard language.

Click here to register and experience SAP HANA Cloud firsthand. Discover how this amazing database is enabling companies to deliver applications that focus on the success of the user.

ACH/ WIRE RETURN Process in S/4 Hana

2024-05-07T09:22:40.375000+02:00

ACH/ WIRE RETURN Process in S/4 Hana

Problem Statement:

ACH or wire returns are received in Electronic Bank Statement for the paid invoices. The return process needs to be applied to the invoices and reset the payments.

Since returns are usually not received soon, the actual payments are already cleared/reconciled with bank statements.

To reissue the correct payments, the reversal original payment would not possible unless bank reconciliation postings are reverted which if done can open all payments received in that bank statement. To avoid such hassles, Account Payables team posts duplicate invoice.

Adverse Impact of the existing process:

Duplicate postings for one invoice
Account Payables & Bank Reconciliation issues.
Delays in Audit Reporting
Issues with 1099 reporting due to over tax exposure for supplier due duplicate invoice postings & payments.

The following document suggests the best way to manage ACH/Wire returns:

Instructions for the Treasury Team

Step1: Apply the ACH/Wire return received in Bank Statement using “Reprocess Bank Statement Items” Fiori App”

Step2: Select “Post to G/L Account” and then select “Add.” Enter and Post ACH/wire return amount to Accounts Payable adjustment GL account viz. 2100XXX.

Step3: Select “Simulate” to ensure the accounting and posting date are correct and post.

Step4: Now, we need to find the original ACH/wire payment which cleared in SAP. Get the bank reconciliation clearing document from there and reverse the entire batch. The posting period should be the date of the ACH/wire return. This is needed to allow reversal of the payments.

Step5: Reprocess all the payments under the ACH payment batch except for the returned payment (note: this step is not necessary for wire payments). Select “Post to G/L Account” and debit same AP adjustment GL account 2100XXX for the returned ACH/wire payment amount.

Step6: Select “Simulate” to ensure the accounting and posting date are correct and post.

Step7: The Treasury Team will send an email to the Accounts Payable Team to show a payment was returned and provide the supporting information from the Bank.

Instructions for the AP Team

Step8: Confirm A/P invoice is open- Reset and Reverse the payment document.

Please note if the ACH/wire return occurred during A/P close, the A/P team will need to book the following reversing journal in GL SAP, until A/P opens in the following period. This Journal entry will auto reverse in first working day of the following month.

Dr ACH/Wire Clearing Account XXXX
Cr AP adjustment Account XXXX

AP team will reset and reverse in the first working day of the following month.

If ACH/WIRE return received on 28th of Feb-2024, AP will book a JE in SAP GL with posting date of 28th Feb-2024. This JE will be reversed on 1st March -2024. (First working day of the month). AP will reset and reverse the payment document with the with the posting date of 01st March-2024.

Original Payment for Returns

Step9: Go to the “Reset cleared items” App window displaying the payment document to be reversed.

Step10: Select the “Reset and Reverse” option (use the date of the return for the posting date) in case in same month.

Step11: Now you can see payment reversal document.

Step12: Confirm reversed document. The original invoice is open to repayment and no duplicate invoice needs to be posted.

Once, reason of payment failure is resolved you can repay the supplier.

Please find below the accounting entries for the ACH /Wire return process

Debit Credit

1) Initial Invoice processed in AP
Expense		2,216.12
	A/P		-2,216.12

2) Initial Invoice paid in AP
A/P		2,216.12
	Bank Clearing Account		- 2,216.12

3) Initial Payment Cleared (EBS File)
Bank Clearing Account		2,216.12
	Cash Account		-2,216.12

4) ACH Return (EBS File)
Cash Account		2,216.12
	AP Adjustment Account		-2,216.12

5) Reverse Initial Payment Cleared (EBS File)
Cash Account		2,216.12
	Bank Clearing Account		-2,216.12

*AP Adjustment Account*		2,216.12
	Cash Account		- 2,216.12

7) AP Team Reverse Initial Payment
Bank Clearing Account		2,216.12
	A/P		- 2,216.12

8.AP Team Reissues Payment
A/P		2,216.12
	Bank Clearing Account		- 2,216.12

9) Reissued Payment Cleared (EBS File)
Bank Clearing Account		2,216.12
	Cash Account		- 2,216.12

The ACH/Wire return process explained above is simple and easily adoptable. It helps us resolves all the adverse impacts of faulty process and ensures accurate reporting. Hope, this document equips you well to tackle his problem in future.

Embedding Business Context with the SAP HANA Cloud, Vector Engine

2024-05-07T12:27:05.550000+02:00

Replay: Embedding Business Context with the SAP HANA Cloud Vector Engine

Here we will see the methods to ground & adapt Generative AI to business context though Retrieval Augmented Generation (RAG), extending to external domain knowledge by retrieving & injecting information via embeddings thanks to the SAP HANA Cloud, Vector Engine.

Here you will find the source code of both prototypes: SAP-samples / btp-generative-ai-hub-use-cases

The basics

Ok, let’s start with a quick warm-up with the basics: you know that LLMs are trained on vast amounts of text data from various sources, which allows them to generate responses and perform tasks across a wide range of topics.

However, they don't naturally understand the nuances of specific industries or domains. So, while LLMs can be valuable tools for generating text and performing certain tasks, they may require additional guidance or input from domain experts to ensure accuracy and relevance in business-specific contexts.

The question you might hear from customers worldwide and that I pose to you here is:

How do I make the large language model smarter
by adding details about my business and stop it from hallucinating?

Here are some known key techniques to enhance the output of large language models (LLMs). We have categorized them based on how well they optimize context and the model itself.

Let's start with one of the fundamental strategies: prompt engineering.

Prompt engineering

Prompt engineering involves crafting specific instructions or queries to guide the model towards desired outputs. It's like giving clear directions to get the best results. One advantage is its simplicity, it requires less technical expertise and computational resources, but a downside could be the need for some trial and error to find the most effective prompts.

Model fine-tuning

Then we have model fine-tuning. This is all about training the model to better fit the requirements at hand, to perform a NEW SKILL or NEW TASK that is just hard to articulate in a prompt. It’s like showing the model HOW to do it, not TELLING how to do it. It can require a good understanding of the model's inner workings, it’s time and resource consuming, therefore VERY COSTLY.

Retrieval Augmented Generation

Moving up on the context optimization axis we have the Retrieval Augmented Generation (so called RAG). This technique enhances the model's capabilities by providing more context to the given task. It's like providing the model access to a library of information to help it generate more accurate outputs. One advantage is its ability to generate relevant and domain-specific content, but it HEAVILY DEPENDS on the quality of external corpus used for retrieval.

By using a mix of prompt engineering, model fine-tuning, and RAG, companies can leverage the strengths of each approach to further improve the performance of the LLM.

Let's focus on RAG. The key characteristics of RAG includes:

Reduce Hallucinations: RAG can mitigate the occurrence of hallucinations or incorrect outputs by cross-referencing the generated content with the retrieved information, thereby improving the overall accuracy of the model's responses.

Increased Knowledge: Even if the base LLM has not been trained on certain information, as long as that information exists in the corpus used for retrieval, RAG can still provide relevant answers.

Flexibility: By changing the underlying corpus, RAG can be adapted to different domains or knowledge bases.

Memory Efficiency: Instead of having to increase the size of the language model to store more information, RAG leverages external data sources, keeping the model size manageable.

Let’s see that in action with a quick and very simple example here:

I asked the LLM since what quarter SAP HANA supports native vector datatype for running vector similarity. The answer was wrong, there was no such support in 2020.

Then I provided the context, in this case, the actual release date. And asked the same question. The model took the date and converted to the right quarter, providing me with the correct answer.

So, the concept is clear: if I want the model to be precise, I need to provide the information as the context to the questions I want it to answer.

However, the challenge is: where and how do I store all that information that I want to send to the model, and ALSO how to retrieve the right piece according to the question asked.

Every company store their unstructured information in different formats. So think about that bunch of pdfs, web pages, word documents. We want that to be our source for the RAG technic.

To store them in a format that can be meaningful to the technics we want to apply, we need to convert them to embeddings. Embeddings are a way to represent text in numerical format, typically as vectors.

Let’s use few words as an example - In the first column, you'll notice we've grouped some of them together based on their meanings, and we've color-coded them for clarity. Alongside each word, you'll find its corresponding embedding.

We have simplified them in a 2-dimension embedding representation for better understanding of the concepts we want to demonstrate.

If we were to plot them on a Cartesian graph, here's how they'd appear.

Notice how these words are all clustered together on the graph - It's like they're grouped up in the same area. We can also see their vector representation, which basically shows their 'direction' in the space. Pretty interesting, right?

Alright, so how do we figure out how similar these words are? There are a few methods, but let's focus on two for now.

The first method uses the COSINE of their angle:

And the second one uses the Euclidean distance:

Understanding this is important because it helps us do a lot of useful things like classification, group them together, spot any odd ones out, do semantic searches, and even use it to give recommendations. We can for example group documents, or chunks of documents together to find common topics of interest.

Let's break it down with a practical example. To understand how to use these steps effectively, first, we need to create smaller sections of documents like paragraphs or sentences, also known as chunks, and turn them into embeddings.

Then we save them as vectors, in a vector store.

Once we receive a query from the user, we turn it into a vector and then compare it with all the other vectors in our database. We'll then filter down to just a few, just the most similar ones.

And with those, we enrich the prompt, just like you see here.

Now, let's incorporate the technology into those steps. This diagram is something you'll definitely want to keep handy, like sticking it on your workstation:

In summary: we first generate the Text Chunks, then we convert to embeddings, and save them as vectors in the SAP HANA Cloud, using the Vector Engine. When the user’s query comes, we generate the embeddings and run a similarity search with the vectors we have stored.

Best matches goes together with the query to the LLM that provides the answer to the user.

Now, this is very important to understand: in orange you see the steps we need Generative AI Hub to generate the embeddings and provide the answer.

The model responsible for converting text to embeddings is called Text Embeddings model, in this case here you see ADA-002. And the model responsible for generating the text competition for the final answer is GPT-4. Both accessible, again, through the Generative AI Hub that you have seen in the fist session of this series. These are the models we have chosen and you can always opt for different ones available in the Generative AI Hub – please refer to this link to see the supported models.

Good, now we want to see how to store the embeddings in a vector format and run the similarity search. And this is the role of our SAP HANA Cloud. It will provide us with the right functions and data formats:

SAP HANA Cloud, Vector Engine

The Vector Engine, released last March on the SAP HANA Cloud, provides a new datatype to store high-dimensional vectors and two distance functions to compute vector similarity, and one of the use cases for those features is the Retrieval Augmented Generation.

The new datatype is called REAL_VECTOR and consists of real elements that supports up to 65000 dimensions.

You can get a REAL_VECTOR by using the function TO_REAL_VECTOR by passing either a text, a binary representation or an array.

The distance functions available are the Euclidean distance, represented here by L2DISTANCE which will result in a double number greater or equal to zero. Basically, the closer this number is to zero, the more similar the vectors are.

And the COSINE_SIMILARITY – which results in a double between -1 and 1. The greater the result, the more similar are the vectors.

You can consume them using standard SQL script, Python through hana-ml library or Langchain, and also using CAP.

And we close the introduction looking at the main benefits of Vector Engine: It combines vectors and results of fast vector search with business data to enable specific-domain use cases.

Next up, we will now land this into a practical use case implementation with code, architecture and a cool demo.

Use Case 1: Deduplication of citizen reported issues

We start from the previous session in February where we presented a proof of concept application using Generative AI HUB. The goal of the Citizen Reporting application is to assist the Maintenance Manager, Mary, by extracting insights from citizen social media posts, classifying them using Generative AI, allowing for the creation of maintenance notifications in the SAP S/4 HANA Cloud tenant when the incident is approved.

However a problem presented itself … In this scenario, the maintenance manager, Mary, faces a challenge where citizens frequently report incidents via social media. These reports often pertain to the same issue but originate from different individuals. Mary is tasked with reviewing both duplicate incidents and newly reported incidents in order to determine which ones should be submitted to the backend S/4 HANA maintenance system.

The solution to help Mary in her task, will be to utilize the SAP HANA Vector engine to analyze all incoming social media incidents and identify duplicates or similarities among reported incidents. When duplicates or similarities are identified, a notification will be attached to the incident when viewed in the application, enabling Mary to easily distinguish them from new reports. This approach aims to streamline the incidents requiring Mary's review, enhancing efficiency in the overall process.

John reports an incident by creating a post via social media to the city’s community page using the social media to raise attention about an issue in the community.

The citizen reporting application receives John's post, examining its similarity to other member posts as identified in the SAP HANA Vector Database. If a duplicate incident is detected, the incident will be flagged as “duplicated” as to be easily identified by Mary, the maintenance manager, when she is viewing/managing incidents.

Both similar and dissimilar entries, as compared using the SAP HANA Vector engine, will undergo processing and analysis by the corresponding Large Language Model via SAP Generative AI Hub. This analysis entails extracting key incident details such as issue summary, type, urgency, location, and sentiment.

Subsequently, Mary, as the Maintenance Manager, reviews both new incident details and flagged duplicate incidents where she can then determine whether to approve the incident(s), reject the incident(s) or Link the duplicate or similar incident(s) to an existing S/4 HANA Cloud Maintenance Notification thereby taking action on the reports from citizens but not creating Maintenance notifications for the same reported incidents in the backend system.

Now, let’s have a look at a live demo of the Citizen Reporting Application in action:

As mentioned earlier, in the previous SAP Generative AI session we utilized, SAP Build apps for the client front-end application, the orchestration citizen reporting application on cloud foundry as well as SAP HANA for storage. For this use case we have extended the application to now use the SAP HANA Vector Engine on top of the existing solution for the deduplication by vectorizing the posts.

Let’s look at the designed architecture and how the different services interact between them.

Our Citizen Reporting app is composed by:

A server side application that runs on SAP BTP Cloud Foundry. It can be developed with the language of your choice, in our proof of concept we implemented 2 versions: NodeJS and Python.
A user interface developed with SAP Build Apps, that can be easily deployed in SAP Build Work Zone.
The server side is consuming APIs from different BTP Services: from SAP AI Core service, providing access to different foundations models (remote or hosted) in a trusted and controlled way through the Generative AI Hub capability.
And to securely connect to the SAP AI Core APIs: the Destination service is leveraged.
We also consume the SAP HANA Cloud, Vector Engine through the Cloud Application Programing Model to perform the similarity search needed for the deduplication of issues as well as to store information about the incidents.
To create Maintenance Notifications in SAP S/4HANA Cloud we are consuming SAP S/4HANA Cloud OData APIs also secured through the Destination service.
The SAP Cloud Identity service allow us to share the same Identity Provider among the different components (including SAP S/4HANA Cloud) and to manage authorization and authentication for our application.

Use Case 2: Ask the City

Next up, let's explore into the second use case of how SAP HANA Cloud vector engine could be simply be utilized for citizen's queries!

Citizens like John have the opportunity to submit inquiries to the city through social media platforms like Reddit’s Subreddit channel - Ask SAGenAICity and subsequently receive responses directly on the same platform.

This capability is facilitated by the vectorization of SAGenAICity's knowledge repository, presently accessible through the city's website in formats such as HTML, PDF, and other documents, into the SAP HANA Cloud Vector Database.

Problem: John has questions that he needs to contact SAGenAICity about but he does not know whom or what departments to contact within the city offices.

Solution: In order to streamline the process of addressing John’s questions and guiding him toward appropriate resources, he asks his question(s) through SAGenAICity's community channel on Reddit. This city utilizes the SAP HANA Vector Database to search for similar questions/queries and delivers relevant answers directly to John via Reddit.

Once John’s question is submitted and the citizen reporting application receives the question the system undergoes a process whereby it evaluates whether the question aligns with similar data within the knowledge base of the city, which has been embedded within the SAP HANA Cloud Vector Database.

If similar information exists within the SAP HANA Vector Database, it will be relayed to SAP Generative AI, which will then generate a response tailored to John and the generated response will then be posted back to John via the Reddit social media platform.

By embedding SAGenAICity’s knowledgebase into the SAP HANA Cloud Vector database many questions can be answered automatically without intervention by a SAGenAICity employee thereby freeing up city employees to only have to handle exception questions that may not be part of their knowledgebase. It also enables citizens reporting issues yet another channel to resolve issues.

Now let’s have a look at a demo of the Ask SAGenAICity solution in action:

Here is the solution architecture diagram, similar to the previous one.

The only difference is the python microservice, deployed in the Cloud Foundry runtime, that is actively listening to any new posts in the Reddit page. It is a pretty simple & straightforward solution in this use case.

Please note that the webpages have been chunked using langchain library for the text chunks generation.

Consumption Methods

As we mentioned earlier, there are various consumption methods for SAP HANA Cloud, Vector Engine:

If you are curious to learn more, we explain the source code of both use case implementations in the replay of our session here:

Replay: Embedding Business Context with the SAP HANA Cloud Vector Engine

Here you will find the source code of both prototypes: SAP-samples / btp-generative-ai-hub-use-cases

Do not miss it.
Stay tuned!

Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive

2024-05-08T17:00:00.007000+02:00

Introduction

This blog post is part of a series describing the usage of MLflow with HANA Machine Learning co-authored by @stojanm and @martinboeckling. In this blog post we provide a more technical deep dive on the setup of a MLFlow instance and provide a general introduction how Machine Learning models trained with HANA ML can be logged with MLflow. The first blog post of the blog series is called Tracking HANA Machine Learning experiments with MLflow: A Conceptual Guide for MLOps and gives an introduction to the topic of MLOps with MLflow.

Starting with the python HANA ML package version 2.13, HANA Machine Learning added support for tracking of experiments with the MLflow package, which makes the incorporation of models developed using HANA Machine Learning into a comprehensive MLOps pipeline easy to achieve.

In this blog post we will provide an overview how MLflow can be used together with HANA ML. MLflow, which manages the experiment tracking and artefact management can run as a managed service at a hyperscaler platform, deployed locally or on a remote infrastructure. In the following we describe how to deploy MLflow on SAP Business Technology Platform and how to track your HANA machine learning experiments with MLFlow. In addition, we present which methods and algorithms in the hana ml package currently support the experiment tracking feature. Finally, we touch on the possibility to use logged models in MLflow for prediction.

Prerequisites

In this blog post we solely focus on the technical integration of HANA ML and MLflow as a logging platform. Generally, we assume that Python is already installed together with an already established development environment. Furthermore, we will not completely explain all details of docker and Cloud Foundry, but simply focus on the essential parts for HANA ML and MLFlow within this blog post.

Set up MLFlow on BTP

MLFlow is leveraged and integrated in different solutions. For example Databricks as well Machine Learning in Microsoft Fabric integrate or provide a managed MLflow instance. In case MLFlow is not yet provided, we outline in this section a possibility to deploy MLflow in SAP BTP. We focus for simplicity reasons on the SQLite based deployment of MLflow. However, for productive environments it is recommended to separate the storage from the runtime of the MLFlow instance. A detailed explanation for setting up your own MLFlow server with alternatives to SQLite can be found under the following link: https://mlflow.org/docs/latest/tracking/server.html. In the following paragraphs, we focus on a step by step overview to set up your own MLFlow instance using SQLite in BTP.

As a first step we want to create a local docker file which can be used to upload it to our BTP environment. In the following code snippet, we provide the coding used to construct your own docker container locally. For that, paste the following code into a file called Dockerfile within your desired local folder.

# Retrieve a python version as a base runtime for our docker container
FROM python:3.10-slim
# Run the pip install command for the package of mlflow
RUN pip install mlflow
# Create a temporary folder within our docker container to store our artifact
RUN mkdir -p /mlflow
# Expose the port 7000 to make our application which runs within docker accessable over the defined port
EXPOSE 7000
# Define environment variables BACKEND_URI and ARTIFACT_ROOT to define the backend uri as also the 
# artifact root
ENV BACKEND_URI sqlite:///mlflow//mlflow.db
ENV ARTIFACT_ROOT /mlflow/artifacts
# run the shell comand to setup our mlflow server within our docker container
CMD mlflow server --backend-store-uri ${BACKEND_URI} --host 0.0.0.0 --port 7000

After creating successfully the Docker object, you can run docker build -t {tagname}. to construct your docker container. Afterwards, the local docker image is locally built. To expose the docker image, we push the image to a docker registry. In our example, we assume that you already have a docker registry set up where you can push your image to. For that step, you can run the following commands: docker tag {tagname} {dockerhub repository tag}, docker push {dockerhub repository tag }. After the successful run of the command, you see within your private docker hub the newly published docker container, which contains MLflow and all its dependencies inside of it.

After the successful publishing of your docker image to your registry, we can run the following command to create a BTP app based on the published docker image:

cf push APP-NAME --docker-image REPO/IMAGE:TAG

After successfully publishing the docker image on our BTP Cloud Foundry environment, we can find our published app within our BTP account and are able to access it under the published URL.

Set up tracking for MLflow

With MLflow users have the possibility to track their trained HANA ML models. In the following paragraph, we introduce the aspects that are needed to be able to log HANA ML models into MLflow itself.

To be able to use MLFlow together with HANA ML, we need to first install besides the HANA ML package also the MLFlow package. Therefore, you need to run the following command in your virtual environment, to be able to run the following scripts.

pip install mlflow hana-ml

As a general setup, we first need to run the following command to set up our tracking with MLFlow to our available MLFlow instance. Therefore, place into the following two lines first your personal MLFlow tracking URI and your own custom experiment. In case you do not want to create a separate experiment, the different runs together with the MLFlow model are stored under the default experiment.

The method that allows us to track HANA ML models is implemented in the HANA ML package and is called enable_mlflow_autologging(schema=None, meta=None, is_exported=False, registered_model_name=None). This method can be used for initialised HANA ML models that are under the following methods:

Within the method enable_mlflow_autologging the user has different keywords that can be filled that allows us to influence the behaviour of our MLFlow autologging in HANA ML.

schema: Defines the HANA database schema for MLFlow autologging where the MLflow logging table is stored
meta: Defines the name of the model storage table in HANA database
is_exported: Determines if the hana model binaries should be exported to MLflow
registered_model_name: Name of the model stored in MLflow

In the following section we provide an overview for the Unified Interface how the logging of MLflow can be used.

Run HANA ML Algorithms with MLflow

As we have explained and outlined in the sections above, we have created a MLFlow instance and have introduced the syntax that is needed for the logging of HANA ML models in MLflow. In the following sections we will provide based on an example how the logging of HANA ML models on MLflow is done.

Model training of HANA ML with MLFlow

For the training of HANA ML in combination with MLflow, we focus in this blog post on the Unified Method. We apply for the respective elements a Classification on the sample bank dataset which can be found under the HANA ML sample dataset folder on GitHub.

The dataset can either be uploaded directly to the SAP HANA database or you could also use SAP Datasphere as your starting point. Generally, to use HANA ML directly you would need to store the dataset in a HANA database. However, HANA ML also provides methods to integrate third party files/ data structures. This involves Pandas, Spark as also shapefiles. In addition also HANA Data Lake file tables can be integrated with HANA ML functionalities. An overview of the different methods can be found under the following page. In the following paragraphs, we will go through the sample code that we have created to combine HANA ML and MLFlow.

Connect to HANA database (Deployed under SAP Datasphere)

To be able to connect to the HANA database instance, we first need to build up a connection to the HANA database. In our example, we load the data from the data samples provided by HANA ML. During the time of this blog post, the OpenSQL schema of Datasphere only supports Basic Authentication. Therefore, in this blog post we only elaborate how the connection is done over basic authentication. SAP HANA standalone supports however non-basic authentication, which are also supported in the HANA ML package to connect certificate based to the SAP HANA instance.

To establish the connection to the HANA database, we make use of the implemented HANA ML dataframe class and call the method ConnectionContext. We store the instance of the connection in the variable conn. To now be able to establish the connection to the HANA database view or table, we will need to specify over the method table the connection. The beautiful aspect is that overall, the dataset is not going to be loaded to the Python runtime, but will only be represented with a proxy to the actual table in the HANA database. All transformations, if done over the methods of HANA ML, are then pushed down to the database itself and executed there if the training is executed. In our case, we load the sample dataset into our database by making use of the provided methods of the HANA ML package.

"""
This script provides a short example how HANA ML and MLFlow can be integrated together.
The credentials to the database are following the currently supported authentification (Basic) of the 
SAP Datasphere OpenSQL schema. Overall, HANA Cloud standalone is also able to support multiple other authentification
methods. We have used an abstraction python file (constants) where we retrieve the securely stored 
authentification properties. To get more details about the exact method structure needed, please
have a look at the documentation:
https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.07/en-US/hana_ml.dataframe.html#hana_ml.dataframe.ConnectionContext
"""
from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.algorithms.pal.auto_ml import AutomaticClassification
import mlflow
from hana_ml.algorithms.pal.auto_ml import Preprocessing
from hana_ml.algorithms.pal.partition import train_test_val_split
from constants import db_url, db_user, db_password
# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url, port=443, user=db_user, password=db_password)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

After connecting to the database, the user is able to use the preprocessing methods implemented in HANA ML. Generally, the different changes are pushed down to the HANA database and are not executed within the Python runtime. In our use case, we do not need to use the data preprocessing as we directly retrieve a sample dataset which we can directly use for our ML training.

After finishing the potentially needed transformations, we are now able to implement the tracking of our HANA ML runs with the possibility of MLFlow. Similar to the normal usage of MLFlow, we set up first our tracking uri under which we want to store our HANA ML runs and models. In your case you would need to change the keyword mlflow_tracking_uri with your respective MLflow tracking URL. Furthermore, we then are able to specify the experiment name under which the runs are tracked. If we do not specify a specific experiment, the runs are tracked under the Default experiment.

# set up MLFlow
mlflow.set_tracking_uri(mlflow_tracking_uri)
mlflow.set_experiment("HANA ML Experiment")

In the following chapters, we will provide an outline how the exact training is performed and what components are logged to MLflow.

Unified Method

For the example we use the implemented Hybrid Gradient Boosting Tree as a classification algorithm for our Classification. In order to perform the classification, we use the Unified Classification in order to be able to run our algorithm. On the defined variable, we then use the implemented enable_mlflow_autologging method. This allows us to directly log the model using implemented auto logging behaviour.

uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging() uc.fit(training_data, key="ID", label="responded")

We call the fit method once we have initiated the HANA ML model variable and the associated autologging for MLflow. For the fit method, we have in total two different options. Firstly, the non-partitioned training dataset where we only use the training dataset. If we decide to partition our training dataset, we allow to create a validation dataset for which we can log metrics automatically during training.

If we do not define for our fit function the partitioning, we will not log metrics within MLFlow. In the following image, you can see how a potential HANA ML tracked run looks like in MLFlow together with the stored HANA ML model in MLflow.

If we decide to partition our dataset, here for instance to partition the dataset along the defined primary key, we are able to directly log evaluation metrics relevant for the Classification we have used. This includes the following metrics: AUC, Recall, Precision, F1 Score, Accuracy, Kappa coefficient and the Mathews Correlation Coefficient (MCC). This would directly allow us to compare multiple runs within our MLFlow project to one another and measure the different performances.

In addition to the general run, HANA ML also logs the model to MLflow. What is logged to MLflow depends on the parameters set for the method enable_mlflow_autologging. If for instance everything is set to the default settings, we will see the following yaml file to be logged to MLflow.

If within the method enable_mlflow_autologging the parameter is set to is_exported, the model binaries stored in the model storage on HANA are exported to MLflow. This setting would allow us to retrieve the trained model from MLflow and use it in a different HANA database for prediction purposes. In addition to the yaml file containing the metadata we now can see a created subfolder called models which contains the necessary model artefacts normally stored in the HANA database now in MLflow.

After the training is finished, we have besides the auto logging capabilities of HANA ML for MLflow the possibility to track further artefacts in MLflow. In the following section we will outline a few possibilities that exist with the additional tracking.

Additional logging possibilities

Besides the outlined auto logging capabilities, we can track with MLFlow additional artefacts to the respective run. In the following chapters, we outline selected possibilities to further enrich the auto logging for HANA ML runs tracked in MLFlow.

Adding run and experiment description

The description in the experiment section can be handy once the number of your experiments grows in the repository. In addition, mlflow allows to also add individual description to each run of an experiment. Using the following methods you can set up both:

from mlflow.tracking import MlflowClient
current_experiment=dict(mlflow.get_experiment_by_name("HANA ML Experiment"))
experiment_id=current_experiment['experiment_id']
    
run = mlflow.active_run()
MlflowClient().set_experiment_tag(experiment_id,"mlflow.note.content",
"This experiment shows the automated methods of HANA machine learning and how to track them with MLFLOW")
MlflowClient().set_tag(run.info.run_id, "mlflow.note.content", "This is a run tracked with Unified Classification from HANA Machine Learning")

Logging input datasets

Sometimes it is important to keep the input dataset also as part of the tracking with MLflow. Since HANA machine learning datasets are located in HANA, they need to be converted to pandas DataFrames to be tracked as shown in the following code:

# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

This results in the change, that the respective state of the training data is logged to the current run. The logged dataset can be found in the associated MLflow run, where the schema of the dataset is provided together with some metadata information about the number of rows and number of elements. In addition, also the provided context is marked in the UI of MLflow.

Logging a model report

In addition to the logging of the dataset, it might also be important to add a model report to MLFlow. HANA ML generally provides different interactive visualisations for the trained model artefact, which can be stored as an HTML file. After the storing of the model report to your local repository, we can log the input of the model report to our current run. This allows us to interactively explore the model report automatically generated by HANA ML and make it accessible in MLFlow. To log the HANA ML model report, you can use the following code snippet.

# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")

After the Model report is stored successfully under the current run, we can see in the artefact tab in MLFlow the interactive model report:

The complete script used for this section can be found in the following code snippet:

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLflow
mlflow.set_tracking_uri(tracking_uri)
mlflow.set_experiment("HANA ML Experiment")
# set up classification
uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging(is_exported=True)
# train model
uc.fit(training_data, key="ID", label="LABEL", partition_method="stratified", stratified_column="ID", partition_random_state=43, build_report=True)
# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")
# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

Apply of trained model

After we have finished our training, we are able with HANA ML to retrieve the model from MLFLow and use it for our prediction purposes. For this purpose, we will create a separate Python script where we will provide an overview to retrieve the trained MLflow model.

Similar to our training script, we first set up our connection to the HANA database and establish the connection to our table. In our case, we simply use the sample dataset provided by HANA ML.

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

Similar to our training script, we need to set the tracking url for MLflow and need to initiate the model storage of HANA. If we have decided to not export the HANA ML model to MLflow, we need to specify the same schema for the model storage where our HANA ML model is stored after the successful run. In case we have exported our model, we are able to specify a different schema. In the following, you can see the necessary script in order to retrieve the logged HANA ML model from MLflow.

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_url)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)

After the model storage has been initiated, we are able to retrieve the stored HANA ML model from MLflow. In order to select the correct model, you need to extract the correct run id associated to the model you would like to apply for your prediction dataset. In our case, this is the test dataset we have received from the sample dataset method. The model_uri needed for the model retrieval is consisting of the following pattern 'runs:/{run id}/model', in which you would need to exchange the run id with your respective run. For the actual retrieval of the model, we use the initiated model storage, in our case called model_storage and call the method load_mlflow_model to load the MLflow model to our HANA database and assign the respective proxy to our variable mymodel. The variable mymodel is then used to call the predict method in order to apply our model to our dataset. In the end we transform our prediction dataset into a Pandas DataFrame to look at the content of the created DataFrame. Normally, we could directly persist the created temporary table with the save method and therefore make the dataset available for further processing.

# load logged run from MLflow to HANA ML
logged_model = 'runs:/d8a763b7b81940598633605e447cd880/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

After running the script, you should be able to see the following terminal output, for which we can see the download of the artefact stored in MLflow and the created prediction dataset, which consists in our case of 4 columns: ID (primary key), SCORE (predicted label), CONFIDENCE (prediction confidence for applied row) and REASON_CODE (influence of individual variables to prediction output).

In case we have exported our model, the output of our terminal look slightly different indicating that we also download the respective model artefacts stored additionally to the yaml file. In the following you see the complete script used for applying the model to a new dataset.

from hana_ml import dataframe
from hana_ml.model_storage import ModelStorage
from hana_ml.algorithms.pal.utility import DataSets
import mlflow
from constants import db_url, db_user, db_password

conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

# full_set, diabetes_train, diabetes_test, _ = DataSets.load_diabetes_data(conn)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_uri)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)
# load logged run from MLflow to HANA ML
logged_model = 'runs:/ed7b8d4734cb42ca90c417f932957b40/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

Key take aways

In this blog post we have showcased an end to end example how MLflow can be integrated in your HANA ML workload by providing the possibility to share and compare multiple tracked runs in MLflow. If the data is already stored in HANA, this allows you to directly interact with MLflow while being able to run your Machine Learning algorithms on data stored in the HANA database without the need to transfer your data between multiple systems. This blog covered an essential part of the automated logging capabilities of HANA ML models into MLflow.

We highly appreciate your thoughts, comments and questions under this blog post. In case you want to reach out for general questions around HANA, or specifically HANA ML, don't hesitate to use the Q&A tool with the respective tags that describe your question.

SAP CodeJam HANA ML In Poznań 2024-04 Recap

2024-05-09T16:09:45.370000+02:00

This week, we had the Getting Started with Machine Learning using SAP HANA and Python in Poznań, Poland, kindly hosted by All-for-One Poland and organized by @JulianG

We got participants not only from Poznań, but as well from Polkowice and Warsaw.

I would like to thank Gabriela, Danka, and Julian for the organization!

And, as always, it was great to catch up with local SAP community veterans @DominikTylczyn and @GregMalewski during a spontaneous SAP Stammtisch the night before 🍻

If you want to host the SAP CodeJam on this topic, then please check: https://community.sap.com/t5/technology-blogs-by-sap/quot-getting-started-with-machine-learning-using-sap-hana-quot-as-a-new-sap/ba-p/13574098

Automated check for SAP HANA Cloud availability with SAP Automation Pilot

2024-05-09T19:21:58.623000+02:00

Introduction

SAP HANA Cloud serves as a component in various applications and systems and therefore it is critical for the business operations of numerous organizations. That’s why keeping tabs on the availability of your HANA Cloud instances is crucial for your success. Yet, manually monitoring them, especially when you have multiple instances, can be a tedious and time-consuming task. Wouldn't it be more optimal to have an automated helper keeping a watchful eye for you, alerting you if an instance stops, or better still, starting your HANA instances when they become unavailable?

Welcome to SAP Automation Pilot and SAP Alert Notification Service for SAP BTP – two powerful tools from the SAP Business Technology Platform (SAP BTP) DevOps portfolio. In this blog post, I’m going to walk you through a step-by-step guide on how you can use these innovative tools to transform the way you monitor your HANA Cloud instances, saving you time, preventing potential disruptions, and ensuring your operations continue to run smoothly.

What is SAP Automation Pilot?

Among the vast array of available tools, SAP Automation Pilot has emerged as a solid and competitive cloud application within the SAP Business Technology Platform (BTP) DevOps portfolio, revolutionizing the way DevOps team streamline their ops processes. In a nutshell, SAP Automation Pilot offers low-code / no-code automation capabilities that allow these teams to automate sequences of ops steps, use catalogs of automated commands provided by SAP or build custom executions (if needed), and run scripts in a serverless manner. More details about the product can be found at:

What is SAP Alert Notification service for SAP BTP?

SAP Alert Notification Service for SAP BTP facilitates your DevOps journey in SAP BTP by providing an API that allows providers to publish and consumers to receive events, and if needed, trigger predefined actions (including sending instant notifications over email, Microsoft Teams, etc.). Working seamlessly across cloud-native and hybrid solutions, it sends real-time notifications on events that are crucial for business and operational efficiency. It not only collects vital technical information from SAP BTP services but also possesses the ability to handle information from custom scenarios within your own environment. In essence, it offers a tailored alerting solution adapted to your needs. More details about the product can be found at:

How does the HANA Cloud Availability Check work?

SAP Alert Notification service for SAP BTP provides an array of built-in events that are ingested from HANA Cloud. (For more information, see SAP HANA Cloud Service Database Events.)

However, these events rely on the availability of the HANA Cloud instance. If this instance suddenly becomes unreachable, it won’t be able to trigger an event and you won’t receive a notification about the problem. For this reason, this use case requires a different approach:

SAP HANA Cloud provides a list of metrics, which are exposed through the metrics service REST API and can be used for monitoring and troubleshooting purposes. This metrics service REST API is a generic central service provided by SAP HANA Cloud which can be used by customers to retrieve metrics from each individual SAP HANA Cloud instance through an external secure API from any HTTPS endpoint. Among the many available metrics within this API, the one that SAP Automation Pilot uses to evaluate the availability of your HANA Cloud instance in this scenario is "HDBAccessible". This metric is gathered at 30-second intervals.

SAP Automation Pilot provides you with a sample command, which calls the HANA Cloud Metrics REST API and gets the data from the “HDBAccessible” metric. If the last 3 pings fail, this means that the HANA Instance is not accessible, and SAP Automation Pilot fires an event to SAP Alert Notification service. If the needed subscription is made in Alert Notification service, the service will send a notification (via email or other delivery channel) to the respective team alerting them that their HANA Cloud instance is unavailable. In the meantime, SAP Automation Pilot will trigger a remediation command, which will start the HANA Cloud instance. The process is visualized in the graphic below:

Step-by-Step Guide

Prerequisites:

SAP HANA Cloud database provisioned on Cloud Foundry Space level;
SAP BTP Technical User (See Managing technical users for BTP platform access) with HANA Cloud Administrator role;
SAP Automation Pilot (See Initial Setup);
SAP Alert Notification service for SAP BTP enabled in Cloud Foundry (see Initial Setup);
Integration between SAP Automation Pilot and SAP Alert Notification service for SAP BTP (See Integrate with SAP Alert Notification Service for SAP BTP);

After you’ve made sure that you meet all the prerequisites, you can proceed with the tutorial.

Step-By-Step Guide:

1. Verify that you have provisioned SAP Automation Pilot. You should be able to see it in the “Subscriptions” section in your subaccount:

NOTE: SAP Automation Pilot can interact with resources in any BTP landscapes. All you need is to have at least one instance of SAP Automation Pilot.

2. Verify that you have an Alert Notification service instance. You should be able to see it in “Instances” in your space:

3. You need to create a Service Key for your Alert Notification service instance. This key will be used on a later stage by the SAP Automation Pilot command. To create a Service Key, choose the “Create Service Key” option:

4. Enter a name for your service key and choose “Create”:

5. Repeat the previous steps (3, 4) to create a Service Key for your HANA Cloud instance. It will be used on a later stage by the SAP Automation Pilot command.

6. SAP Automation Pilot team has prepared a catalog with various example commands, which can be imported directly into your SAP Automation Pilot instance. Here's how you do it - just copy the sample content catalog available at this link here:

7. Now you need to import the catalog with example commands in your SAP Automation Pilot instance. From the left-side menu, choose “My Catalogs” and then “Import”:

Paste the catalog and choose “Import”:

8. You can see the imported catalog “Automation Pilot Examples” in “My Catalogs”. From this catalog, choose “Commands”:

9. You can see the CheckHanaCloudAvailabilitySample command. You can open the command by clicking on it:

10. When you open the command, you can explore its components – the Input Keys, the Output Keys, and the Executors:

11. Scroll down to “Configuration” and choose the “pingHanaCloud” executor:

12. Go to the “Parameters” section and choose “Edit”:

You need to provide the URL that Automation Pilot will use to ping your HANA Cloud Instance. Edit the URL with the correct values for your cluster (production or pre-production environment) and data center.

It should be in the following format: https://api.gateway.orchestration.<cluster>-<datacenter>.hanacloud.ondemand.com/metrics/v1/serviceInstances/$(.execution.input.hanaServiceInstance)/values?names=HDBAccessible . (For more information, see Access to the Host.)

Please feel free to update the API Gateway URL accordingly based on the correct values for <cluster>-<datacenter> in your setup it and click on the “Update” button (see an example underneath):

13. Now you need to define the event that SAP Automation Pilot will ingest into Alert Notification service. To do that, click on the “notifyANS” executor. Then choose “Edit” to edit the properties of your event:

14. You can use these example values for the required event properties of your event:

body: The availability check for HANA Cloud with Service Instance ID $(.execution.input.hanaServiceInstance) has failed. Now the Automation Pilot proceeds with an automated START for this HANA Cloud instance.
category: ALERT
eventType: FailedHCAvailabilityCheck
resourceName: hana
resourceType: database
serviceKey: $(.execution.input.AnsServiceKey) -> use a dynamic expression to get the AnsServiceKey from the inputs
severity: ERROR
subject: Failed HANA Cloud Availability Check $(nowMillis | toDate("yyyy-MM-dd"))

15. After you have done that, trigger the command:

16. You need to provide values for the following input keys:

AnsServiceKey - Service Key for your Alert Notification service instance.
BtpTechnicalUserPassword - Password for the SAP BTP technical user you have created and currently maintain active.
BtpTechnicalUserUsername - Username for the SAP BTP technical user you have created and currently maintain active.
hanaClientID - HANA Client ID - available in your HANA Cloud service key file, look for "clientid".
hanaClientSecret - HANA Client Secret ID - available in your HANA Cloud service key file, look for "clientsecret".
hanaClientTokenURL - The URL needed to be called in order to generate a token to access your HANA Cloud instance. The URL is built as follows: in the HANA Cloud service key file, in the "uaa" section, look for the specified "url" + add to it /oauth/token?grant_type=client_credentials.
hanaCloudRegion - The region where your SAP BTP sub-account is located at, e.g. "cf-eu10".
hanaCloudResourceGroup - The name of the CF Space where your HANA Cloud is located at, e.g. "dev", "prod", etc.
hanaCloudResourceName - The exact name of your HANA Cloud instance, e.g. "hanaDemo", "hanaProd", etc.
hanaCloudSubAccount - The exact "Org Name" of your SAP BTP subaccount where your HANA Cloud instance is located.
hanaServiceInstance - The Instance ID of your HANA Cloud instance - it can be found from the provisioned service in BTP Cockpit, look for "Instance ID".
You can also look at the prepared outputs (no need to provide values for them but it is important to understand what data is returned as the command's output):
- checkPing - The entire response for the metric "HDBAccessible" returned by HANA Cloud Metrics Service REST API.
- isHanaCloudAvailable - Check on whether the HANA Cloud instance is available, where: "true" = HANA Cloud is available; "false" = HANA Cloud is not available.
- resultHanaStart - Check on whether the HANA Cloud instance has been started successfully, where: "true" = HANA Cloud has been started; "false" = HANA Cloud was not started'.

17. Provide values for all input keys and then trigger the command:

There are two possible options for the outcome of the execution:

Option 1: Your HANA Cloud instance is accessible and the command for starting HANA Cloud has not been triggered. You should be able to see the following execution details:

If you click on “Output Values”, you can see the output values for checkPing, isHanaCloudAvailable, and resultHanaStart:

You can verify in SAP Hana Cloud Central that your HANA Cloud instance is up and running:

Option 2: Your HANA Cloud instance is not accessible and the command for starting the HANA Cloud has been triggered:

You can verify in SAP Hana Cloud Central that your HANA Cloud instance is starting:

If you click on “Output Values”, you can see the output values for checkPing, isHanaCloudAvailable, and resultHanaStart:

You have successfully performed the HANA Cloud Availability Check via SAP Automation Pilot. Now you can make a subscription in Alert Notification service, so that you can receive a notification when SAP Automation Pilot detects that your HANA Cloud instance is not accessible. If you want to do that, continue with the following steps:

18. Navigate to your Alert Notification service instance and open it. Choose the “Subscriptions” tab form the left-side menu and then “Create” to create a new subscription:

19. Enter a name for your subscription and click “Create” to move on to the next step:

20. Next, create a Condition. Give your condition a name. Then define your condition by choosing the event property “eventType”, operand “Is Equal To”, and, in our case, expected value “FailedHCAvailabilityCheck”:

The expected value for “eventType” is the one you provided in step 14:

21. To finish your subscription in Alert Notification service, select an Action by choosing “Create Action”:

Select the Action Type of your preference, for example “Email”, and choose “Next”:

Give your Action a name, and then provide the email address where you want to receive the email notification:

NOTE: After you have created some of the actions, they are labeled as Not Confirmed. To enable them, you have to confirm them as described in Confirming Actions.

22. Choose “Save” to finish making your subscription. You will see a confirmation that your subscription was created:

You can find more information about Subscriptions, Conditions, and Actions in Configuration Management Using the SAP BTP Cockpit.

23. Now you can test your subscription. If you stop your HANA Cloud Instance and trigger the CheckHanaCloudAvailabilitySample command again, you will receive an email from Alert Notification service with information that the HANA Cloud Availability Check has failed. You can see an example email below:

24. Alternative: Send notification to a Microsoft Teams channel

To send a notification to a Microsoft Teams channel of your choice, first you need to add an Incoming Webhook to this channel. To do this, follow these steps:

1) In your channel, select “Manage channel” from the “More options” menu:

2) Choose “Edit”:

3) Add the Incoming Webhook:

4) Provide a name for the incoming webhook and click “Create”:

5) You will be provided with a unique webhook URL. Copy it, as you will need to use it when creating an Action in Alert Notification service. Click “Done” and the webhook will be added to your MS Teams channel.

6) When creating an Action in Alert Notification service, choose the Microsoft Teams Action Type:

7) Give your Action a name and paste the webhook URL in the URL field. Then choose “Create”:

With this configuration, if the HАНА Cloud Availability Check fails, you will receive a notification in the MS Teams channel of your choice. Here’s an example MS Teams notification:

And there we have it! You’ve successfully automated the monitoring of your HANA Cloud instances!

Wrap-up

In this blog, you've learned to utilize SAP Automation Pilot and SAP Alert Notification service for SAP BTP to effectively monitor the availability of your HANA Cloud instances, and even start the ones that are not accessible. Combining these cutting-edge tools gives you unprecedented control over your operations. So, march ahead, keep disruptions at bay, and embrace a smooth and efficient journey with your HANA Cloud instances with the help of SAP Automation Pilot and SAP Alert Notification service for SAP BTP!

SAP Community - SAP HANA Cloud

Webinar Replay: Better Business Outcomes with the SAP HANA Cloud Vector Engine

Connection restrictions and their relation to user groups in SAP HANA Cloud, SAP HANA Database

Nested JSON to SAP HANA Tables with SAP Integration Suite

Value Unlocked! How customers benefit from SAP HANA Cloud

🏇 Building a powerful, reliable, high-performance data solution with Black Horse One

✈️ Developing a strong and trusted data application for price management, purchasing and supply chain of spare parts for Lufthansa Technik

🛒 Efficient data access for apps and services in Żabka Polska's hybrid landscape

🖨 Lexmark maximizes technology investments and improves cost-of-ownership with a seamless migration to the cloud

Consuming CAPM Application's OData service into SAP Fiori Application in Business Application Studio

Top Picks: Innovations Highlights from SAP Business Technology Platform (Q1/2024)

SAP BTP Innovation Highlights Q1 2024

Application Development and Automation:

Integration:

Data and Analytics:

Artificial Intelligence:

Cross SAP BTP:

SAP BTP Innovation Highlights - outlook:

Check out more on innovation highlights:

New Machine Learning features in SAP HANA Cloud 2024 Q1

Classification and Regression enhancements

Auto-ML and ML pipeline function improvements

Text Processing

New financial data analysis functions

Python ML client (Hana-ML) enhancements

SAP Offers Maximum Attention for Customers’ High Standards

New Machine Learning features in SAP HANA Cloud

SAP Partners unleash Business AI potential at global Hack2Build

Hack2Build on Business AI – Highlighted Use Cases

Improving Time Management in SAP S/4HANA Cloud: A GenAI Solution

SAP Datasphere's updated Pricing & Packaging: Lower Costs & More Flexibility

HDI Artifact Recovery Wizard

Potential use cases are:

Prerequisites:

How to use Artifact Recovery?

The result:

Summary:

Explore the SAP HANA Cloud vector engine with a free learning experience

ACH/ WIRE RETURN Process in S/4 Hana

Embedding Business Context with the SAP HANA Cloud, Vector Engine

Replay: Embedding Business Context with the SAP HANA Cloud Vector Engine

The basics

Prompt engineering

Model fine-tuning

Retrieval Augmented Generation

SAP HANA Cloud, Vector Engine

Use Case 1: Deduplication of citizen reported issues

Use Case 2: Ask the City

Consumption Methods

Replay: Embedding Business Context with the SAP HANA Cloud Vector Engine

Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive

Introduction

Prerequisites

Set up MLFlow on BTP

Set up tracking for MLflow

Run HANA ML Algorithms with MLflow

Model training of HANA ML with MLFlow

Connect to HANA database (Deployed under SAP Datasphere)

Unified Method

Additional logging possibilities

Adding run and experiment description

Logging input datasets

Logging a model report

Apply of trained model

Key take aways

SAP CodeJam HANA ML In Poznań 2024-04 Recap

Automated check for SAP HANA Cloud availability with SAP Automation Pilot

Introduction

What is SAP Automation Pilot?

What is SAP Alert Notification service for SAP BTP?

How does the HANA Cloud Availability Check work?

Step-by-Step Guide

Prerequisites:

Step-By-Step Guide:

Further reading

Wrap-up