SAP Community - SAP HANA

Advance shipment via Procurement

2024-02-19T07:54:29.338000+01:00

In continuous of blog 1 of Advance Shipment functionality in Service Order

Scenario 2: Service Parts Are Not in Stock in a Warehouse

In this scenario, spare is not available in the company stock. It will be procured from a vendor and then delivered to customer location. In order to procure from vendor, we will create a purchasing requisition followed by purchased order from a service order item.

And a sales order needs to be created to perform the outbound delivery on the basis. This setting is controlled and maintained at the configuration level for Item category.

Since this stock we have procured for specific service order hence at the time of goods receipt, parts are recorded as sales order stock (E) so that no one can use this stock.

Once the goods are procured then rest of the logistic process of delivering a spare part and consumption of the stock post remain identical as Scenario 1 (Service Parts Are Already in Stock in a Warehouse)

After the service technician confirms the use of the service part with a service confirmation, the final goods issue from the customer consignment stock (-) to the customer is posted including the material consumption.

Process flow of End-to-end process –

Conclusion

Thanks for reading this blog post. Hopefully the blog post was informative. Please do let us know in case you have any question or query about this process in our Q&A community : here

References:

Best practices for SAP S/4HANA: here
SAP S/4HANA Cloud User Community: Follow it

SAP Landscape Transformation Replication Server - Book Review

2024-02-22T08:52:50.707000+01:00

In this blog post you will find the Book Review of SAP Landscape Transformation Replication Server (SLT)

About the Book
Here is the book which is published by SAP Press about SAP SLT.

About the publisher
SAP Press is reputable due to its status as the leading worldwide publisher of SAP software manuals, known for producing high-quality, authoritative, and cutting-edge content in the field of SAP. The SAP Community boasts a vast membership, comprising millions of users and consultants who are well-acquainted with the great reputation of SAP Press

The authors of the book are

Shashidhar Garimella is a principal architect at SAP America Inc., known for his exceptional skills and expertise in the field of enterprise software solutions.

Prathyusha Garimella is a senior consultant at SAP America Inc. With over a decade of IT experience spanning the globe, Prathyusha has successfully deployed various SAP solutions for large corporations, mostly featuring in the Fortune 100 companies worldwide, enabling financial and business transformation capabilities.

What's the latest and greatest book about SAP SLT?

The book "SAP Landscape Transformation Replication Server: The Practical Guide," and upon reading i am convinced that it will serve as an invaluable asset for SAP consultants, especially those with a focus on SAP Analytics and Data Management. This comprehensive guide caters to a wide audience, from novice SAP Consultants to seasoned enterprise architects, offering a well-structured explanation of the core concepts of SAP SLT, complete with architecture diagrams and detailed information, making it highly beneficial for beginners. Advanced SAP architects and consultants will find significant value in the book's comprehensive coverage of overarching topics. Furthermore, the book serves as a practical guidebook for typical SAP consultants and developers, providing detailed steps, screenshots, and elaborate explanations that can be directly applied to project implementations.

What sets this book apart?
What sets this book apart is its emphasis on pragmatic customer implementations for advanced consultants, a perspective that is often lacking in many blogs, articles, and even official SAP documentation. The meticulous attention to practicality makes this book an indispensable resource for consultants working in the field of SAP Analytics and data management. Notably, the book's coverage of topics such as Troubleshooting, Monitoring, and Performance Tuning fills a crucial knowledge gap, particularly for support consultants engaged in live projects. Overall, "SAP Landscape Transformation Replication Server: The Practical Guide" stands out as a meticulously crafted resource that caters to the practical needs of consultants working in the dynamic realm of SAP Analytics and Data Management. This book also covers niche topics such as SLT integration with SAP BW and BW/4HANA, Data Intelligence, Data Services.

If you are working on migrating your SAP system to the cloud(Azure, GCP, AWS,etc), transferring data between SAP systems, or initiating a new SAP project that need SLT, this book is an indispensable resource that should be at the top of your reading list. Kudos to the Shashidhar Garimella and Prathyusha Garimella for their exceptional work in crafting a book that truly encapsulates the expertise derived from decades of experience in the field. Their wealth of knowledge and practical insights are evident throughout the book.

Where can you get the book?

From the SAP PRESS website and bookstore (all formats; hardcovers also sold elsewhere)

Thanks
Renjith Kumar

Global Explanation Capabilities in SAP HANA Machine Learning

2024-02-28T00:53:14.190000+01:00

Machine learning (ML) has great potential for improving products and services across various industries. However, the explainability of ML models is crucial for their widespread adoption. First, explanation helps build trust and transparency between the users and the models. When users understand how ML model works, they are more likely to trust its results. Moreover, explainability allows for better debugging of complex models. By providing explanations for models’ decisions, researchers can gain insights into the underlying patterns, which helps identify potential biases or flaws. Furthermore, the explainability of models enables auditing, a prerequisite for its usage in regulated industries, such as finance and healthcare.

To benefit from an explainable model, we introduced permutation feature importance as a global explanation method to SAP HANA Predictive Analysis Library (PAL) in the past several months. In this blog post, we will show how to use it in Python machine learning client for SAP HANA (hana-ml), which provides a friendly Python API for many algorithms from PAL.

After reading this blog post, you will learn:

Permutation feature importance from its theory to usage
Two alternative global explanation methods available and their comparison to permutation feature importance

Permutation feature importance

Permutation feature importance is a feature evaluation method that measures the decrease in the model score when we randomly shuffle the feature's values. It reveals the extent to which a specific feature contributes to the overall predictive power of the model by breaking the association between the feature and the true outcome.

Behind the screen the permutation importance is calculated in the following steps:

Initially, a reference score is evaluated on the original dataset.
Next, a new dataset is generated by permuting the column of a specific feature, and the score is evaluated again.
Then the permutation importance is defined as the difference between the reference score and the score obtained from permuted dataset.

By repeating the second and third steps for each feature, we can get the importance scores for the entire dataset.

Permutation importance provides highly compressed, global insight to gauge the relative importance of each feature, enabling data scientists and analysts to prioritize their efforts on the most influential variables when building and optimizing models. This approach is particularly useful for handling high-dimensional datasets, as it helps identify the most informative features amidst a vast number of possible predictors.

Here we use the well-known Titanic dataset to illustrate the usage of permutation importance. In hana-ml, there is a class called DataSets that offers various public datasets. To load the dataset, we can utilize the load_titanic_data method.

from hana_ml import dataframe
from hana_ml.algorithms.pal.utility import DataSets

conn = dataframe.ConnectionContext(url, port, user, pwd)
titanic_full, _, _, _ = DataSets.load_titanic_data(conn)

Titanic dataset describes the survival status of individual passengers on the RMS Titanic. The objective is to predict based on passenger data (i.e. name, age, gender, socio-economic class, etc.) whether a passenger can survive the shipwreck. In our dataset we have 12 columns, and the meaning of each column is below:

PassengerId - Unique ID assigned to each passenger.
Pclass - Class of ticket purchased (1 = 1st class, 2 = 2nd class, 3 = 3rd class).
Name - Full name and title of the passenger.
**bleep** - Gender of the passenger.
Age - The Age of the passenger in years.
SibSp - Number of siblings and spouses associated with the passenger aboard.
Parch - Number of parents and children associated with the passenger aboard.
Ticket - Ticket number.
Fare - The fare of the ticket purchased by the passenger.
Cabin - The Cabin number that the passenger was assigned to. If NaN, this means they had no cabin and perhaps had not assigned one due to the cost of their ticket.
Embarked - Port of embarkation (S = Southampton, C = Cherbourg, Q = Queenstown).
Survived - Survival flag of passenger (0 = No, 1 = Yes), target variable.

To keep things simple and stay on track with our example, we will remove columns with a high number of null values and then build a predictive model to forecast survival status using the remaining features. We rely on PAL's built-in support in classification algorithm for handling other data preprocessing issues like missing values and dataset splitting.

from hana_ml.algorithms.pal.unified_classification import UnifiedClassification

rdt_params = dict(n_estimators=100,
                  max_depth=56,
                  min_samples_leaf=1,
                  split_threshold=1e-5,
                  random_state=1,
                  sample_fraction=1.0)
uc_rdt = UnifiedClassification(func = 'RandomDecisionTree', **rdt_params)

features = ["PCLASS", "NAME", "**bleep**", "AGE", "SIBSP", "PARCH", "FARE", "EMBARKED"]
uc_rdt.fit(data=titanic_full, key='PASSENGER_ID', features=features, label='SURVIVED',
           partition_method='stratified', stratified_column='SURVIVED', partition_random_state=1,
           training_percent=0.7, output_partition_result=True,
           ntiles=2, categorical_variable=['PCLASS','SURVIVED'], build_report=False,
           permutation_importance=True, permutation_evaluation_metric='accuracy',
           permutation_n_repeats=10, permutation_seed=1, permutation_n_samples=None)

RandomDecisionTree has a practical method for estimating missing data. When it comes to training data, the method calculates the median of all values for numerical variable or the most frequent non-missing value for categorical variable in a certain class, and then uses that value to replace all missing values of that variable within that class. As for test data, the class label is absent, so one missing value is replicated for each class and filled with the corresponding class’s median or most frequent item.

UnifiedClassification has a method for dataset splitting, so we can use it to randomly split our dataset, using 70% for training and leaving the rest for validating. In addition, RandomDecisionTree has built-in support for categorical variables; all we need to do is specify the parameter categorical_variable for variables that come in integer type.

To enable the calculation of permutation feature importance, set permutation_importance to True. Additionally, use permutation_evaluation_metric to define the evaluation metric for importance calculation. For classification problems, options include accuracy, auc, kappa and mcc, while for regression problems, options are RMSE, MAE and MAPE. Set permutation_n_repeats to specify the number of times a feature is randomly shuffled. Because shuffling the feature introduces randomness, the results might vary greatly when the permutation is repeated. Averaging the importance measures over repetitions stabilizes the measure at the expense of increased computation time. Use permutation_seed to set the seed for randomly permuting a feature column, which ensures reproducible results across function calls. Moreover, set permutation_n_samples to determine the number of samples to draw in each repeat. While this option may result in less accurate importance estimates, it helps manage computational speed when evaluating feature importance on large datasets. By combining permutation_n_samples with permutation_n_repeats, we can control the trade-off between computational speed and statistical accuracy of this method.

Permutation importance does not indicate the inherent predictive value of a feature but how important this feature is for a specific model. It is possible that features considered less important for a poorly performing model (with a low cross-validation score) could actually be highly significant for a well-performing model. Therefore it is crucial to assess the predictive power of a model using a held-out set prior to determining importances.

uc_rdt.statistics_.collect()

STAT_NAME	STAT_VALUE	CLASS_NAME
AUC	0.7385321100917431	None
RECALL	0.9674418604651163	0
PRECISION	0.7247386759581882	0
F1_SCORE	0.8286852589641435	0
SUPPORT	215	0
RECALL	0.29464285714285715	1
PRECISION	0.825	1
F1_SCORE	0.43421052631578944	1
SUPPORT	112	1
ACCURACY	0.7370030581039755	None
KAPPA	0.3097879442371883	None
MCC	0.37957621849462986	None

We can check the model performance on validation set directly from fitted attribute statistics_. Its validation performance, measured via the accuracy score, is significantly larger than the chance level. This makes it possible to use permutation importance to probe the most predictive features.

import matplotlib.pyplot as plt

df_imp = uc_rdt.importance_.filter('IMPORTANCE >= 0').collect()
df_imp = df_imp.sort_values(by=['IMPORTANCE'], ascending=True)

c_title = "Permutation Importance"
df_imp.plot(kind='barh', x='VARIABLE_NAME', y='IMPORTANCE', title=c_title, legend=False, fontsize=12)
plt.show()

Feature importances are provided by the fitted attribute importances_. We can visually represent the feature contributions using a bar chart.

While there is some element of luck involved in surviving, it seems some groups of people were more likely to survive than others. The most important features for predicting survival status with a random forest are **bleep**, Pclass and fare, whereas passenger’s family relations or name are deemed less important.

This is reasonable because women were given priority access to the lifeboats, so they were more likely to survive. Also, both Pclass and fare can be regarded as a proxy for socio-economic status (SES). People with higher SES may have had better access to information, resources, and connections to secure a spot on a lifeboat or be rescued more quickly. They may also possess more experience with navigating emergency situations and better access to survival skills and knowledge.

Compared to gender and SES, factors such as port of embarkation, family relations, or name played a limited role in survival. Because the chaotic and rapidly evolving nature of the sinking meant that all passengers were subject to the same evacuation protocols, these factors were less relevant in determining a passenger's likelihood of survival.

Apart from permutation feature importance, there are two additional techniques existing in PAL can be used to gain a global explanation. One is impurity-based feature importance computed on tree-based models and another is SHAP feature importance obtained by aggregating local Shapley values for individual predictions. We will delve into these methods individually through the subsequent two case studies.

Case study: impurity-based feature importance

Tree-based models provide an alternative measure of feature importance deriving from nodes’ splitting process.

Individual decision trees intrinsically perform feature selection by selecting appropriate split points. This information can be used to measure the importance of each feature; the basic idea is if a feature is frequently used in split points, it is considered more important. In practice, importance is calculated for a single decision tree by evaluating how much each attribute split point improves performance, weighted by the number of observations under each node. The performance measure may be the purity used to select the split points or another more specific error function.

This notion of importance can be extended to decision tree ensembles by simply averaging the impurity-based feature importance of each tree. By averaging the estimates over several randomized trees, the variance of the estimate is reduced, making it suitable for feature selection. This is known as the mean decrease in impurity, or MDI.

Note that this computation of feature importance is based on the splitting criterion of the decision trees (such as Gini index), and it is distinct from permutation importance which is based on permutation of the features.

We show the calculation of impurity-based importance on Titanic dataset. The calculation is incorporated in the fitting of RandomDecisionTree, as demonstrated in the code below.

from hana_ml.algorithms.pal.unified_classification import UnifiedClassification

rdt_params = dict(n_estimators=100,
                  max_depth=56,
                  min_samples_leaf=1,
                  split_threshold=1e-5,
                  random_state=1,
                  sample_fraction=1.0)
uc_rdt = UnifiedClassification(func = 'RandomDecisionTree', **rdt_params)

features = ["PCLASS", "NAME", "**bleep**", "AGE", "SIBSP", "PARCH", "FARE", "EMBARKED"]
uc_rdt.fit(data=titanic_full, key='PASSENGER_ID', features=features, label='SURVIVED',
           partition_method='stratified', stratified_column='SURVIVED', partition_random_state=1,
           training_percent=0.7, output_partition_result=True,
           ntiles=2, categorical_variable=['PCLASS','SURVIVED'], build_report=False
          )

uc_rdt.statistics_.collect()

Prior to inspecting feature importance, it is important to ensure that the model predictive performance is high enough. Indeed, there is no point in analyzing the important features of a non-predictive model. Here we can observe that the validation accuracy is high, indicating that the model can generalize well thanks to the built-in bagging of random forests.

The feature importance scores of a fitted model can be accessed via the importance_ property. This dataframe has rows representing each feature, with positive values that add up to 1.0. Higher values indicate a greater contribution of the feature to the prediction function.

import matplotlib.pyplot as plt

df_imp = uc_rdt.importance_.collect()
df_imp = df_imp.sort_values(by=['IMPORTANCE'], ascending=True)

c_title = "Impurity-based Importance"
df_imp.plot(kind='barh', x='VARIABLE_NAME', y='IMPORTANCE', title=c_title, legend=False, fontsize=12)
plt.show()

A bar chart is plotted to visualize the feature contributions.

Oops! The non-predictive passenger’s name is ranked most important by the impurity-based method which contradicts the permutation method. However, the conclusions regarding the importance of the other features still hold true. The same three features are detected most important by both methods, although their relative importance may vary. The remaining features are less predictive.

So, the only question is why impurity-based feature importance assigns high importance to variables that are not correlated with the target variable (survived).

This stems from two limitations of impurity-based feature importance. First, impurity-based importance can inflate the importance of high cardinality features, that is features with many unique values (such as passenger’s name). Furthermore, impurity-based importance suffers from being computed on training set statistics and it cannot be evaluated on a separate set, therefore it may not reflect a feature’s usefulness for make predictions that generalize to unseen data (if the model has the capacity to use the feature for overfit).

The fact that we use training set statistics explains why passenger’s name has a non-null importance. And the bias towards high cardinality features explains further why the importance has such a large value.

As shown in previous example, permutation feature importance does not suffer from the flaws of the impurity-based feature importance: it does not exhibit a bias toward high-cardinality features and can be computed on a left-out validation set (as we do in PAL). Using a held-out set makes it possible to identify the features that contribute the most to the generalization power of the inspected model. Features that are important on the training set but not on the held-out set might cause the model to overfit. Another key advantage of permutation feature importance is that it is model-agnostic, i.e. it can be used to analyze any model class, not just tree-based models.

However, the computation for full permutation importance is more costly. There are situations that impurity-based importance is preferable. For example, if all features are numeric and we are only interested in representing the information acquired from the training set, limitations of impurity-based importance don’t matter. If these conditions are not met, permutation importance is recommended instead.

Now that we have completed our exploration of impurity-based importance, let's shift our focus to SHAP feature importance.

Case study: SHAP feature importance

SHAP (SHapley Additive exPlanations) is a technique used to explain machine learning models. It has its foundations in coalitional game theory, specifically Shapley values. These values determine the contribution of each player in a coalition game. In the case of machine learning, the game is the prediction for a single instance, features act as players, and they collectively contribute to the model’s prediction outcome. SHAP assigns each feature a Shapley value and uses these values to explain the prediction made by the model.

The SHAP calculation can be invoked in the prediction method of UnifiedClassification. Once again, we show its application on Titanic dataset. The RandomDecisionTree model is trained as before. To ensure a more valid comparison to permutation importance, we deliberately employ SHAP on the validation set.

uc_rdt.partition_.set_index("PASSENGER_ID")
titanic_full.set_index("PASSENGER_ID")
df_full = uc_rdt.partition_.join(titanic_full)

features = ["PCLASS", "NAME", "**bleep**", "AGE", "SIBSP", "PARCH", "FARE", "EMBARKED"]
pred_res = uc_rdt.predict(data=df_full.filter('TYPE = 2'), key='PASSENGER_ID', features=features, verbose=False,
                          missing_replacement='feature_marginalized',
                          top_k_attributions=10, attribution_method='tree-shap')

pred_res.select("PASSENGER_ID", "SCORE", "REASON_CODE").head(5).collect()

SHAP by itself is a local explanation method explains the predictions for individual instances. Since we run SHAP for every instance, we get a matrix of Shapley values. This matrix has one row per data instance and one column per feature. To get a global explanation, we need a rule to combine these Shapley values.

In practice, there are different ways to aggregate local explanations. For instance, we can assess feature importance by analyzing how frequently a feature appears among the top K features in the explanation or by calculating the average ranking for each feature in the explanation. In our case, we opt to use mean absolute Shapley values as an indicator of importance.

The idea behind this is simple: Features with large absolute Shapley values are considered important. Since we want the global importance, we average the absolute Shapley values for each feature across the data. We can then arrange the features in descending order of importance and present them in a plot, like what we have done before. Another simpler solution is to utilize the ShapleyExplainer module as a visualizer and let it handle the task.

from hana_ml.visualizers.shap import ShapleyExplainer

features=["PCLASS", "NAME", "**bleep**", "AGE", "SIBSP", "PARCH", "FARE", "EMBARKED"]
shapley_explainer = ShapleyExplainer(feature_data=df_full.filter('TYPE = 2').select(features), 
                                     reason_code_data=pred_res.select('REASON_CODE'))
shapley_explainer.summary_plot()

There is a big difference between SHAP feature importance and permutation feature importance. Permutation feature importance is based on the decrease in model performance, while SHAP is based on magnitude of feature attributions. In other words, SHAP feature importance reflects how much the model’s prediction varies can be explained by a feature without considering its impact on performance. If changing a feature greatly changes the output, then it is considered important. As a result, SHAP importance gives higher importance to features that cause high variation in the prediction function.

Although model variance explained by the features and feature importance are strongly correlated when the model generalizes well (i.e. it does not overfit), this distinction becomes evident in cases where a model overfits. If a model overfits and includes irrelevant features (like the passenger’s name in this instance), the permutation feature importance would assign an importance of zero because this feature does not contribute to accurate predictions. SHAP importance measure, on the other hand, might assign high importance to the feature as the prediction can change significantly when the feature is altered.

Additionally, it is noteworthy that calculating SHAP can be computationally demanding, especially for models that are not based on trees. If you are only looking for a global explanation, it is suggested to use permutation importance.

Summary

Explainability is of vital importance in machine learning models. It builds trust, aids in understanding, enables debugging, and helps with regulatory compliance. Throughout this blog post, we have presented permutation feature importance as a model-agnostic global explanation method, exploring its theory and practical application on real accessible data. Furthermore, we have showcased its effectiveness by comparing it with two other explanation methods, empowered readers to choose the most suitable approach for their specific use case. Ultimately, incorporating explainability into machine learning models is not only beneficial for the users but also for the overall progress and ethical deployment of AI technologies.

Protecting Your SAP HANA Cloud and SAP HANA with Credential Rotation Best Practices

2024-02-29T18:29:16.597000+01:00

In today's cyber-threat landscape, prioritizing data security is more important than ever. Security is one of the key cornerstones of SAP BTP and SAP HANA Cloud strategies.

SAP HANA Cloud takes a comprehensive approach to security, providing a layered defense system encompassing authentication, authorization, encryption, and auditing.

At the heart of this security strategy lies strong credentials management practices, one of which is the credentials rotation. For example, organizations can significantly reduce the risk of unauthorized access and data breaches by enforcing regular password rotation.

Certificate-based credentials are the game-changer

While strong passwords and their rotation are a must, they're no longer enough. We recommend using certificate-based credentials, a robust security practice that adds an extra layer of protection and should be adopted by all SAP HANA users.

Certificate-based credentials offer a far more robust solution. Here's what you get:

Multi-factor Authentication (MFA): Imagine needing both a certificate and a secondary factor (e.g., PIN, fingerprint) to access your HANA environment. MFA is typically done at identity provider and mainly in combination with short-lived SAML/JWT tokens for the actual database access.
Rotation: Hackers adore static credentials, but with certificates, they expire regularly, rendering stolen ones useless, and you can no longer use them for login. It's like changing the locks on your house now and then, depending on how often you need it. The benefit of certificates is that you can have multiple certificates for an overlapping time range when changing them.
Reduced Attack Surface: Fewer passwords to manage mean fewer potential entry points for attackers. It's like having fewer windows to climb through.

In SAP HANA Cloud, there is a possibility of using X509 certificates, JWT tokens, and SAML assertions. On top of it, HANA provides an extensive certificate management workflow and guidance, which you can read more about here. For SAP HANA, please refer to the onprem version of security documentation.

In case there is still the need to work with password-based authentication, there are a couple of password rotation tricks on how you can enhance your security and which HANA supports.

How Password Rotation Works in SAP HANA Cloud

Password rotation, also known as password aging, is a security protocol that mandates the periodic change of passwords for user accounts. This practice helps to mitigate the risk of unauthorized access if a password is compromised.

In SAP HANA Cloud, password rotation is controlled by password policies that can be customized to meet specific organizational requirements. These policies can specify a wide range of configuration options like the minimum password length, the required character types, the minimum/maximum password lifetime, and the frequency with which passwords must be updated.

The SAP Cloud HANA database comes with a default password policy. You can change the default password policy of the database in line with your organization’s security requirements.

More information about it you can find in ours SAP HANA Cloud Database Security Guide in a section Password policy.

Disclaimer: from the HANA side there is no difference for password management between on-premise HANA and HANA Cloud.

Scenario 1: "Zero down time password change" for database users

In case your organisation needs to sustain changing passwords for HANA database users without experiencing any interruption in service or accessibility, or in other words, the HANA Cloud DB should remain fully functional and accessible while changing/updating passwords for those users – it is supported by the product.

On the other hand, in case under “zero downtime password change” is also meant changing the password of a user in HANA without having to change the password used to connect to HANA at the same time in their application to avoid failed connect attempts (something like this can be achieved by allowing connect for one user with multiple passwords) – SAP HANA Cloud DB only supports a single password per user at a time.

The proper way to avoid this issue is to use other means of authentication, typically X.509. In that case you would only configure trust to the X.509 issuer in HANA and set a new X.509 certificate in the application before the current one expires. The same can be achieved if the application supports our SAML/JWT authentication.

As a short summary, this is not a limitation on the SAP HANA Cloud DB side but on the application side.

Scenario 2: How to manage technical users with regards to password rotation and their authentication in general?

For technical users, our recommendation remains the same: to use X.509 or SAML/JWT authentication when possible. If password-based authentication is used, this will require any potential password change to be performed simultaneously both in the HANA database and on all clients to avoid failed logins or locking of the technical user due to such failed logins.

Moving all application access of the technical users to X.509-based authentication will also allow them to update the certificates of the services individually without having to coordinate this as it would be required for a central password change.

Individual users could still authenticate using passwords, and this distinction would even allow to differentiate between service access and user access. On top of this, you can still rotate the passwords safely if anyone is leaving.

Important: it is recommended to use dedicated database users and not mix technical and named user access in case you have a technical user that is used by services to access the HANA DB, and they seem to also use that same database user for their real users to access the database.

More details on user management you can find here.

Additionally, what you can do for password change is to:

1. Suspend the system during the change of the password, see the SAP notes:

2. Disable on the HANA DB the user lock temporarily by setting User Lock Time to 0 when changing the password. Then, you have time to change the password on the database and the clients.

Important: However, there will almost be a timespan (change of password in user store and change of password on SAP HANA DB) where no application user can work on the system.

Scenario 3: Several teams are working on a project and sharing the same HANA database user across the group

If your concern is that if anyone leaves the project or company, they are still able to access the database through, for example, Database Explorer, or if they have an S-User ID, we recommend the following:

Presumably, when someone leaves the company, their user will be removed from the identity provider and will no longer be able to log in. As for the database users, it might be possible to setup SSO with a mapping between the user used to sign into the identity provider and the database. In that way, once the user is removed from the identity provider, they would no longer be able to log in to the database.
Deletion of separate DB users for each person and make sure the DB user is deleted if a person moves off the project team or company.
Move away from password-based authentication, so another option would be to set up SSO or certificated-based authentication (e.g., X509 or JWT) for created users. Still, you need to delete the respective mapping after the user leaves the project to ensure it has no access like pointed out above for identity provider set up.
Perform a password rotation in the current setup (if possible) whenever a user is leaving the project if you decide to stay with password-based authentication:

You can use the 'ALTER USER <user> FORCE PASSWORD CHANGE' syntax to force a password rotation on the next login for any user (including SYSTEM); details you can find here in the documentation.

Summary

Password rotation is an essential component of a comprehensive cybersecurity strategy. By following the best practices outlined in this blog post, you can significantly enhance the security of your data and applications in SAP HANA Cloud and SAP HANA Platform.

Still, our primary recommendation is to adopt certificated-based authentication and rotation by making use of X509, JWT or SAML, more details on it you can find here: SAP HANA Cloud, SAP HANA Database Security Guide.

Also, some recent additional security settings can be found under the topic connection restriction of SAP HANA Cloud, HANA DB Security Guide, and connect restriction policies, which will be a topic for our next blog post, so stay tuned!

Workload in the SAP HANA

2024-03-06T03:19:54.266000+01:00

Options for Managing Workload

Workload management can be configured at multiple levels: at the operating system-level, by using global initialization settings, and at the session level.

There are a number of things you can do to influence how workload is handled:

Outside the SAP HANA system on the operating system level you can set the affinity of the available cores.
You ca
n apply static settings using parameters to configure execution, memory management and peak load situations.
You can influence workload dynamically at system runtime by defining workload classes.

All of these options have default settings which are applied during the HANA installation. These general-purpose settings may provide you with perfectly acceptable performance in which case the workload management features described here may not be necessary.

Before you begin with workload management, you should ensure that the

System generally is well configured

SQL statements are tuned
Distributed environment tables are optimally distributed and that indexes have been defined as needed.

Area	Possible Actions
CPU	Settings related to affinity are available to bind server processes to specific CPU cores. Processes must be restarted before these changes become effective.
CPU Thread Pools	Global execution settings are available to manage CPU thread pools and manage parallel execution (concurrency).
Memory	Global memory manager settings are available to apply limits to the resources allocated to expensive SQL statements.
Admission Control	Global admission control settings can be used to apply system capacity thresholds above which SQL statements can be either rejected or queued.
Priority and Dynamic Workload Class Mapping	A more targeted approach to workload management is possible by setting up pre-configured classes which can be mapped to individual user sessions. You can, for example, map an application name or an application user to a specific workload class. Classes include the option to apply a workload priority value.

1.Controlling CPU Consumption

SAP HANA can exploit the processor’s NUMA architecture to optimize performance by applying NUMA core affinity settings to server processes and by specifying NUMA node location preferences for specific database objects.

Applying NUMA core affinity settings to server processes.

Specifying NUMA node location preferences for specific database objects

Note: As an alternative to applying CPU affinity settings you can achieve similar performance gains by changing the parameter [execution] max_concurrency in the global.ini configuration file. Changing the configuration file may be more convenient and doesn’t require the system to be offline.

Examining the System Topology

The lscpu command returns a listing of details of the system architecture.
hdbcons

The database management client console hdbcons can also be used to retrieve topology information

Information about the system topology and the CPU configuration is available in system views

select * from SYS.M_HOST_INFORMATION where key in ('cpu_sockets','cpu_cores','cpu_threads');

select MAX_NUMA_NODE_COUNT, MAX_LOGICAL_CORE_COUNT from SYS.M_NUMA_RESOURCES; select HOST, NUMA_NODE_ID, NUMA_NODE_DISTANCES, MEMORY_SIZE from SYS.M_NUMA_NODES;

Applying Affinity Configuration Changes

You can use the affinity configuration parameter to restrict CPU usage of SAP HANA server processes to certain CPUs or ranges of CPUs.You can set restrictions for the following servers: nameserver, indexserver, compileserver, preprocessor, and xsengine ,Few examples for restrictions

ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini', 'SYSTEM') SET ('nameserver', 'affinity') = '0,16'

ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini', 'SYSTEM') SET ('indexserver', 'affinity') = '8-15,24-31'

2. Controlling Parallel Execution of SQL Statements

You can apply ini file settings to control the two thread pools SqlExecutor and JobExecutor that control the parallelism of statement execution.The settings described here should only be modified when other tuning techniques like remodeling, repartitioning, and query tuning have been applied. Modifying the parallelism settings requires a thorough understanding of the actual workload since they have impact on the overall system behaviour. Modify the settings iteratively by testing each adjustment.

You can set a limit for both SqlExecutor and JobExecutor to define the maximum number of threads. You can use this for example on a system where OLAP workload would normally consume too many CPU resources to apply a maximum value to the JobExecutor to reserve resources for OLTP workload.

Lowering the value of these parameters can have a drastic effect on the parallel processing of the servers and reduce the performance of the overall system. Adapt with caution by iteratively making modifications and testing. For more information, see Understand your Workload and SAP Note 2222250 - FAQ SAP HANA Workload Management which contains more details of the workload configuration parameters.

The following SqlExecutor parameters are in the sql section of the indexserver.ini file. sql_executors - sets a soft limit on the target number of logical cores for the SqlExecutor pool.

max_sql_executors - sets a hard limit on the maximum number of logical cores that can be used.

Caution

SAP HANA will not accept new incoming requests if the limit is exceeded. Use this parameter with extreme care.The following JobExecutor parameters are in the execution section of the global.ini or indexserver.ini.

max_concurrency - sets the target number of logical cores for the JobExecutor pool.

max_concurrency_hint - limits the number of logical cores for job workers even

if more active job workers would be available

default_statement_concurrency_limit - restricts the actual degree of parallel execution per connection within a statement.

3. Setting a Memory Limit for SQL Statements

statement_memory_limit - defines the maximum memory allocation per statement in GB After setting this parameter, statements that exceed the limit you have set on a host are stopped by running out of memory.
statement_memory_limit_threshold - defines a percentage of the global allocation limit. Parameter statement_memory_limit is respected only if total used memory exceeds the global allocation limit by this threshold percentage. The default value is 0% (of the global_allocation_limit) so statement_memory_limit is always respected.

Example

4.Managing Peak Load (Admission Control)

The admission control feature is enabled by default Use the admission control feature to apply processing limits and to decide how to handle new requests if the system is close to the point of saturation.Limits can be applied at two levels so that

Request placed in the queued
Higher threshold for maximum workload

and the related threshold values and configurable parameters are available in the indexserver.ini file.

Parameter	Default	Detail
enable	True	Enables or disables the admission control feature.
queue_cpu_threshold	90	The percentage of CPU usage above which requests will be queued. Queue details are available in the view M_ADMISSION_CONTROL_QUEUES. The value 0 or 100 means that no requests are queued.
queue_memory_threshold	0	The percentage of memory usage above which requests will be queued. The value 0 or 100 means that no requests are queued.
reject_cpu_threshold	0	The percentage of CPU usage above which requests will be rejected. The default value 0 means that no requests are rejected, but may be queued.
reject_memory_threshold	0	The percentage of memory usage above which requests will be rejected. The default value 0 means that no requests are rejected, but may be queued.

There are many other parameters which is related to Queue Management such as

queue_wait_time_threshold
record_limit
max_queue_size
dequeue_interval
dequeue_size
queue_timeout
Queue_timeout_check_interval

5. Managing Workload with Workload Classes

You can manage workload in SAP HANA by creating workload classes and workload class mappingsYou can classify workloads based on user and application context information and apply configured resource limitations (related to statement memory or thread limits) or a priority value. Workload class settings override other ini file configuration settings which have been applied and also override user parameter settings workload class settings only apply for the duration of the current session, whereas changes applied to the user persist

Properties for Workload Classes
Priority
This prioritizes statements in the current execution. Priority values of 0 (lowest priority) to 9 (highest) are available; the default value is 5.

CPU and Memory Limits

STATEMENT THREAD LIMIT
default_statement_concurrency_limit
STATEMENT MEMORY LIMIT
statement_memory_limit_threshold
TOTAL STATEMENT THREAD LIMIT
default_statement_concurrency_limit
TOTAL STATEMENT MEMORY LIMIT
total_statement_memory_limit

Timeout Values

STATEMENT TIMEOUT
WRITE TRANSACTION LIFETIME
IDLE CURSOR LIFETIME

Ref and further notes

2938708 - How to create a HANA workload class and mapping for a particular user

2222250 - FAQ: SAP HANA Workload Management

Help.sap.com

Troubleshooting: Removal of deleted Shipment POs from MRRL Batch jobs

2024-03-06T08:57:10.666000+01:00

Introduction:

In this blog post I am going to explain the steps/actions required to avoid the existing deleted/cancelled Shipment POs from appearing into MRRL batch jobs in SAP S/4 Hana.

Scenario:

Scenario at client place is Creating Purchase Order (PO), Goods Receipt (GR) and invoice via MRRL (Background job).

In above mentioned process, purchase order, GR posting & reversal & Invoice posting & reversal happened & then deleted the line item from Shipment PO.

when this MRRL batch job is running that deleted line-item Shipment PO is still occurring in MRRL batch job report.

We are getting below message for above PO in spool batch job report.

“Item 00001 of purchasing document XXXXXXXXX does not exist.”

Steps to avoid this:

We must first uncheck the ERS field in the PO and save it.

Then cancel the invoice of that PO (via MIRO)
After unchecking the ERS field in PO and cancelling the invoice, the further steps to be processed namely GR cancellation (cancel shipment via Transaction code VI02) and PO line-item deletion (via Transaction code ME22N).

And that’s it, with this we can avoid the cancelled/deleted PO from appearing into MRRL background Batch jobs.

Background of ERS: Evaluated receipt settlement involves an agreement between the vendor (creditor) and the buying entity that the former does not issue invoices for purchasing transactions. Instead, the invoice document is posted automatically in the buying entity's system based on data from the PO and goods receipts.

ERS should be maintained in Vendor Master and Purchasing info record

Vendor Master :

Purchase Info Record : Info Category as Standard

Select the GR-Bsd IV indicator
De-select the No ERS indicator

===0====

I sincerely hope that the information provided in this post helps you. Please feel free to comment, ask questions and post your thoughts on how you liked it.

Best Regards,

Prasanna Tiwari

Custom Fields and Logics - Create and Capturing the Changes into Transport Request

2024-03-06T09:38:04.081000+01:00

Custom fields and logics are strong tools that enable users to modify software systems to meet their unique requirements, increasing efficiency and functionality.

Custom fields and logics are frequently implemented using the software system's administration interfaces or configuration tools. Without needing to know programming, users with the proper permissions can use these interfaces to add, edit, or remove custom fields as well as define custom logics.

Custom Fields and logic Fiori app can be used by SAP business specialists and implementation consultants can design their own fields and enhance the existing applications that have enabled their extensibility for the key users.

Custom Logic, Data Source Extensions, and Custom Fields tabs are available by default to fulfill the business requirement.

Custom Fields: We can use this process to create and manage custom fields that can be used to improve applications that have allowed key users to extend their functionality.

Data Source Extensions: Create, edit, and delete data source extensions in order to enable the usage of existing fields in predelivered data sources.

Custom Logic: Through this process, We can develop and manage custom logic to improve applications that allow key users to extend the application logic.

We can determine whether it is or not already exists by using the search option. You can register a new filed enablement that is necessary as part of a business requirement by using the Plus (+) button.

Following the click of the + button, you will be able to search the Business context according to your needs. It will display several entries in the business contexts; you can then choose the appropriate entry based on your needs.

Complete the required fields as shown on the screen below, then click the "Create" button and proceed for publish.

Additionally, we can use the other tabs, such as UIs and reports, mail templates, form templates, and business scenarios, and add the field that is necessary to enable into any of the Fiori Apps in accordance with the requirement.

SAP has provided the options below to save changes into a Transport Request (TR) so that they can be moved into higher systems, such as the Preproduction and Production systems.

Using Configure Software Packages App, we can register the development packages that are required for the project development.

We can add our project development packages, which are already in the system, by using the Add Registration button.

and also have the choice to allow tasks and requests to be handled automatically. A Transport Request and associated task will be generated automatically for this package.

To Register the extensions into a Transport request, SAP has provide another app called Register Extensions for Transport. By using this app, the changes that done can be saved into Transport for further landscape systems.

Note : if Automatic Request and Task handling are chosen, this will be generated automatically in the event that there isn't a transport request. Using the "Assign to Transport Request" button, we can also manually assign the transport request.

Now you can check the changes into your transport request.

Conclusion : This blog post describes how to move the enhancements or extensions made with Custom fields and logic app.

Using Hints with SQLScript procedures

2024-03-12T14:51:12.837000+01:00

Introduction

To resolve a problematic SQL statement inside a SQLScript procedure may require adding a hint to resolve the issue. For general SQL statements, a statement hint can be used to apply and manage hints without the changing the original source code of the calling application.

For SQLScript procedures, table variables are used to assign results of a SQLStatement and the variable can be referenced in a subsequent statement. During the execution call of the procedure, the procedure logic will be compiled and cached. Some logic will be inlined into another statement and some can be materialized into a local temp table. The name of the temp table is determined during the optimization of the procedure.

Due to such behavior, the lifecycle of an inner SQL statement referencing local temp table(s) is only valid until the procedure gets recompiled or invalidated. The recompilation can occur due to logic changes and/or dependent object changes such as metadata changes. So, to add a hint for an inner SQL statement, changing the SQLScript source application code was the only available option. If the source was part of an application standard code, it was even more difficult to add the hint.

To simplify managing hints applied to inner SQL statements, 2 features have been recently delivered for SAP HANA Cloud and HANA 2.0 SPS07.

Adding hints to SQLScript procedure code with cascade
Pinning SQL Hints to SQLScript procedure using statement hints

The details of the functionality can be found in the documentation for both SAP HANA Cloud, SAP HANA database and SAP HANA, platform edition.

Demo environment setup

1. Let's first create a very simple table with some basic data

CREATE TABLE ORDERS(PRODUCT_ID NVARCHAR(10), DESCRIPTION NVARCHAR(100), PRIMARY KEY(PRODUCT_ID));
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000001', 'Description of product 1');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000002', 'Description of product 2');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000003', 'Description of product 3');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000004', 'Description of product 4');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000005', 'Description of product 5');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000006', 'Description of product 6');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000007', 'Description of product 7');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000008', 'Description of product 8');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000009', 'Description of product 9');
insert into ORDERS(PRODUCT_ID, DESCRIPTION) VALUES ('0000000010', 'Description of product 10');

2. Then create 2 procedures that is similar to how AMDP are structured with stub or wrapper procedure and main procedure which is called by the wrapper procedure.

For this example, the procedures are manually created which is very simple. The product_id is passed as input parameter and the data record is returned from ORDERS table for the given PRODUCT_ID.

The main procedure with the following logic to return the data record as output

CREATE OR REPLACE PROCEDURE GET_ORDERS(
	in  "IV_PRODUCT_ID" NVARCHAR(10), 
	out "ET_ORDER" ORDERS)
language sqlscript 
sql security invoker as 
begin
	ET_ORDER = SELECT * FROM ORDERS where PRODUCT_ID = :IV_PRODUCT_ID;
end;

And the stub procedure is created with a timestamp as a wrapper procedure of the main procedure.

CREATE OR REPLACE PROCEDURE "GET_ORDERS#20220227123129"
(
	IN "IV_PRODUCT_ID" NVARCHAR(10)
)
LANGUAGE sqlscript 
sql security invoker as
begin
	call GET_ORDERS("IV_PRODUCT_ID" => :IV_PRODUCT_ID, "ET_ORDER" => :ET_ORDER);
	select * from :ET_ORDER;
end;

Applying hints to inner SQL statements of SQLScript procedure

First, we will check the default behavior prior without apply hints.

call "GET_ORDERS#20220227123129"('0000000010');

A single row will be returned for the given product_id

Looking into the SQL plan cache, only the stub procedure call was cached while the main procedure call was inlined to the stub procedure logic during the SQLScript procedure compilation.

Also, the inner select statement is cached as the following

/* procedure: "DBADMIN"."GET_ORDERS" variable: ET_ORDER line: 7 col: 2 (at pos 141) */ SELECT * FROM ORDERS where PRODUCT_ID = CAST(N'0000000010' AS NVARCHAR(10))

The statement is prefixed with a comment indicating that the result was assigned to ET_ORDER table variable at line: 7 col:2 in GET_ORDERS procedure.

This time we would like to check the behavior of the new features starting with adding hints to SQLScript procedure code with cascade.

Let's clear the SQL plan cache

ALTER SYSTEM CLEAR SQL PLAN CACHE;

Now call the procedure again with adding a hint cascading to all inner statements.

call "GET_ORDERS#20220227123129"('0000000010') with hint (NO_USE_HEX_PLAN cascade);

The result is the same returning 1 record

Then check the SQL plan cache again for the inner SQL statement cached and find the hint cascaded down from the procedure call and used.

/* procedure: "DBADMIN"."GET_ORDERS" variable: ET_ORDER line: 7 col: 2 (at pos 141) */ SELECT * FROM ORDERS where PRODUCT_ID = CAST(N'0000000010' AS NVARCHAR(10)) WITH HINT(NO_USE_HEX_PLAN)

Finally, we will check the 2nd option. Let's clear the SQL plan cache again

ALTER SYSTEM CLEAR SQL PLAN CACHE;

We will now use the statement hint to add USE_HEX_PLAN directly to the inner SQL statement this time.

ALTER SYSTEM ADD STATEMENT HINT (USE_HEX_PLAN) ON PROCEDURE GET_ORDERS 
FOR SELECT * FROM ORDERS where PRODUCT_ID = :IV_PRODUCT_ID;

We call the procedure again without any hints as we had for the default call and check the default behavior prior without apply hints.

call "GET_ORDERS#20220227123129"('0000000010');

Then finally check the SQL plan cache for the inner SQL statement. Notice that the hint was appended with a comment indicating that the hint was added by the Statement Hint.

/* procedure: "DBADMIN"."GET_ORDERS" variable: ET_ORDER line: 7 col: 2 (at pos 141) */ SELECT * FROM ORDERS where PRODUCT_ID = CAST(N'0000000010' AS NVARCHAR(10)) /*added by Statement Hint:*/ with hint(USE_HEX_PLAN)

Conclusion

We have checked 2 different methods of adding hints to inner SQL statements of SQLScript procedure which would be useful to manage the performance of the inner SQL statement of a SQLScript procedure without changing the SQLScript code itself.

But one thing to keep in mind is that mixing the 2 methods can cause unexpected results such as the following by adding 2 conflicting hints.

Effectively adopting Recurring Service process from S/4 HANA Service Management into your business

2024-03-13T15:18:20.952000+01:00

Regular maintenance aids in maintaining a customer's assets at their best performance levels while reducing the likelihood of breakdowns and the costs associated with repairs.

Our preventive maintenance solution, designed to sustain customer assets, facilitates consistent production by mitigating unexpected breakdowns. This solution creates and executes orders at specified intervals/usage, drastically improving asset production efficiency and minimizing breakdowns. Notably, this solution also reduces maintenance costs and contributes to recurring service revenues for customers. It adeptly manages both commercial aspects through service orders, as per the service contract, and technical aspects through execution orders.

There are different types of Maintenance Plans, namely: Single cycle plans based on time or performance, strategy plans, or multiple counter maintenance plans that are implemented on need basis . Once you create a maintenance plan, you can schedule the plan to generate call objects when the system is due for maintenance.

The normal recurring service is the process that is more straightforward and is suitable for simple servicing scenarios and is inclusive of commercial aspects in the same document. The Service with Advanced Execution is a process that combines commercial aspects of service with maintenance management for tasks that require detailed planning and execution. Maintenance task lists help in planning and organizing maintenance work efficiently, saving time during scheduling. They're especially useful for frequent tasks requiring the same materials, and for recurring and one-time tasks. The service part handles the customer's needs, pricing, satisfaction, and profit, while the maintenance management part focuses on planning, understanding equipment technicalities, and carrying out the service. The process creates Execution Order Items, which can be part of a service order or a maintenance plan. These items generate a maintenance order, enabling the service's planning and execution.

The information below explains how to differentiate between the two.

Image 1 : Apps related to Recurring Service

Image 2: Recurring Service with Advanced Execution

S.I No	Category	Recurring Service	Recurring Service with Advanced Execution
1	Scenarios	Classic Recurring Service	Preventive Maintenance
2	ABAP Call Type Technical code	Maintenance Service Call Type '5'	Maintenance Service Call Type '6'
3	Types of MPLAN supported	Single Cycle (Time or Performance), Multiple-Counter	Single Cycle (Time or Performance), Multiple-Counter, Strategy Time Based. Strategy Performance Based
4	Preceding Document	Service Template	Service Contract
5	Mandatory Inputs for Scheduling	Service Template Either Technical Object OR Service Contract Call Object Type (Service Order or Service Quotation)	Service Contract Work Centre Planning Plant Call Object Type (Service Order)
6	Output	Service Order or Service Quotation	Service Order with Execution Order
7	Source of Service Transaction Items	Service Order Template	Service Contract Items
8	Task List	Table OISD (Service Product with Task List)	Directly from Maintenance Item, else from table OISD.
9	Supported Service Document Types	Service Order and Service Quotation	Service Order
10	Applications support and GUI support	Supports on Fiori Apps	GUI (TCODES: IP01/02/03 and IP10, etc) and Fiori Apps

The ABAP code flow for simple recurring service is as follows:

FAELLIGE_MHIS_ORDER ( FM) ->
PERFORM service_order_create ->
CL_CRMS4_MPLAN_SCHEDULING-IF_CRMS4_MPLAN_SCHEDULING~CREATE_ORDER( ) ->
CL_CRMS4_MPLAN_ORDER_CREATION-IF_CRMS4_MPLAN_ORDER_CREATION~CREATE_ORDER( )->
CL_CRMS4_MPLAN_ORDER_CREATION->CREATE_ORDER_FROM_TEMPLATE( )->
CL_CRMS4_MPLAN_ORDER_PREPARE-IF_CRMS4_MPLAN_ORDER_PREPARE~GET_DATA_FOR_CREATION ( )->
CRM_ORDER_MAINTAIN->( call back to FAELLIGE_MHIS_ORDER )->
PERFORM service_order_post_mhio (calling CRM_ORDER_SAVE).

The ABAP code flow for recurring service with execution order item ( Advanced Execution ) is as follows:

FAELLIGE_MHIS_ORDER (FM) ->PERFORM create_srvo_pmo->
SERVICE ORDER CREATION: PERFORM service_order_create ->
CL_CRMS4_MPLAN_SCHEDULING-IF_CRMS4_MPLAN_SCHEDULING~CREATE_ORDER( ) ->
CL_CRMS4_MPLAN_ORDER_CREATION-IF_CRMS4_MPLAN_ORDER_CREATION~CREATE_ORDER( )->
CL_CRMS4_MPLAN_ORDER_CREATION->CREATE_ORDER_FROM_CONTRACT( )
CL_CRMS4_MPLAN_ORDER_PREPARE-IF_CRMS4_MPLAN_ORDER_PREPARE~GET_DATA_FOR_CREATION ( )->
CRM_ORDER_MAINTAIN->(call back to PERFORM create_srvo_pmo )->
MAINTENANCE ORDER in BUFFER :
PERFORM order_create_f13 -> (call back to FAELLIGE_MHIS_ORDER)->
PERFORM update_srvo_pmo->
EXACT OBJECT ID of PLANT MAINTENANCE ORDER:
PERFORM order_update_f13->( call back to PERFORM update_srvo_pmo )->
(Logic for Creation of the DOCFLOW between Plant Maintenance Order and Service Order and vice versa).

In conclusion, the distinction between normal Recurring service and Service with Advanced Execution lies in their complexity and scope. The decision to select one over the other depends on the specific needs and requirements of the business.

How Data Anonymization in SAP HANA secure Generative AI applications through SAP Datasphere

2024-03-15T20:12:44.703000+01:00

Introduction

As Generative AI continues to evolve, the complexities surrounding the management of Personally Identifiable Information (PII) also increase.AI systems can sometimes deduce PII from seemingly harmless data or unintentionally reveal PII in their results. Therefore, it's essential for AI developers and users to understand these potential risks and take proactive measures to address them.

Personally identifiable information (PII) refers to any information that can be used to identify an individual. This might include, but is not limited to, name, social security number, date and place of birth, mother’s maiden name, or biometric records. In the digital age, it could also include digital identity data such as IP addresses or mobile device IDs.

Handling PII responsibly is critical for many reasons:

Respecting Privacy
Ensuring Security
Building Trust

On the regulatory front, there are laws including the European Union's General Data Protection Regulation (GDPR) and the U.S.'s California Consumer Privacy Act (CCPA) that offer a framework for PII management. These laws empower individuals with rights over their personal data. However, it's worth noting that these regulations are relatively recent and still evolving. There's also ongoing discourse about the optimal way
to regulate AI systems to balance the need for privacy protection and technological innovation. SAP HANA Data Anonymization provides methods in order to overcome all the above potential risks. The methods supported by SAP HANA are :

k-anonymity
l-diversity
Differential privacy

More information on the SAP HANA Documentation :
https://help.sap.com/docs/SAP_HANA_PLATFORM/f88e51df089949b2af06ac891c77abf8/ee693d6584d243e1a0daf7c137f9600c.html

Architecture

On this blog we will explore the various solutions for tackling PII-related challenges through the SAP Business Technology Platform. We'll be focusing on SAP Datasphere's capabilities, which not only enable us to consolidate all necessary data sources, SAP or non-SAP, but also leverage SAP HANA's powerful data anonymization methods. This combination ensures your data's security while maximizing its potential.

Considering the unique capabilities and skills of the personas interacting with SAP Datasphere, we have come to the following options.

We really hope that the first option , direct usage of Data Anonymization annotations from SAP Datasphere interface to be on the roadmap really soon.

A generic architecture of a Generative AI project, where SAP Datasphere takes center stage, can be seen below :

More information regarding the CAP LLM Plugin , please check this link

https://community.sap.com/t5/technology-blogs-by-sap/elevating-data-privacy-in-sap-cap-based-genai-applications-with-cap-llm/ba-p/13585745

Hybrid Approach, focus mainly for Business personas

The starting point for business users will be a local table accessible through the SAP DSP interface (INPUT).
We'll elaborate more on this in the upcoming images. The entire process will be seamlessly managed behind the scenes, offering a user-friendly experience (PROCESS). It will be set up just once on the underlying SAP HANA of SAP DSP. A procedure will generate an Anonymization view, which will be displayed on the SAP Datasphere interface (OUTPUT). This procedure is programmed to run every minute. Only when the user activates the "enable" flag during the initial step will this procedure execute. Otherwise, it simply remains inactive. The final output will be an SAP Datasphere view, tailor-made for business users based on their specific requirements. The anonymized view produced can be distributed by the Data Controller to the necessary DSP spaces, substituting the original non-anonymized dataset. This straightforward step allows us to exert additional control over the access permissions for the Data Consumer.

Hybrid approach

Details regarding the Configuration (SAP Datasphere Local Table)

Information regarding SAP HANA Procedure and Scheduler

Conclusion

In conclusion, the use of SAP Datasphere in a generative AI scenario presents an array of advantages. It significantly enhances data handling and security, ensuring the protection of personally identifiable information through SAP HANA's anonymization methods. The versatility of SAP Datasphere allows it to cater to both IT and Business personas, offering a range of implementation methods including direct, SQL, Python or via a CAP Application. Being a one-stop solution for all generative AI needs, its hybrid approach is highly flexible, capable of extending and adapting to any specific requirement.

This makes SAP Datasphere a highly efficient and user-friendly tool for handling generative AI scenarios.

Taking Data Federation to the Next Level: Accessing Remote ABAP CDS View Entities in SAP HANA Cloud

2024-03-25T13:35:26.317000+01:00

Accessing data in ABAP systems, such as SAP BTP ABAP environment, S/4HANA Cloud, BW/4HANA, and others, from an external SAP HANA database is not merely a new requirement or passing trend; it has been a longstanding and ongoing need to enhance and streamline business processes. This need becomes even more crucial when considering the SAP HANA Cloud, which serves as a single gateway for all your enterprise data and where next-generation Intelligent Data Applications are built and run.

However, there are several reasons why directly accessing the ABAP-managed standard objects in the underlying SAP HANA database of an ABAP system is not recommended. These include the instability of names and internal structures, unexpected typecasts, incorrect session variables, and bypassed ABAP-level security, among other concerns. For more details on these issues, please refer to SAP Note 2511210.

To overcome these challenges, data federation through the Smart Data Integration (SDI) ABAP adapter has been proposed as a potential solution, while data replication is supported by both the SDI ABAP adapter and SAP Landscape Transformation (SLT). It is important to mention, though, that the SDI ABAP adapter, as explained by Zili Zhou in her blog, is primarily designed for data replication and may have limitations when it comes to data federation due to limited pushdown capabilities.

In this context, I'm glad to share that starting with SAP HANA Cloud, SAP HANA Database (SAP HANA database in SAP HANA Cloud) QRC 01/2024, a new Smart Data Access (SDA) ABAP adapter has been released. This adapter allows for accessing ABAP CDS view entities in a remote ABAP system using SQL via Open Database Connectivity (ODBC) from an SAP HANA database in SAP HANA Cloud. Please find a nutshell of this new capability below:

Create a remote source to an ABAP system based on SAP ABAP Platform 2108 or later.
Virtualize remote ABAP CDS view entities by creating read-only virtual tables that point to them, and if needed, also pass parameters to the remote parameterized ABAP CDS view entities.
All limitations from SAP Note 2511210 have been lifted, providing the most optimal pushdown capabilities for data federation.
This capability is only available in SAP HANA database in SAP HANA Cloud.

Prerequisites & considerations

First, please ensure that the versions of the ABAP system and SAP HANA database in SAP HANA Cloud are as mentioned below:

The ABAP system should be based on SAP ABAP Platform 2108 or a later version where SQL service is available. For instance, starting with S/4HANA 2021, SQL service has been available.
SAP HANA Cloud, SAP HANA Database (SAP HANA database in SAP HANA Cloud) should be QRC 01/2024 or a later version where ODBC driver for ABAP is available, and the script server should be enabled and running.

Another aspect that should be considered is whether the ABAP system is protected by a corporate firewall (e.g., on-prem S/4HANA). If it is, then SAP Cloud Connector is required.

If the above conditions are met, the next step is to expose ABAP CDS view entities from the ABAP system side so that they can be accessed from SAP HANA database in SAP HANA Cloud via the SQL service.

Since developing and exposing an SQL service in the ABAP system is not the focus of this blog and has already been well-explained by Frank-Martin Haas in his blog, please refer to his blog and the following links.

SAP Business Technology Platform: Developing and Exposing an SQL Service in the ABAP System
SAP Tutorials: Using ODBC driver for SQL Service (From Step 3 to Step 9)

Getting started with SDA ABAP adapter (abapodbc)

If everything is ready in the ABAP system to expose ABAP CDS view entities, let’s configure the SDA ABAP adapter from the SAP HANA database in SAP HANA Cloud.

The configuration steps are almost the same as using SDA with other remote sources: downloading the DigiCert SSL certificate, importing the certificate for SSL connection, creating a remote source, and finally, creating a virtual table. Please refer to the examples below.

Download the DigiCert SSL certificate:

-- DigiCert Global Root CA
curl -O https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem

-- DigiCert Global Root G2
curl -O https://dl.cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem

Since DigiCert has stopped signing certificates with the root certificate authority (CA) "DigiCert Global Root CA" and is now moving forward with the "DigiCert Global Root G2" certificate, it is recommended to add both certificates during the switching period. However, using only one certificate is basically sufficient.

Import the certificate for SSL connection:

-- import the certificate for SSL connection (CA)
CREATE PSE SSL;
CREATE CERTIFICATE CA_CERT FROM '
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
...
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----
';

-- import the certificate for SSL connection (G2)
CREATE CERTIFICATE G2_CERT FROM '
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
...
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----
';

ALTER PSE SSL ADD CERTIFICATE CA_CERT;
ALTER PSE SSL ADD CERTIFICATE G2_CERT;
SET PSE SSL PURPOSE REMOTE SOURCE;

Create a remote source:

-- create a remote source to the ABAP SQL service endpoint
CREATE REMOTE SOURCE ABAP ADAPTER "abapodbc" 
CONFIGURATION 'uidtype=alias;driver=ODBC_driver_for_ABAP.so;servicepath=/sap/bc/sql/sql1/sap/s_privileged;host=<abap_host_name>;port=<abap_port_number>;language=EN;typemap=semantic;'
WITH CREDENTIAL TYPE 'PASSWORD' USING 'user=<user_name>;password=<password>';

Alternatively, like other SDA adapters, the remote source can be created using the SAP HANA Database Explorer UI. Once the remote source is created, the exposed ABAP CDS View entities can be browsed using the SAP HANA Database Explorer as described below.

Create virtual tables:

-- create virtual tables by pointing to the exposed ABAP CDS view entities
CREATE VIRTUAL TABLE DEMO.V_ORDERS AT "ABAP"."NULL"."ZORDERS"."Orders";
CREATE VIRTUAL TABLE DEMO.V_ORDERITEMS AT "ABAP"."NULL"."ZORDERS"."OrderItems";

Alternatively, virtual tables can also be created using the SAP HANA Database Explorer UI by selecting the remote object and clicking on the 'Create Virtual Object(s)' button in the screenshot above.

Please also refer to the links below for further details.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Import Certificates for SSL Connections to Remote Source
SAP HANA Cloud, SAP HANA Database Data Access Guide: Create an ABAP Cloud Remote Source

Before moving forward, please don’t forget to create statistics on virtual tables. Creating statistics is always strongly recommended to ensure optimal decision-making on the query execution plan by the query optimizer.

Create statistics on a virtual table:

CREATE STATISTICS ON DEMO.V_ORDERS TYPE SIMPLE;
CREATE STATISTICS ON DEMO.V_ORDERITEMS TYPE SIMPLE;

Further details can be found at the link below.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Create Statistics on a Virtual Table or Linked Database

Query execution and execution plan

With above configurations, if the same statement from the blog by Frank-Martin Haas is executed but with virtual tables, the result will be as follows.

SELECT OI."Item", SUM(OI."Amount")
FROM DEMO.V_ORDERS O
INNER JOIN DEMO.V_ORDERITEMS OI
ON O."Id" = OI."Orderid"
GROUP BY "Item";

Now, let’s see the query plan using Analyze - Explain Plan.

As you can see, both aggregation and join are pushed down to the remote ABAP system and executed there. In other words, both aggregation and join are performed on the remote side, and then the result is transferred to the SAP HANA database in SAP HANA Cloud. Since SDA is a part of SAP HANA core like the query optimizer, if statistics are available, the query optimizer will create the most optimal execution plan based on the tight integration with SDA.

Other limitations

The most important limitation to keep in mind is that the SDA ABAP adapter supports only read-only virtual tables at the moment. This leads to the following limitations:

DMLs (e.g., INSERT/UPDATE/DELETE) cannot be executed on virtual tables.
WITH REMOTE is not supported when executing CREATE VIRTUAL TABLE.
Join relocation to a remote ABAP system, which requires temporary table creation, is not supported. However, if another remote source with temporary table creation capability is involved additionally, join relocation to that remote source is possible but it will be decided by the query optimizer.

FAQs

Q: Is this SDA ABAP adapter only available in the SAP HANA database in SAP HANA Cloud? Is there any plan to support this in SAP HANA Platform 2.0?
A: Yes, it is only available in the SAP HANA database in SAP HANA Cloud. There are no concrete plans to support this in SAP HANA Platform 2.0.

Q: Is there any plan to support real-time replication, such as SDA-based remote table replication described in this blog? If not, what is the recommended technology for replicating ABAP CDS view entities?
A: There are no plans to support real-time replication with the SDA ABAP adapter. The SDA ABAP adapter is designed for data federation purposes only, even though toggling to a snapshot replica is supported to improve query performance. For replicating ABAP CDS view entities, it is recommended to use SDI ABAP adapter or SLT.

Q: Is it possible to pass parameters to the remote ABAP CDS view entity through the virtual table?
A: Yes, passing parameters to the remote ABAP CDS view entity is supported. Please refer to the link below.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Virtualizing Parameterized Views

Q: Is it possible to connect to the on-prem ABAP system? If so, how can it be done?
A: Yes, the way of connecting the on-prem ABAP system can be found in the links below.

SAP HANA Cloud, SAP HANA Database Data Access Guide: Create an ABAP On-Premise Remote Source

However, please ensure that the following conditions are met:

The on-prem ABAP system is based on SAP ABAP Platform 2108 or later version.
SAP Cloud Connector is properly configured so that the on-prem ABAP system is ‘reachable’ from the SAP Cloud Connector and SAP BTP Cockpit.
SAP Cloud Connector is enabled, and the corresponding IP addresses are allowed in the SAP HANA database configuration in SAP HANA Cloud.
The CREATE REMOTE SOURCE statement includes the correct specifications for useCloudConnector and ConnectivityLocationId properties.

Q: It seems that only the ‘technical user’ can be used for authentication. Are there any plans to support other authentication methods?
A: Yes, other authentication methods will be considered in the future.

Q: Is it also possible to access remote ABAP CDS (DDIC-based) views or other remote ABAP-managed objects?
A: No, only remote ABAP CDS view entities or remote parameterized ABAP CDS view entities can be accessed. For the differences between ABAP CDS (DDIC-based) views and ABAP CDS view entities, please refer to this blog.

Q: Is it possible to use the linked database feature?
A: The linked database feature with a 3-part name is supported. However, passing parameters through the linked database feature is not supported. Linked database optimized mode is also not supported in SAP HANA Cloud.

Conclusion

To conclude, it is evident that accessing data in ABAP systems plays a vital role in the development of Intelligent Data Applications on the SAP HANA database in SAP HANA Cloud. This also aligns perfectly with the concept of Clean Core, which involves minimizing customizations and rearchitecting them on SAP BTP, particularly on the SAP HANA database in SAP HANA Cloud. By elevating data federation to the next level with SDA ABAP adapter and embracing these practices, organizations can unlock the full potential of their data, leveraging the power of SAP HANA Cloud. So, don't miss out on the opportunity to harness the benefits of Intelligent Data Applications and propel your business towards success with the SAP HANA database in SAP HANA Cloud.

Product Updates for SAP Business Application Studio - March 2024 Edition

2024-03-26T08:39:28.924000+01:00

Dear community,

Welcome to our detailed exploration of some recent enhancements within the SAP Business Application Studio. This post will guide you through key updates, including our streamlined Storyboard, a newly designed 'Welcome' page, the introduction of SAPUI5 adaptation projects, a comprehensive SAPUI5 Walkthrough Tutorial with TypeScript, and the new feature – release of the powerful UI5 linter. Each feature mentioned contributes to overall user productivity, designed to improve your working experience with us.

Enhancing Productivity with our Updated Storyboard

SAP Business Application Studio now provides a streamlined storyboard. The improved interface allows you to readily view which service actions and functions your application includes. This overview aids in the comprehension and management of your tools, ensuring you are using your resources to their full potential.

We've also added the option to trigger the creation of unbound actions and functions directly. This eliminates the need for extra steps and consolidates key tasks in one location, reducing the time required for task navigation and management.

The described enhancements improve the experience in two major ways: by having a more complete, clearer visualization of your app's functions and by simplifying the creation and editing process.

Have a look at some sample of our updated storyboard below:

Updated Storyboard of SAP Business Application Studio

New Welcome Page for SAP HANA Applications

Our new 'Welcome' page, designed with your needs in mind, serves as an efficient gateway to interact with SAP HANA-related features within SAP Business Application Studio. This hub offers enriched documentation, insightful blogs, and direct access to various integral aspects that can facilitate and speed up your SAP HANA development journey.

SAP HANA Welcome Page in SAP Business Application Studio

As soon as you start your SAP Business Application Studio instance for SAP HANA Native Applications, you will find this new page. Some of the hyperlinks point to guides within SAP Business Application Studio, others to outside of it, including blogs on community.sap.com and material on help.sap.com.

The information is segmented into five structured sections. The 'Start' segment provides straightforward access to SAP HANA project creation, while the 'Development' section enlists tools to accelerate initiating a project or service.

The 'Blog posts' segment serves as a crucial knowledge base, offering valuable insights into SAP HANA development, including vital components like calculation view modeling and best practices. For those aspiring to refine their skills, the 'Get Started with SAP HANA' section presents comprehensive learning materials related to SAP HANA development, facilitating continuous skill improvement.

Lastly, the 'Help' section plays the role of an informative guide, providing exhaustive documentation pertaining to SAP HANA development topics. In essence, our 'Welcome' page aims to be a comprehensive, easy-to-navigate resource, dedicated to refining your SAP HANA developmental experience, right from the outset.

SAPUI5 Adaptation Projects for SAP S/4HANA Cloud Public Edition

SAP Business Application Studio now enables a significant feature for enhancing SAP S/4HANA Cloud Public Edition - the ability to employ SAPUI5 adaptation projects. These projects enable the creation of application variants tailored to fulfill the specific business requirements of diverse business roles, organizational units, and countries. They offer the ability to design efficient processes by concealing unnecessary fields, representing data in chart formats, or alternation between responsive and grid tables for enhanced data comprehension.

Creation of an SAPUI5 Adaptation Project

The primary advantage of an adaptation project lies in allowing you to leverage the original SAP Fiori application, its OData service, and application logic, to exclusively govern modifications specific to the application variant. Upon deployment, the source application and the derived application variant co-exist under distinct titles, accessible to corresponding business users via uniquely named application tiles in ‘My Home’, the new SAP S/4HANA Cloud Public Edition’s starting point.

See this blog post from Elisabeth Riemann linking to an extensive, end-2-end tutorial.

Did you know that adaptation projects are a part of ‘SAPUI5 flexibility’? Everything you want to know about this toolset is described in our Help Portal.

SAPUI5 Walkthrough Tutorial with TypeScript

Our UI5 team has created this tutorial to introduce all development paradigms of SAPUI5 and also demonstrate the benefits of TypeScript with SAPUI5.

This tutorial provides extensive guidance on key SAPUI5 development principles and data binding concepts through creating an application from scratch. After this, the tutorial guides you through enhancing the app with advanced features like navigation, control extension, responsiveness, and exploring built-in support tools.

Introducing UI5 linter

We have introduced UI5 linter, a static code analysis tool that enhances code quality, readability, security, and efficiency. Its initial features focus on validating compliance with the Best Practices for UI5 Developers, reporting areas of concern like deprecated APIs, synchronous loading mechanisms, and global variables usage. The UI5 linter can be easily installed and used via .npm package manager and is instrumental in preparing your project for future versions of UI5.

Use the UI5 linter to identify problematic code that does not follow UI5 best practices.

Learn more about UI5 linter in a blog post from Oliver Graeff.

That brings us to the end of this update. We hope you found this insight into our recent enhancements beneficial. As we continue to enhance SAP Business Application Studio, we encourage you to explore and leverage these new features to increase your productivity. Thank you for your ongoing commitment to our products, and we ensure that we remain dedicated to improving your user experience. Stay connected for our next round of updates!

Happy coding!

Safeguarding Data Privacy in KSA - Leveraging SAP to navigate NDMO’s Regulations in Digital Era

2024-03-28T12:05:30.971000+01:00

Blog v1.0 | Published On: 28 March 2024

Authors: @asadkhan02 , @AyeshaSafeer , @Zainab_ASalam

_________________________________________

In today's digital era, data privacy is a crucial issue for both individuals and organizations. The Saudi National Data Management Office (NDMO), in partnership with the Saudi Data and Artificial Intelligence Authority (SDAIA), has introduced stringent Data Governance and Personal Data Protection Standards. These regulations mandate all organizations operating across various industries in the Kingdom of Saudi Arabia comply by September 2024.

Failure to comply with these regulations can result in hefty financial fines reaching SAR 3 million or higher in some cases, reputational damage, legal consequences, and loss of trust among customers and partners.

To ensure compliance with these new regulations, organizations are encouraged to implement processes aligned with the 15 domains outlined by the NDMO for Data Governance and Personal Data Protection Standards. Leveraging technology as an enabler, organizations can implement robust data privacy measures and effectively meet these requirements.

Top 5 Areas part of 15 Domains outlined by NDMO for Data Governance and Personal Data Protection Standards

SAP, a global leader in enterprise software, provides advanced technologies with artificial intelligence (AI) capabilities that provide a solid foundation for organizations to implement data governance and personal data protection processes. SAP solutions enable Saudi organizations to efficiently navigate and fulfill regulatory requirements:

By implementing SAP solutions, Saudi organizations can empower their data privacy practices, mitigate compliance risks, and build trust among stakeholders. SAP technologies uphold data integrity, safeguard personal information, offer the framework for regulatory compliance implementation, and adapt to the demands of the digital age.

As a team at SAP, we are committed to supporting organizations in achieving their regulatory compliance initiatives. We invite you to take the next step by exploring how our technologies and solutions can assist you on this journey.

Contact us today to learn more about how SAP can help you navigate the complex landscape of data privacy regulations in Saudi Arabia and ensure compliance.

Elucidation of Associations in CDs Views

2024-04-02T20:20:00.019000+02:00

ASSOCIATIONS:

ASSOCIATIONS are kind of Joins to fetch data from multiple tables on Join Conditions but these are ‘JOINS ON-DEMAND’ i.e. they will only be triggered when user would access the required data which needs the Association of tables. For example, your CDS view has 4 Associations configured and user is fetching data for only 2 tables, the ASSOICATION on other 2 tables will not be triggered and the system would return the results quickly, so it enables really high turn-around time as compared to regular SQL JOINS.
Associations are defined with ‘Cardinality’. Syntax : association[<cardinality>]
Cardinality concept is not new and holds the same concept with CDS views as well. There are 4 types of Cardinality possible based on the data and relationship in the tables joined;

0..1
0..n or 0..*
1..0
1..n or 1..*

NOTE: If you are confused on what kind of association I should configure in my CDS view then you can apply a rule of thumb: ‘always use this : association[1]. This will always trigger an OUTER join and will work in all the cases.
Let’s create a CDS view with ASSOCIATION concept.

Create View with Association template;

Below is the default structure you will get. As you can see the association cardinality is defined as ‘Association[1]’. You can keep it as it is, if you are not sure what association you have to assign.
Check the default Association name as _association_name. This can be any name of your choice but SAP has created a naming convention standard to start this name with an underscore(_) and it is always good to follow SAP standard best practices.

Add the required data elements to complete the view. Lets take the same example we did with JOIN view.

EXPOSED Association

Right click and select ‘Show SQL CREATE Statement;

As you can see, NO Join is created;

We can add separate fields in the CDS view as well like a join and that is called Ad-Hoc Association Concept. In this scenario a join will be performed beforehand.

AD-HOC Association:

Check the SQL CREATE statement. Note that a ‘LEFT OUTER MANY TO ONE JOIN’ is created. The join created because we selected individual fields instead of exposing the whole Association and it is LEFT OUTER MANY TO ONE i.e. *..1 due to the fact that we configured cardinality like ASSOCIATION[1].

As we saw above the ASSOCIATION concept with EXPOSED and AD-HOC Associations. Now lets activate the view with Exposed Association and see how it performs a join at RUN Time.

Activate this view and hit F8 to execute;

Execute and look at highlighted – CDS View name and CARRID data AA and AZ. Also note that NO data from table SPFLI is displayed as NO join is yet performed.

Now if you want to see further Airport information for CARRID ‘AA’ from table SPFLI. right click on any row with CARRID ‘AA’ and choose ‘Follow Association’. Another pop-up window will opn, click the Association type;

Check data: Join is now performed ON-DEMAND and data only for CARRID ‘AA’ is displayed based on Association Key.

In summary, Association concept is one of the wonderful things I liked about CDS views. So intelligently, SAP developed this to save the effort in reading only the information which is required for the analytics purpose. You can feel the real difference after designing a very complex CDS view with a combination of multiple table and other CDS views. The very complex CDS view still performance great when Associations are used. In this blog post, we saw the Association concept together with Exposed and Ad-Hoc associations.

Workload Analysis for HANA Platform Series - 2. Analyze the CPU, Threads and Numa Utilizations

2024-04-02T22:26:10.409000+02:00

In the previous blog, we had discussed the workload pattern. This blog dissects how the CPU, threads and NUMA related topic will influence the workload, and how to figure out the most useful information through the workload analysis. The conclusion can be used for further tuning the HANA system into a good shape on CPU resources consumption.

How to Begin with CPU Utilization Analysis

The analysis of CPU utilization stands as a key element within the broader scope of workload analysis. Determining the right approach to analyze CPU usage is crucial. It's essential to first identify the type of CPU utilization that may be problematic—be it user CPU utilization within the HANA instance, system CPU utilization governed by the OS, or a combination of both. A clear understanding of this distinction is necessary to properly address the issue.

Should the user CPU utilization be significant, it would be prudent to delve into thread information to gain insights into ongoing processes. Conversely, if the focus is on system CPU utilization from the OS, gathering thread data alongside call stack histories is advisable. Collaboration with the OS Administrator might also be necessary to investigate any scheduled jobs or activities that coincide with peak usage times.

In this blog, our primary focus is on user CPU utilization, which is directly impacted by the workload within the HANA instance. A valuable starting point for gaining deeper insights is to analyze thread samples. The monitoring views pertinent to this are:

Although both views contain very similar data, for the purpose of workload analysis, we particularly concentrate on specific columns within these views:

HOST	VARCHAR(64)	Displays the host name.
TIMESTAMP	TIMESTAMP	Displays the timestamp of the record.
THREAD_TYPE	VARCHAR(128)	Displays the thread type.
THREAD_METHOD	VARCHAR(256)	Displays the thread method.
THREAD_DETAIL	NVARCHAR(256)	Displays the thread detail (truncated).
THREAD_STATE	VARCHAR(32)	Displays the thread state.
STATEMENT_HASH	VARCHAR(32)	Displays the unique identifier for an SQL string.
ROOT_STATEMENT_HASH	VARCHAR(32)	Displays the MD5 hash value for the root statement string.
USER_NAME	NVARCHAR(256)	Displays the SQL user name.
APPLICATION_NAME	NVARCHAR(256)	Displays the name of the application.
APPLICATION_USER_NAME	NVARCHAR(256)	Displays the application user name.
APPLICATION_SOURCE	NVARCHAR(256)	Displays that the application can define which source file SAP HANA is called from. The usage is up to the application. This value is also displayed in M_PREPARED_STATEMENTS.APPLICATION_SOURCE.
STATEMENT_THREAD_LIMIT	INTEGER	Displays the effective statement thread limit.
STATEMENT_MEMORY_LIMIT	INTEGER	Displays the effective statement memory limit.
PASSPORT_COMPONENT_NAME	NVARCHAR(32)	Displays the passport component name.
PASSPORT_ACTION	NVARCHAR(40)	Displays the passport action.
NUMA_NODE_INDEX	SMALLINT	Displays the last known NUMA node that the thread was executed on.
WORKLOAD_CLASS_NAME	NVARCHAR(256)	Displays the name of the workload class.

Please find more information in SAP Note 2114710 - FAQ: SAP HANA Threads and Thread Samples

Considerations for CPU Utilization in HANA Workload Analysis

Now that we have the tools at our disposal, where should we initiate our analysis of CPU utilization? Addressing CPU usage within SAP HANA is multifaceted. A high CPU usage, peaking at 90% of total capacity—for instance, with 8 sockets, 208 CPU cores, and hyper-threading—suggests that approximately 374 threads (90% of 208 x 2 logical CPU cores) are active. This scenario often leads to system hang-ups and prevents new user connections. However, this is not the sole concern.

We also face perplexing situations where a mere 50% CPU utilization can cause similar system hang-ups, or times when the system appears idle yet experiences high system CPU utilization. Moreover, there are instances where a single statement monopolizes all threads despite workload classifications and global concurrency limits being in place.

To unravel these complexities, a thorough review of thread samples, supplemented by other monitoring views, is necessary for deeper insight into the system's behaviour.

Typically, the following factors can lead to CPU-related issues in a HANA system:

OS configurations
Expensive SQL statements alongside impractical concurrency settings
Absence of secondary indexes on column tables
NUMA node contentions
System CPU utilization at the OS level
Concurrency controls implemented by applications
Conflicting schedules of background jobs
Activities of the statistics server
Admission control setting
And more.

It's worth noting that high CPU usage doesn't inherently indicate a problem. At times, it may signify that the HANA system is effectively utilizing its capacity to handle the workload, which is not a concern. Our goal is to identify workload contributors to minimize CPU consumption and safeguard the system against potential future issues.

Please find more information in SAP Note 2100040 - FAQ: SAP HANA CPU

Analyzing CPU Utilization Issues: Two Practical Examples

In this section, we simulate two real-world scenarios to provide deeper insights into CPU utilization analysis in SAP HANA environments.

Before diving into the analysis of expensive statements, it's crucial to establish an understanding of the system's configuration that affects workload management. In the examples that we will explore, the system is equipped with :

8 sockets, and each socket houses 52 logical CPU cores.
global.ini -> [execution] -> default_statement_concurrency_limit set to 12
Admission control set to default (indexserver.ini -> [admission_control] -> queue_cpu_threshold = 90)

Example 1 - CPU Spikes Caused by Expensive Statements

Expensive statements in a database context typically refer to queries that run longer than anticipated or consume a significant amount of memory during execution. In the realm of workload analysis, especially concerning CPU consumption, we focus on those statements that utilize a large number of running threads. In our example, various types of statements have been identified that could potentially cause extremely high CPU utilization. These could be so intensive as to lead to system hang-ups.

This situation might involve complex join operations, extensive data aggregation, or poorly optimized queries that put a heavy load on the CPU. Identifying and optimizing these expensive statements is crucial to prevent CPU spikes and maintain system stability. This process involves analyzing query execution plans, reviewing indexing strategies, and possibly restructuring or simplifying the queries themselves to reduce their resource demands.

In the given example, we observe distinct CPU utilization patterns over a span of two weeks on the HANA platform, with utilization nearing 100%. The first week’s pattern is characterized by abrupt spikes in CPU usage, quickly rising and falling, whereas the second week displays more frequent and enduring peaks.

Peak CPUs Lasting for a Few Minutes: Such short-term CPU spikes are not unusual in database environments. They typically arise from intensive data loading, execution of complex queries, or during routine operations like backups or batch processing. However, the spikes observed here warrant close examination due to their extreme frequency and high utilization rates, which could indicate underlying issues needing immediate attention.
Peak CPUs Lasting for Hours: The prolonged CPU peaks, extending over several hours, present a more critical challenge. They could be symptomatic of deeper issues such as inefficient query design, absence of necessary secondary indexes, or even physical hardware constraints. Persistent high CPU utilization of this nature demands a comprehensive analysis to pinpoint the fundamental causes and to devise appropriate strategies for optimization and adjustment to prevent potential system degradation or failure.

It turns out that three major contributors are responsible for the high CPU utilizations within these two weeks.

Statement 1: Executed by user axxx from application source axxx:1, this statement has been stable in production for years. Following a logical adjustment on the application side intended to enhance capacity during request processing, the number of connections initiated by this statement increased. Despite the implementation of a global statement concurrency parameter and a designated workload class, this statement alone occupied 90% of CPU resources when activated. The immediate solution is to curb the number of concurrent connections either at the application level or by refining the HANA workload class parameters. Specifically, adjusting the workload class "wlc_1" to impose a stricter statement concurrency limit is recommended, especially by employing the Hierarchies of Workload Classes feature available in HANA 2.0 SPS 6 and above.
Statement 2: Run by the application user SYSTEM, this activity is associated with HANA's internal optimize compression function, which is automatically calculated and optimized as part of the delta merge operation. Historically, it has been challenging to restrict the CPU consumption of this process. However, with the introduction in HANA 2.0 SPS03, the parameter indexserver.ini -> [optimize_compression] -> row_order_optimizer_threads can now be set to a specific value to moderate the concurrency of this activity effectively.
Statement 3: A select statement executed by user cxxx and application source axxx:3, it has been noted for its prolonged execution time and exceptionally high CPU usage. Thread sampling revealed that most threads were engaged in "IndirectScanVecOutJob<range>" or "ClusterScanBvOutJob<BV>", indicative of extensive table scans. More information on thread methods related to table scans can be found in SAP Note 2114710 - FAQ: SAP HANA Threads and Thread Samples

When look into the detail, we found that table MATDOC and MATDOC_EXTRACT, had lack of secondary index on the key column "MATNR" or "WERKS". Per seconds, the number of column scan can reach to 10 billions+, which is quite significant. Creating a secondary index on this key column should alleviate the CPU strain by optimizing the data retrieval process.

TABLE_NAME	COLUMN_NAME	SCANNED_RECORDS	SCR_PER_S
MATDOC	MATNR	828290874021485	19304757728
MATDOC_EXTRACT	WERKS	109487724317894	10032855641

Example 2 - System Hanging due to NUMA Contention

Sometimes we suffer system hanging situation, but seeing that the overall CPU consumption is not reach to its maximum capacity, or even lower than 50%. This may indicate that the short board is on the NUMA Node (CPU socket) side. The following example showing a NUMA contention analysis from high level, using thread samples. There are also tools to monitor the NUMA awareness from OS level, which gives more details, but thread sample could tell, in the first picture, which contributor may cause the contention issue.

The chart offers a detailed view of the running threads across NUMA nodes in a HANA system equipped with 8 sockets, each hosting 52 logical CPUs. Here’s a refined analysis of the patterns observed and their potential implications:

NUMA Node 4: The count of running threads frequently exceeds the threshold of 52, occasionally spiking to over 95. This pattern suggests excessive parallelism from specific statements or multiple statements accessing the same table or partition. These occurrences appear to bypass the constraints set by global parameters or workload classes, which either lack a total concurrency limit or have a single statement concurrency limit set too high. Such scenarios necessitate a review of the implicated statements to understand the cause of the surge in connections. Repartitioning tables may be beneficial, distributing data across partitions aligned with different NUMA nodes to balance the load. Moreover, the surges in thread count may be attributed to specific thread types like JobWorkers or SQLExecutors. A detailed statement-level investigation is warranted to identify and implement appropriate solutions.
NUMA Node 4: There’s a significant, consistent surge that extends for 30 minutes, signalling NUMA node contention. This is a critical observation as it suggests that the transactions competing for memory resources within this node are likely causing system performance degradation or even system hangs. This could be due to a variety of factors such as the lack of secondary indexes which forces full table scans, or database operations, e.g. optimize compression, that have not been sufficiently controlled. Implementing limitations on internal job concurrency, as prescribed by HANA's configuration settings, might serve as a remedial measure.
Across the remaining NUMA nodes, exceeding the 52-thread limit occasionally doesn’t necessarily indicate a problem; it may reflect the system making full use of the resources at its disposal, which is typically a positive scenario. Nevertheless, it is crucial to ensure that the workload is evenly distributed to prevent any single NUMA node from being overloaded while others are underutilized. Vigilant monitoring for NUMA contention is essential, particularly when the overall CPU capacity is not fully leveraged, to mitigate any adverse effects on system behaviour.

Reducing CPU Utilization on Primary Site by Using Active/Active Read Enable Features

Utilizing Active/Active Read-Enabled (AARE) in SAP HANA can swiftly reduce CPU and memory usage on the primary site. By rerouting read operations to a secondary site—through hints in SQL, ABAP, Procedure or CDS views—resource consumption can be balanced without altering existing job logic. Monitoring the secondary site is vital to ensure it doesn't affect system replication performance. For details on AARE, refer to the official SAP documentation.

🙉🙈🙊

Workload Analysis for HANA Platform Series

This blog post is part of the 'Workload Analysis for HANA Platform Series'. In upcoming posts, we will demonstrate how to analyze the issue related to CPU, threads and NUMA Node . Here's what you can look forward to in this series:

0. HANA Workload Analysis Overview
1. Define and Understand the Workload Pattern
2. Analyze the CPU, Threads and Numa Utilizations (This blog)
3. Identify the Memory Consumption
4. How the Network, I/O and locks influent the workload
5. Performance Impact by HANA Internal Functions
6. HANA Workload Management Methodology

Stay tuned as we explore these aspects in detail, providing insights and strategies to optimize your HANA environment.

Mastering SAP HANA Cloud Vector Engine for RAG-driven LLM Applications: An Insightful Guide

2024-04-08T18:23:37.926000+02:00

Introduction to Vector Similarity in RAG

Retrieval-Augmented Generation (RAG) techniques significantly improve the contextual relevance of large language model (LLM) outputs by incorporating real-time database lookups. SAP HANA Cloud vector engine plays a key role here, enabling efficient retrieval of pertinent data through vector similarity measures, thereby augmenting the performance of RAG tasks.

This tutorial provides a comprehensive walkthrough, using Python, to showcase the utilization of SAP HANA cloud vector engine in a RAG-based LLM application.

Setting Up: Connecting to SAP HANA

Start by establishing a secure connection to your SAP HANA instance.

from hdbcli import dbapi

# Establish a connection. Replace placeholders with actual credentials.
connection = dbapi.connect(
    address="HANA_DB_ADDRESS",  # Your database address
    port="HANA_DB_PORT",        # Typically 443 for HTTPS
    user="YOUR_USERNAME",       # HANA DB username
    password="YOUR_PASSWORD",   # Corresponding password
    charset="utf-8",            # Ensures correct encoding
    use_unicode=True            # Allows for Unicode characters
)

Creating an Embedding Table

For RAG to work, we need to store document vector representations. This involves creating a table in SAP HANA for embeddings.

cursor = connection.cursor()

# SQL statement to create an embedding table
create_table_sql = """
CREATE COLUMN TABLE embeddings_table (
    id INTEGER PRIMARY KEY,     -- Unique identifier for each document
    document NVARCHAR(5000),    -- Text content of the document
    embedding REAL_VECTOR(768)  -- Vector representation of the document
)
"""

# Executing the table creation SQL command
cursor.execute(create_table_sql)
connection.commit()
cursor.close()

Populating the Table with Document Embeddings

Before retrieval, our table needs populating with document embeddings. This involves defining documents, simulating embedding generation, and batch inserting these into SAP HANA.

Defining Documents

# A curated list of documents for embedding
documents = [
    "What is natural language processing?",
    "How do vector embeddings work?",
    "Examples of machine learning applications.",
    "Understanding deep learning for text analysis.",
    "The impact of artificial intelligence on society."
]

Generating Embeddings

Simulate LLM calls for embedding generation. In practice, replace this with actual model interactions.

def get_embedding(document):
    # Placeholder function to simulate embedding generation
    return [float(i) for i in range(768)]  # Returns a fixed-size (768) dummy vector

Batch Insertion into SAP HANA

Efficiently insert the document embeddings into the database.

cursor = connection.cursor()

# SQL command template for inserting document embeddings
insert_sql = """
INSERT INTO embeddings_table (id, document, embedding) VALUES (?, ?, TO_REAL_VECTOR(?))
"""

# Iteratively inserting each document and its embedding
for i, document in enumerate(documents, start=1):
    embedding_str = str(get_embedding(document)).replace(" ", "")
    # Execute insert command for each document
    cursor.execute(insert_sql, (i, document, embedding_str))

connection.commit()
cursor.close()

Executing Vector Similarity Searches

With the database prepared, perform a similarity search to find relevant documents for a given query.

Generating Query Embedding

First, generate an embedding for the query.

query = "What is an example of vector similarity search?"
query_embedding = get_embedding(query)

Conducting Similarity Searches

Use L2 distance and cosine similarity measures to find the most relevant documents.

cursor = connection.cursor()

# L2 Distance search
l2_query = """SELECT TOP 5 id, document FROM embeddings_table ORDER BY L2DISTANCE(embedding, TO_REAL_VECTOR(?))"""
cursor.execute(l2_query, (str(query_embedding).replace(" ", ""),))
l2_results = cursor.fetchall()

# Cosine Similarity search
cosine_query = """SELECT TOP 5 id, document FROM embeddings_table ORDER BY COSINE_SIMILARITY(embedding, TO_REAL_VECTOR(?)) DESC"""
cursor.execute(cosine_query, (str(query_embedding).replace(" ", ""),))
cosine_results = cursor.fetchall()

cursor.close()

The l2_results and cosine_results variables in the code snippet above contain the top 5 outcomes of similarity searches performed on the database, ordered from the most relevant to the least relevant.

Conclusion

This tutorial demonstrates how SAP HANA cloud's vector engine can be utilized in a RAG-based LLM application. The approach enhances LLM responses by ensuring that the generated outputs are informed by the most relevant data.

Reference

Official SAP HANA Cloud Vector Engine Documentation: https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-vector-engine-guide/sap-hana-cloud-sap-hana-database-vector-engine-guide

SAP Datapshereでの HANA Cloud, Data Lake の利用方法

2024-04-12T11:12:57.669000+02:00

はじめに

SAPは、インメモリ、カラムストアという優位なアーキテクチャを前提として設計された SAP HANA およびSAP HANA Cloud をご提供しています。この高速な SAP HANA Cloud のデータ処理機能を、Data Warehouse（DWH、あるいは情報系システムやOLAPと呼ばれるかもしれません）の用途でお使いいただくために、DWHの利用や運用に適した様々な機能をパッケージ化した製品としてSAP Datasphere を提供しています。SAP HANAのインメモリアーキテクチャはすべてのデータをメモリに配置する事で高速な検索処理能力を発揮しますが、一方で相対的にメモリはストレージ（HDDやSSD）よりも高価であり、大規模なデータを扱う場合にはコストも重要な課題になります。SAP HANA Cloudにおけるこのコスト最適化問題の解決策として、Native Storage Extension と HANA Cloud, Data Lakeという機能を提供していることを、過去のブログでご紹介してきました。今回のブログでは、これらの機能をSAP Datasphereで利用する方法についてご紹介いたします。

Native Storage ExtensionとHANA Cloud, Data Lakeについては、それぞれのブログ（HANA Cloud でデータの階層化を実現するNative Storage Extension（NSE)および HANA Cloud, Data Lake First step）をご参照いただくとして、ここでは概略をご説明します。

いずれの機能もSAP HANA Cloudの階層化データ管理を担うための機能です。データベースに格納されるデータは時間の経過とともに利用頻度が低下する傾向にあり、利用頻度の高いデータと、低いデータに対して同じコストをかけて維持するのは得策ではありません。利用頻度の高いデータは高い処理速度で、利用頻度の低いデータはコストに見合った処理性能の領域に格納する事で、コストとシステム性能のバランスを取る事ができます。

SAP HANA Cloud の Native Storage Extensionは、SAP HANAソフトウエアに組み込まれたデータ階層化機能で、「page loadable」という属性を与えた「データの単位」を対象に、データアクセスがなくなるとメモリから揮発し、次回のデータアクセス要求に応じてHANA Cloudのストレージ領域（永続領域）からメモリ領域に配置され、処理されることでメモリ領域のサイズを抑える事ができます。

SAP HANA Cloud, Data Lake はSAP HANAとは独立したData Lake relational engine にテーブルが作成され、HANA Cloud 上の仮想テーブルとして利用されます。Data Lake relational engineはインメモリアーキテクチャではありませんが、DWH用途には最適なカラムストアアーキテクチャを有しており、ストレージベースでありながら、（HANAには及ばないものの）高速にデータ処理を行えるエンジンとなっています。

図１ : 階層化機能　概要図

SAP DatasphereにおけるNative Storage Extensionの利用

前述のNSEに関するブログにあるように、HANA CloudではNSEを利用する対象テーブルに割り当てられたパーティションに対し、SQLコマンドによる操作で「Page Loadable」属性を与える事で、対象のパーティションのデータが階層化管理の対象として機能しました。

SAP Datasphereでは、この設定方法をより簡易に行う事ができる様、管理画面のGUIを使用して設定できるようになっています。一方で、階層化管理の対象の単位は、パーテイションではなく、テーブル単位となっています。また、デフォルトの設定はNSEの機能が有効なウォームストアへの格納となっています。Databuilder の画面から、該当のテーブルのプロパティを表示させ、「テーブルデータをメモリに保存」のスライドボタンを有効にして、NSEの機能を無効化して、パフォーマンスを上げる事ができます。

図２ : テーブルデータをメモリに保存（NSEを利用しない）設定

このことから、Datasphereのテーブルを設定する際は、２つの点を考慮して行っていただく必要があります。

テーブル単位で、アクセス頻度が高いか、低いか
アクセス頻度が高いテーブルであると判断される場合はデータビルダ画面の「テーブルサービス」の設定を「テーブルデータをメモリに保存」フラグをONにする

HANA Cloudでは、パーティション単位でNSEの機能が設定できるため、一つのテーブル（主に時間の経過とともにデータが増加するトランザクションデータのテーブル）に対して最近のデータと過去のデータを分けて階層化管理が可能でしたが、現時点ではSAP Datasphereではパーティションの作成はできますが、NSE機能の設定の粒度はテーブル単位となっていますので、テーブル単位でアクセス頻度が高いか、低いかを指針にして、NSEの機能を活用するようにして下さい。

SAP DatasphereにおけるData Lakeの利用

一方で、多くのお客様が、一つのテーブルのデータに対して階層化管理を行いたいというニーズをお持ちになると想定されますので、その際は以降でご紹介するData Lakeの機能をご検討下さい。本ブログの以下の手順で設定できる構成を以下に示します。

図３: 本ブログにおける Data Lake 構成例

図3では、SAP DatasphereのスペースAにあるメモリテーブルと、Data Lakeに配したストレージベースのテーブルを活用する場合の構成を示しています。この場合でも、オンメモリほどのパフォーマンスではないものの、常にオンラインで利用する事ができる様にData Lakeのデータへは仮想テーブルを介してアクセスできるようになっています。

Open SQLスキーマの利用

最初にご説明した通り、SAP DatasphereはHANA Cloudのデータベースを利用しており、通常のデータはSAP Datasphereから操作をする事で、テーブルなどのオブジェクトの管理が行われています。一方でData Lakeで利用されるテーブルは現在のところHANA Cloud から直接操作する必要があります。そのためにSAP Datasphereの Open SQL スキーマを利用します。図3で示されているように、アクティブに利用するデータは通常のスペースで管理され、Data Lakeに格納するデータは、Open SQL スキーマを介してHANA Cloud経由で利用する、という点にご注意下さい。Open SQLスキーマについての詳細はこちらをご参照ください。

Data Lakeの利用方法

最初に、Data Lakeの機能を利用するための準備設定を行います。左画面のメニューから「スペース管理」を選択します。Data Lakeの機能を使用するスペースの「編集」ボタンをクリックします。

図4：Data Lakeを使用するスペースの選択

続いてスペース管理の画面から「概要」タブが選択されている事を確認し、「データレイクアクセス」の項で「データレイクにアクセスするには、このスペースを使用します」のチェックボックスをチェックし、保存した後、デプロイをクリックします。

図5：Data Lakeアクセスするための設定

ここでの注意点は、Data Lakeにアクセスするスペースは、１つのインスタンスにおいて１つとなります。複数のスペースでData Lakeにアクセスする場合は、Data Lakeテーブルをそれぞれのスペースで共有するようにして利用します。本ブログでは、Data Lakeアクセス可能なスペースでの手順をご紹介します。

続いてOpen SQLスキーマを利用する準備を行います。

左端のメニューから「スペース管理」を選択し、「データベースアクセス」タブをクリックします。データベースユーザにある「作成」ボタンをクリックし、OPEN SQL スキーマにアクセスするためのデータベースユーザを作成します。これはSAP Datasphereとは異なる、HANA Cloudへアクセスするためのユーザになります。

図6: Open SQL Schemaにアクセスするためのデータベースユーザの作成

データベースユーザの作成画面では、「データベースユーザ名の接尾辞」を設定し、「読み込みアクセスを有効化」、「書き込みアクセスを有効化」をチェックして「作成」をクリックし、データベースユーザを登録します。画面が閉じたら、「保存」をし、続けて「デプロイ」を実行してください。

図7 : データベースユーザの登録

図8 : 保存とデプロイの実行

デプロイが完了し、ユーザが追加されたら、そのユーザのチェックボックスにチェックを入れ、行の右端にあるインフォメーションアイコンをクリックします。

図9 : 追加されたユーザの情報確認

データベースユーザ詳細画面が表示されますので、Open SQL スキーマパスワードを取得します。「新しいパスワードを申請」ボタンをクリックし、可視化ボタン（目玉アイコン）を押してから「パスワードのコピー」をクリックし、作成したユーザ用のパスワードを入手します。以後、この画面からは同じパスワードを入手する事はできませんので、必要があれば記録しておいてください。ただし、本シナリオの場合はログインの度に毎回新しいパスワードを生成しても特に問題はありません（ので記録し忘れても大丈夫）。

図10 : データベースユーザのパスワードの取得方法

取得したパスワードを使って、Open SQL スキーマの領域にアクセスします。前述しましたが、Open SQL スキーマとは、SAP Datasphereの管理下ではないHANA Cloud から操作する必要があります。そのため、SQLコマンドを使用することができる、データベースエクスプローラーというツールを介して操作します。作成したデータベースユーザにチェックを入れ、メニューにある「データベースエクスプローラを開く」をクリックします。

図11 : データベースエクスプローラの起動

ログインプロパティの入力画面に先ほど取得したパスワードを入力し、「OK」すれば、Open SQL スキーマを管理するHANA Cloudへのログインが完了します。

図12 : ログインプロパティの入力

ログインに使用したデータベースエクスプローラは、HANA Cloud用のSQL入力用のツールです。このツールを使用して、SAP DatasphereのOpen SQL スキーマで使用しているデータベース領域に直接SQLコマンドを実行する事ができます。以降の操作では、下の図のData Lakeにあるテーブルを作成し、このテーブルをOpen SQL スキーマにある仮想テーブルで参照する設定を行います。

図３: 本ブログにおける Data Lake 構成例（再掲）

まずData Lakeの物理テーブル（図中 ①　SALES_Datalake）を作成します。HANA Cloudに接続したDatabase ExplorerからData Lakeにテーブルを作成する場合 ”DATA_LAKE_EXECUTE" というシステムプロシージャを使用し、Data Lakeを管理するData Lake relational engineに対してSQL（Create table）文を間接的に実行させます。

call "DWC_GLOBAL"."DATA_LAKE_EXECUTE"('CREATE TABLE  SALES_Datalake(
Sales_ID INTEGER,
ShopID	INTEGER,
Purchase_Num	INTEGER,
Prod_ID	INTEGER,
Price	INTEGER,
Order_date date
)');

上記のSQLプロシージャ文を、Database ExplorerのSQLコンソール画面から実行します。左上にあるSQLコンソール起動アイコンをクリックし、実行してください。

図13 : Data Lake テーブル作成

次に、今作成したData Lake内のテーブルを参照する、Open SQL スキーマ内の仮想テーブル（図中②　V_SALES_Datalake)を作成します。仮想テーブルの作成は ”DATA_LAKE_CREATE_VIRTUAL_TABLE" というシステムプロシージャを使用します。

CALL "DWC_GLOBAL"."DATA_LAKE_CREATE_VIRTUAL_TABLE"
	( VIRTUAL_TABLE_NAME => 'V_SALES_Datalake',
      DATA_LAKE_TABLE_NAME => 'SALES_Datalake'
    );

Database Explorerから上記SQLプロシージャ文を実行すると、Data Lakeにある実テーブルを、SAP Datasphereから参照するための仮想テーブルが作成されます。実行完了を確認したのち、左上のペインにある「catalog」フォルダを展開し、「Table」アイコンをクリックすると、左下のペインに作成された仮想テーブルがリストされることが確認できます（図中にもあるWarningに関しては、本シナリオでは特に影響はありません）。

図14 : Open SQL スキーマ仮想テーブル作成と確認方法

作成した仮想テーブルに対して挿入および更新するための権限を SAP Datasphere のスペースに付与します。これにより、SAP Datasphereのデータフローによってテーブルにデータを挿入する権限を与えます。次の手順で、スペースにすべての権限を付与します（JAPAN_D&Aは実施した環境のスペース名です、適宜ご自身のスペース名に置き換えてください）。

GRANT ALL PRIVILEGES on "JAPAND&A#YAGAWA"."V_SALES_Datalake" 
to JAPAND&A with grant option

図15 : 仮想テーブルに権限付与

以上で、Data Lakeに物理テーブルを作成し、Open SQL スキーマにこれを参照する仮想テーブルが作成されました。それではSAP Datasphereからこの仮想テーブルを操作してみましょう。

Data Lakeに作成したテーブルに、SQL Open スキーマにある仮想テーブル経由でデータをインポートするため、ソースとなるCSVファイルを、AMAZON AWS S3に配置し、これをDatasphereのデータフローを使ってデータを格納します。AWS S3との接続設定に関しては、こちらのブログをご参照ください。このブログに基づき、Import_Location_from_S3 という接続設定を作成しておきます。また、接続先のAWS S3 には、インポートファイルをアップロードしておいてください。サンプルとして SampleData_Datalake2024.csv を添付します。

ブラウザのSAP Datasphereのタブを選択し左のメニューから「データビルダ」を選択、「フロー」タブをクリックし、「新規データフロー」タイルをクリックします。

図16 : AWS S3からCSVファイルをインポートするデータフローの作成

新規データフロー作成画面にて、左のペインから「ソース」タブをクリックし、「接続」フォルダを展開すると、上記で準備した Import_Location_from_S3がリストされます。これを展開すると、アップロード済のファイルがリストされます(下の画面では、AWS S3に作成したフォルダ dspimportlocation の下)。この中からData Lakeに配置するためのデータ SampleData_Datalake2024.csv　を、Drug ＆Dropし、中央のワークスペースに配置します。

図17 : AWS S3に配置したCSVをソースとして定義

つづいて「リポジトリ」タブをクリックし、作成した仮想テーブル「V_SALES_Datalake」をクリックし、ワークスペースにDrug & Dropします。この時同時に表示されるコンテキストメニューから「ターゲット」を選択します。

図18 : 仮想テーブルをターゲットとして定義

ワークスペースのソースオブジェクト（AWS S3ファイル）の黒い　●　を、ターゲット（V_SALES_Datalake）にDrug ＆ Drop で結線し、ターゲットオブジェクトをクリックしたのち右のペインのプロパティ画面を下方にスクロールして、ソースとターゲットの項目をマッピングします。こちらも左の項目をDrug ＆Dropすることでマッピングする事ができます。なお、ソースのCSVファイルの１行目にヘッダー行としてターゲットテーブルのカラム名を記入しておくと、自動マッピングする事も可能です。

図19 : ソースとターゲットの関連付け

設定したデータフローを左上のアイコン「保存」し（データフロー名を指定します）続いて「デプロイ」します。ワークスペースの中央下に「Deployed...」のメッセージが表示されたらデータフローの準備は完了です。このメッセージは自動的に消えますのでご注意下さい（しばらくすると右ペインのプロパティの表示が「未デプロイ」から「デプロイ済み」に更新されるので、こちらで確認可能です）。

図20 : データフローの保存とデプロイ

このデータフローにより、AWS S3にアップロードされたCSVファイルを、Data Lakeにインポートできるようになりました。このワークフローをスケジュールに基づき実行させる場合は、右ペインのプロパティの「実行ステータス」にある「スケジュール」アイコンをクリックし、スケジュールを作成する事ができます。ここでは、すぐにデータフローを実行したいので、左上にある「実行」アイコンをクリックし、データフローを実行します。

図21 : データフローのスケジュール実行と直接実行方法

データフロー完了のメッセージ（DIRECT run of DATA_FLOW .... completed このメッセージは自動で消えます）を確認するか、右ペインのプロパティにある実行ステイタスで「完了」を確認したら、ターゲットオブジェクトをクリックし、表示されるコンテキストアイコンから「データのプレビュー」をクリックすると、下方にこのオブジェクトのデータを確認する事ができます。このオブジェクトは仮想テーブルであり、その実態はData Lakeを参照しているので、これでData Lakeにデータをインポートしたことが確認できました。

図22 : データフローの実行確認と、その実行結果の確認

SAP Datasphereにおける Data Lakeの利用方法については以上となります。

SAP DatasphereにてData Lakeを利用したときのパフォーマンスに関する情報として、以下のブログが参考になりますので、リンクをご紹介しておきます。

Working with the Embedded Data Lake in SAP Datasphere

Why Don't We Use Data and Analytics Group on SAP Community?

2024-04-18T18:28:14.375000+02:00

Hey Data and Analytics Folks! Why Don't We Use Data and Analytics Group on SAP Community instead of Technology?

Of course, it does not matter at the end. As long as if you tag your posts with SAP Managed tags everybody can find your posts. But there is a really one huge difference for usability that here in Interests Groups we can reply-the-reply but under Products and Technology we cannot! Check out this blog post SAP Datasphere is ready to take over the role of SAP BW by @kenneth_dalvik and all responses that are very all boring sequential --there is no reply-the-reply option! 🙃

So let's start using this group Here is already available Data and Analytics. The group will be definitely a perfect fit for SAP Analytics Cloud SAP Datasphere SAP Analytics Cloud, hybrid analytics SAP HANA Cloud, SAP HANA database SAP Analytics Cloud, connectivity SAP Analytics Cloud, analytics designer SAP S/4HANA Cloud, extended edition SAP S/4HANA Embedded Analytics SAP Business Planning and Consolidation, version for SAP NetWeaver SAP Business Warehouse Accelerator BW (SAP Business Warehouse) SAP BW/4HANA BW SAP BEx Analyzer BW SAP BEx Web SAP ASE - BW Enablement BW SAP HANA Data Warehousing SAP Data Services SAP Data Intelligence SAP Data Custodian SAP Big Data Services SAP BusinessObjects - Platform Administration SAP BusinessObjects - Semantic Layer SAP BusinessObjects Mobile SAP BusinessObjects - Semantic Layer - SDK SAP BusinessObjects - Web Intelligence (WebI)

These are welcome too SAP Master Data Governance SAP NetWeaver Master Data Management SAP Data Quality Management for SAP NetWeaver MDM

One thing I've noticed we cannot submit the blog posts directly. We need to "Submit for Review" only. You can still post discussions aka questions directly.

Thanks owners @ColinC, @meganhoy, @moshenaveh, @caroleighdeneen, @craigcmehil for their support!

Regards,
Tuncay Karaca

Understanding Data Modeling Tools in SAP

2024-05-02T16:25:43.459000+02:00

Choosing the right SAP data modeling solution can be a challenging task, as there are multiple options available, including cloud-based or on-premise solutions with or without data visualization capabilities etc..so to select the best tool that can meet the project requirements and specifications, several factors need to be considered including, reviewing the raw data source and understanding how it will be used, assessing the need for data warehousing capabilities, evaluating the team's skill sets, and deciding whether a cloud or on-premise solution is preferred.

During the last few years, I've worked with almost all the SAP Data Modeling tools that are available, and In this blog article, I would like to briefly highlight the strength of each tool and where would be the best fit for it 🤗.

Understanding Data Modeling

so what it's data modeling in SAP and why it's important? 🤔

when we talk about Data modeling we are defining the process of creating diagrams that show how data flows in and out of a database. These diagrams define the data formats, structures, and functions required to support the data flow. Once the database is built, the data model becomes documentation for its existence and design. Data models provide a framework of relationships between data elements within a database. They enable different applications to share the same data and create a simple and logical database that eliminates redundancy, reduces storage requirements, and enables efficient retrieval.

SAP HANA Calculation Views

SAP HANA calculation views are the key data modeling object on HANA, it's a graphical object that describes all the data processing steps from raw data to the final, consumable output.

Common data processing steps include filtering data, combining data, aggregating data, calculating new values, converting values, and ranking data. Calculation views are built in layers so that each calculation view can consume other calculation views to form a stack of super-models. This approach promotes the reuse of calculation views at the lower levels.

Strengths:

In-Memory Processing: Data processing occurs directly in memory, ensuring high performance.
Graphical Interface: Intuitive tools make it easy to create and visualize models.
Custom Logic: Ability to integrate custom SQL for tailored data processing.
Advanced Analytics: Supports spatial, graph, text analytics, and predictive/machine learning.

Best Fit: Developing flexible data marts with custom logic requirements.

SAP BW/4HANA

SAP BW/4HANA is an on-premise data warehouse. The main purpose of a data warehouse is to extract data from any data source, integrate the data, store the cleansed data, and provide modeling tools so that data models can be created ready for use by any analytical tool. This is exactly what SAP BW/4HANA can do.

Strengths:

Ready-Made Models: Comprehensive data models are provided for virtual and physical storage.
Data Warehousing Features: Supports dimensional modeling, hierarchies, and efficient data storage.
Graphical Tool: User-friendly interface for efficient model visualization.
On-Premise Deployment: Ideal for industries requiring high security and data control.

Best Fit: Establishing on-premise data warehouses with strong data integration needs.

ABAP CDS Views

ABAP CDS Views enable developers to create data models that are semantically rich. These models are then exposed to UI clients by the application services. CDS Views is the central pillar of S/4HANA development and is used as the core technology in most of SAP's programming models.

Strengths:

High Performance: Data processing is pushed down to the database level, ensuring efficiency.
Ready-Made Models for SAP S/4HANA: Simplifies operational reporting on S/4HANA.
Flexible Development: Allows for code-based customization leveraging SQL skills.

Best Fit: Operational reporting on SAP S/4HANA with ready-made content.

SAP Analytics Cloud

SAP Analytics Cloud is an end-to-end cloud solution that brings together business intelligence, augmented analytics, predictive analytics, and enterprise planning in a single system. It is a solution aimed at business users for self-service analytics.

Strengths:

Integrated Modeling: Modeling seamlessly integrated with reporting tools for streamlined analytics.
Cloud-Based: No need for software installation; data modeling is conducted in the cloud.
Self-Service Analytics: Empowers business users to create their own models.

Best Fit: Organizations seeking self-service analytics with integrated modeling and reporting.

SAP Datasphere

which is so far My Absolute favorite tool 😁! It is a public cloud software as a service (SaaS) data warehouse that supports the business data warehouse architecture. This tool effortlessly harmonizes data across the enterprise, making it a must-have for any business. It includes services for data modeling, data integration, cataloging, and data warehousing across SAP and non-SAP data, making it a one-stop-shop for all data management needs

Strengths:

Comprehensive Data Warehouse Features: Supports dimensional modeling, data history management, and hierarchy modeling.
Scalability: Easily scale resources in the cloud, supporting new innovations.
Cloud Service: Simplifies software management through cloud consumption.

Best Fit: Cloud-based data warehousing with advanced features and scalability.

SAP BusinessObjects BI Semantic Layer (Universes)

An organized collection of objects that called a Universe, which consists of dimensions, measures, and attributes grouped by business topics. The skill of a Universe developer is to create objects that resemble business terms by mapping them to the columns of the database tables. In some cases, SQL coding may be required to generate the value of the objects.

Strengths:

User-Friendly Interface: Graphical tool simplifies data modeling tasks.
SQL-Based Development: Easy to understand for database developers.
Mature Tool: Offers a wide range of features for diverse modeling needs.

Best Fit: Supporting SAP BusinessObjects reporting tools with Universes.

for more information, I would recommend this Learning journey from SAP which will give you an overview of all the SAP data modeling solutions 😊

Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive

2024-05-08T17:00:00.007000+02:00

Introduction

This blog post is part of a series describing the usage of MLflow with HANA Machine Learning co-authored by @stojanm and @martinboeckling. In this blog post we provide a more technical deep dive on the setup of a MLFlow instance and provide a general introduction how Machine Learning models trained with HANA ML can be logged with MLflow. The first blog post of the blog series is called Tracking HANA Machine Learning experiments with MLflow: A Conceptual Guide for MLOps and gives an introduction to the topic of MLOps with MLflow.

Starting with the python HANA ML package version 2.13, HANA Machine Learning added support for tracking of experiments with the MLflow package, which makes the incorporation of models developed using HANA Machine Learning into a comprehensive MLOps pipeline easy to achieve.

In this blog post we will provide an overview how MLflow can be used together with HANA ML. MLflow, which manages the experiment tracking and artefact management can run as a managed service at a hyperscaler platform, deployed locally or on a remote infrastructure. In the following we describe how to deploy MLflow on SAP Business Technology Platform and how to track your HANA machine learning experiments with MLFlow. In addition, we present which methods and algorithms in the hana ml package currently support the experiment tracking feature. Finally, we touch on the possibility to use logged models in MLflow for prediction.

Prerequisites

In this blog post we solely focus on the technical integration of HANA ML and MLflow as a logging platform. Generally, we assume that Python is already installed together with an already established development environment. Furthermore, we will not completely explain all details of docker and Cloud Foundry, but simply focus on the essential parts for HANA ML and MLFlow within this blog post.

Set up MLFlow on BTP

MLFlow is leveraged and integrated in different solutions. For example Databricks as well Machine Learning in Microsoft Fabric integrate or provide a managed MLflow instance. In case MLFlow is not yet provided, we outline in this section a possibility to deploy MLflow in SAP BTP. We focus for simplicity reasons on the SQLite based deployment of MLflow. However, for productive environments it is recommended to separate the storage from the runtime of the MLFlow instance. A detailed explanation for setting up your own MLFlow server with alternatives to SQLite can be found under the following link: https://mlflow.org/docs/latest/tracking/server.html. In the following paragraphs, we focus on a step by step overview to set up your own MLFlow instance using SQLite in BTP.

As a first step we want to create a local docker file which can be used to upload it to our BTP environment. In the following code snippet, we provide the coding used to construct your own docker container locally. For that, paste the following code into a file called Dockerfile within your desired local folder.

# Retrieve a python version as a base runtime for our docker container
FROM python:3.10-slim
# Run the pip install command for the package of mlflow
RUN pip install mlflow
# Create a temporary folder within our docker container to store our artifact
RUN mkdir -p /mlflow
# Expose the port 7000 to make our application which runs within docker accessable over the defined port
EXPOSE 7000
# Define environment variables BACKEND_URI and ARTIFACT_ROOT to define the backend uri as also the 
# artifact root
ENV BACKEND_URI sqlite:///mlflow//mlflow.db
ENV ARTIFACT_ROOT /mlflow/artifacts
# run the shell comand to setup our mlflow server within our docker container
CMD mlflow server --backend-store-uri ${BACKEND_URI} --host 0.0.0.0 --port 7000

After creating successfully the Docker object, you can run docker build -t {tagname}. to construct your docker container. Afterwards, the local docker image is locally built. To expose the docker image, we push the image to a docker registry. In our example, we assume that you already have a docker registry set up where you can push your image to. For that step, you can run the following commands: docker tag {tagname} {dockerhub repository tag}, docker push {dockerhub repository tag }. After the successful run of the command, you see within your private docker hub the newly published docker container, which contains MLflow and all its dependencies inside of it.

After the successful publishing of your docker image to your registry, we can run the following command to create a BTP app based on the published docker image:

cf push APP-NAME --docker-image REPO/IMAGE:TAG

After successfully publishing the docker image on our BTP Cloud Foundry environment, we can find our published app within our BTP account and are able to access it under the published URL.

Set up tracking for MLflow

With MLflow users have the possibility to track their trained HANA ML models. In the following paragraph, we introduce the aspects that are needed to be able to log HANA ML models into MLflow itself.

To be able to use MLFlow together with HANA ML, we need to first install besides the HANA ML package also the MLFlow package. Therefore, you need to run the following command in your virtual environment, to be able to run the following scripts.

pip install mlflow hana-ml

As a general setup, we first need to run the following command to set up our tracking with MLFlow to our available MLFlow instance. Therefore, place into the following two lines first your personal MLFlow tracking URI and your own custom experiment. In case you do not want to create a separate experiment, the different runs together with the MLFlow model are stored under the default experiment.

The method that allows us to track HANA ML models is implemented in the HANA ML package and is called enable_mlflow_autologging(schema=None, meta=None, is_exported=False, registered_model_name=None). This method can be used for initialised HANA ML models that are under the following methods:

Within the method enable_mlflow_autologging the user has different keywords that can be filled that allows us to influence the behaviour of our MLFlow autologging in HANA ML.

schema: Defines the HANA database schema for MLFlow autologging where the MLflow logging table is stored
meta: Defines the name of the model storage table in HANA database
is_exported: Determines if the hana model binaries should be exported to MLflow
registered_model_name: Name of the model stored in MLflow

In the following section we provide an overview for the Unified Interface how the logging of MLflow can be used.

Run HANA ML Algorithms with MLflow

As we have explained and outlined in the sections above, we have created a MLFlow instance and have introduced the syntax that is needed for the logging of HANA ML models in MLflow. In the following sections we will provide based on an example how the logging of HANA ML models on MLflow is done.

Model training of HANA ML with MLFlow

For the training of HANA ML in combination with MLflow, we focus in this blog post on the Unified Method. We apply for the respective elements a Classification on the sample bank dataset which can be found under the HANA ML sample dataset folder on GitHub.

The dataset can either be uploaded directly to the SAP HANA database or you could also use SAP Datasphere as your starting point. Generally, to use HANA ML directly you would need to store the dataset in a HANA database. However, HANA ML also provides methods to integrate third party files/ data structures. This involves Pandas, Spark as also shapefiles. In addition also HANA Data Lake file tables can be integrated with HANA ML functionalities. An overview of the different methods can be found under the following page. In the following paragraphs, we will go through the sample code that we have created to combine HANA ML and MLFlow.

Connect to HANA database (Deployed under SAP Datasphere)

To be able to connect to the HANA database instance, we first need to build up a connection to the HANA database. In our example, we load the data from the data samples provided by HANA ML. During the time of this blog post, the OpenSQL schema of Datasphere only supports Basic Authentication. Therefore, in this blog post we only elaborate how the connection is done over basic authentication. SAP HANA standalone supports however non-basic authentication, which are also supported in the HANA ML package to connect certificate based to the SAP HANA instance.

To establish the connection to the HANA database, we make use of the implemented HANA ML dataframe class and call the method ConnectionContext. We store the instance of the connection in the variable conn. To now be able to establish the connection to the HANA database view or table, we will need to specify over the method table the connection. The beautiful aspect is that overall, the dataset is not going to be loaded to the Python runtime, but will only be represented with a proxy to the actual table in the HANA database. All transformations, if done over the methods of HANA ML, are then pushed down to the database itself and executed there if the training is executed. In our case, we load the sample dataset into our database by making use of the provided methods of the HANA ML package.

"""
This script provides a short example how HANA ML and MLFlow can be integrated together.
The credentials to the database are following the currently supported authentification (Basic) of the 
SAP Datasphere OpenSQL schema. Overall, HANA Cloud standalone is also able to support multiple other authentification
methods. We have used an abstraction python file (constants) where we retrieve the securely stored 
authentification properties. To get more details about the exact method structure needed, please
have a look at the documentation:
https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.07/en-US/hana_ml.dataframe.html#hana_ml.dataframe.ConnectionContext
"""
from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.algorithms.pal.auto_ml import AutomaticClassification
import mlflow
from hana_ml.algorithms.pal.auto_ml import Preprocessing
from hana_ml.algorithms.pal.partition import train_test_val_split
from constants import db_url, db_user, db_password
# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url, port=443, user=db_user, password=db_password)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

After connecting to the database, the user is able to use the preprocessing methods implemented in HANA ML. Generally, the different changes are pushed down to the HANA database and are not executed within the Python runtime. In our use case, we do not need to use the data preprocessing as we directly retrieve a sample dataset which we can directly use for our ML training.

After finishing the potentially needed transformations, we are now able to implement the tracking of our HANA ML runs with the possibility of MLFlow. Similar to the normal usage of MLFlow, we set up first our tracking uri under which we want to store our HANA ML runs and models. In your case you would need to change the keyword mlflow_tracking_uri with your respective MLflow tracking URL. Furthermore, we then are able to specify the experiment name under which the runs are tracked. If we do not specify a specific experiment, the runs are tracked under the Default experiment.

# set up MLFlow
mlflow.set_tracking_uri(mlflow_tracking_uri)
mlflow.set_experiment("HANA ML Experiment")

In the following chapters, we will provide an outline how the exact training is performed and what components are logged to MLflow.

Unified Method

For the example we use the implemented Hybrid Gradient Boosting Tree as a classification algorithm for our Classification. In order to perform the classification, we use the Unified Classification in order to be able to run our algorithm. On the defined variable, we then use the implemented enable_mlflow_autologging method. This allows us to directly log the model using implemented auto logging behaviour.

uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging() uc.fit(training_data, key="ID", label="responded")

We call the fit method once we have initiated the HANA ML model variable and the associated autologging for MLflow. For the fit method, we have in total two different options. Firstly, the non-partitioned training dataset where we only use the training dataset. If we decide to partition our training dataset, we allow to create a validation dataset for which we can log metrics automatically during training.

If we do not define for our fit function the partitioning, we will not log metrics within MLFlow. In the following image, you can see how a potential HANA ML tracked run looks like in MLFlow together with the stored HANA ML model in MLflow.

If we decide to partition our dataset, here for instance to partition the dataset along the defined primary key, we are able to directly log evaluation metrics relevant for the Classification we have used. This includes the following metrics: AUC, Recall, Precision, F1 Score, Accuracy, Kappa coefficient and the Mathews Correlation Coefficient (MCC). This would directly allow us to compare multiple runs within our MLFlow project to one another and measure the different performances.

In addition to the general run, HANA ML also logs the model to MLflow. What is logged to MLflow depends on the parameters set for the method enable_mlflow_autologging. If for instance everything is set to the default settings, we will see the following yaml file to be logged to MLflow.

If within the method enable_mlflow_autologging the parameter is set to is_exported, the model binaries stored in the model storage on HANA are exported to MLflow. This setting would allow us to retrieve the trained model from MLflow and use it in a different HANA database for prediction purposes. In addition to the yaml file containing the metadata we now can see a created subfolder called models which contains the necessary model artefacts normally stored in the HANA database now in MLflow.

After the training is finished, we have besides the auto logging capabilities of HANA ML for MLflow the possibility to track further artefacts in MLflow. In the following section we will outline a few possibilities that exist with the additional tracking.

Additional logging possibilities

Besides the outlined auto logging capabilities, we can track with MLFlow additional artefacts to the respective run. In the following chapters, we outline selected possibilities to further enrich the auto logging for HANA ML runs tracked in MLFlow.

Adding run and experiment description

The description in the experiment section can be handy once the number of your experiments grows in the repository. In addition, mlflow allows to also add individual description to each run of an experiment. Using the following methods you can set up both:

from mlflow.tracking import MlflowClient
current_experiment=dict(mlflow.get_experiment_by_name("HANA ML Experiment"))
experiment_id=current_experiment['experiment_id']
    
run = mlflow.active_run()
MlflowClient().set_experiment_tag(experiment_id,"mlflow.note.content",
"This experiment shows the automated methods of HANA machine learning and how to track them with MLFLOW")
MlflowClient().set_tag(run.info.run_id, "mlflow.note.content", "This is a run tracked with Unified Classification from HANA Machine Learning")

Logging input datasets

Sometimes it is important to keep the input dataset also as part of the tracking with MLflow. Since HANA machine learning datasets are located in HANA, they need to be converted to pandas DataFrames to be tracked as shown in the following code:

# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

This results in the change, that the respective state of the training data is logged to the current run. The logged dataset can be found in the associated MLflow run, where the schema of the dataset is provided together with some metadata information about the number of rows and number of elements. In addition, also the provided context is marked in the UI of MLflow.

Logging a model report

In addition to the logging of the dataset, it might also be important to add a model report to MLFlow. HANA ML generally provides different interactive visualisations for the trained model artefact, which can be stored as an HTML file. After the storing of the model report to your local repository, we can log the input of the model report to our current run. This allows us to interactively explore the model report automatically generated by HANA ML and make it accessible in MLFlow. To log the HANA ML model report, you can use the following code snippet.

# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")

After the Model report is stored successfully under the current run, we can see in the artefact tab in MLFlow the interactive model report:

The complete script used for this section can be found in the following code snippet:

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLflow
mlflow.set_tracking_uri(tracking_uri)
mlflow.set_experiment("HANA ML Experiment")
# set up classification
uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging(is_exported=True)
# train model
uc.fit(training_data, key="ID", label="LABEL", partition_method="stratified", stratified_column="ID", partition_random_state=43, build_report=True)
# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")
# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

Apply of trained model

After we have finished our training, we are able with HANA ML to retrieve the model from MLFLow and use it for our prediction purposes. For this purpose, we will create a separate Python script where we will provide an overview to retrieve the trained MLflow model.

Similar to our training script, we first set up our connection to the HANA database and establish the connection to our table. In our case, we simply use the sample dataset provided by HANA ML.

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

Similar to our training script, we need to set the tracking url for MLflow and need to initiate the model storage of HANA. If we have decided to not export the HANA ML model to MLflow, we need to specify the same schema for the model storage where our HANA ML model is stored after the successful run. In case we have exported our model, we are able to specify a different schema. In the following, you can see the necessary script in order to retrieve the logged HANA ML model from MLflow.

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_url)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)

After the model storage has been initiated, we are able to retrieve the stored HANA ML model from MLflow. In order to select the correct model, you need to extract the correct run id associated to the model you would like to apply for your prediction dataset. In our case, this is the test dataset we have received from the sample dataset method. The model_uri needed for the model retrieval is consisting of the following pattern 'runs:/{run id}/model', in which you would need to exchange the run id with your respective run. For the actual retrieval of the model, we use the initiated model storage, in our case called model_storage and call the method load_mlflow_model to load the MLflow model to our HANA database and assign the respective proxy to our variable mymodel. The variable mymodel is then used to call the predict method in order to apply our model to our dataset. In the end we transform our prediction dataset into a Pandas DataFrame to look at the content of the created DataFrame. Normally, we could directly persist the created temporary table with the save method and therefore make the dataset available for further processing.

# load logged run from MLflow to HANA ML
logged_model = 'runs:/d8a763b7b81940598633605e447cd880/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

After running the script, you should be able to see the following terminal output, for which we can see the download of the artefact stored in MLflow and the created prediction dataset, which consists in our case of 4 columns: ID (primary key), SCORE (predicted label), CONFIDENCE (prediction confidence for applied row) and REASON_CODE (influence of individual variables to prediction output).

In case we have exported our model, the output of our terminal look slightly different indicating that we also download the respective model artefacts stored additionally to the yaml file. In the following you see the complete script used for applying the model to a new dataset.

from hana_ml import dataframe
from hana_ml.model_storage import ModelStorage
from hana_ml.algorithms.pal.utility import DataSets
import mlflow
from constants import db_url, db_user, db_password

conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

# full_set, diabetes_train, diabetes_test, _ = DataSets.load_diabetes_data(conn)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_uri)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)
# load logged run from MLflow to HANA ML
logged_model = 'runs:/ed7b8d4734cb42ca90c417f932957b40/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

Key take aways

In this blog post we have showcased an end to end example how MLflow can be integrated in your HANA ML workload by providing the possibility to share and compare multiple tracked runs in MLflow. If the data is already stored in HANA, this allows you to directly interact with MLflow while being able to run your Machine Learning algorithms on data stored in the HANA database without the need to transfer your data between multiple systems. This blog covered an essential part of the automated logging capabilities of HANA ML models into MLflow.

We highly appreciate your thoughts, comments and questions under this blog post. In case you want to reach out for general questions around HANA, or specifically HANA ML, don't hesitate to use the Q&A tool with the respective tags that describe your question.

SAP Community - SAP HANA

Advance shipment via Procurement

SAP Landscape Transformation Replication Server - Book Review

In this blog post you will find the Book Review of SAP Landscape Transformation Replication Server (SLT)

About the BookHere is the book which is published by SAP Press about SAP SLT.

The authors of the book are

What's the latest and greatest book about SAP SLT?

Where can you get the book?

Global Explanation Capabilities in SAP HANA Machine Learning

Permutation feature importance

Case study: impurity-based feature importance

Case study: SHAP feature importance

Summary

Protecting Your SAP HANA Cloud and SAP HANA with Credential Rotation Best Practices

Workload in the SAP HANA

Options for Managing Workload

1.Controlling CPU Consumption

5. Managing Workload with Workload Classes

Troubleshooting: Removal of deleted Shipment POs from MRRL Batch jobs

Custom Fields and Logics - Create and Capturing the Changes into Transport Request

Using Hints with SQLScript procedures

Introduction

Demo environment setup

Applying hints to inner SQL statements of SQLScript procedure

Conclusion

Effectively adopting Recurring Service process from S/4 HANA Service Management into your business

How Data Anonymization in SAP HANA secure Generative AI applications through SAP Datasphere

Taking Data Federation to the Next Level: Accessing Remote ABAP CDS View Entities in SAP HANA Cloud

Prerequisites & considerations

Getting started with SDA ABAP adapter (abapodbc)

Query execution and execution plan

Other limitations

FAQs

Conclusion

Product Updates for SAP Business Application Studio - March 2024 Edition

Enhancing Productivity with our Updated Storyboard

New Welcome Page for SAP HANA Applications

SAPUI5 Adaptation Projects for SAP S/4HANA Cloud Public Edition

SAPUI5 Walkthrough Tutorial with TypeScript

Introducing UI5 linter

Safeguarding Data Privacy in KSA - Leveraging SAP to navigate NDMO’s Regulations in Digital Era

Elucidation of Associations in CDs Views

Workload Analysis for HANA Platform Series - 2. Analyze the CPU, Threads and Numa Utilizations

Mastering SAP HANA Cloud Vector Engine for RAG-driven LLM Applications: An Insightful Guide

Introduction to Vector Similarity in RAG

Setting Up: Connecting to SAP HANA

Creating an Embedding Table

Populating the Table with Document Embeddings

Defining Documents

Generating Embeddings

Batch Insertion into SAP HANA

Executing Vector Similarity Searches

Generating Query Embedding

Conducting Similarity Searches

Conclusion

Reference

SAP Datapshereでの HANA Cloud, Data Lake の利用方法

はじめに

SAP DatasphereにおけるNative Storage Extensionの利用

SAP DatasphereにおけるData Lakeの利用

Open SQLスキーマの利用

Data Lakeの利用方法

Why Don't We Use Data and Analytics Group on SAP Community?

Understanding Data Modeling Tools in SAP

Understanding Data Modeling

SAP HANA Calculation Views

SAP BW/4HANA

ABAP CDS Views

SAP Analytics Cloud

SAP Datasphere

SAP BusinessObjects BI Semantic Layer (Universes)

Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive

Introduction

Prerequisites

Set up MLFlow on BTP

Set up tracking for MLflow

Run HANA ML Algorithms with MLflow

Model training of HANA ML with MLFlow

Connect to HANA database (Deployed under SAP Datasphere)

Unified Method

About the Book
Here is the book which is published by SAP Press about SAP SLT.