SAP Community - SAP Datasphere

SAP Datasphere News in April

2024-05-03T10:45:11.774000+02:00

April was for real 💯 with regards to great new features for SAP Datasphere. There are also lots of excellent new blogs in the community and in case you missed it, our What’s New in SAP Datasphere session covering Q1 just ran in early April.

What’s New in SAP Datasphere Q1 2024 Webinar

We just ran our Q1 What’s New in SAP Datasphere session on April 10th 2024. No worries, in case you missed it. The session covering the features and functions delivered in Q1 2024 was recorded and is available on YouTube.
Watch the recording 👇

My top features in April

➡️ Confluent Kafka as Target for Replication Flows

After delivering the native Apache Kafka connection already back in November 2023, you can now use Confluent Kafka and the corresponding connection type to add targets to replication flows.
For more information: Using Confluent Kafka As the Target and Integrating Data via Connections

➡️ Cataloging offers ability to add an SAP Datasphere, SAP BW bridge as a Source System

You can add an SAP Datasphere, SAP BW bridge system to the catalog and have the metadata objects monitored and extracted to the catalog. Once extracted, you can publish these objects as assets to the catalog and use them in the impact and lineage diagram.
For more information: Connecting to Data Sources and Evaluating Catalog Assets

➡️ Measure Type Non-Cumulative Measure for Analytic Models

You can now create non-cumulative measures for analytic models. This type of measure is used for cases where measure values describe a state that is not changing every day. For example, you can use non-cumulative measures to calculate the stock in a warehouse, the bank balance of an account or certain values in HR use cases.
For more information: Create a Non-Cumulative Measure .

➡️ Add a Visual Tenant Type Indicator

When you are working with multiple tenants you can now add a visual tenant type indicator to distinguish which system you are using. For example, between a test or production system. When enabled, a colored information bar is visible to all users of the tenant, and the browser favicon is updated with the matching color.
For more information: Display Your System Information

Blogs for our Japanese Community

My top blogs in April

Excellent blog from @AlexLePape outlining the integration of SAP Datasphere with the Customer Data Platform (CDP) through Direct Data Access
➡️ Unlocking the Power of Customer Data – SAP Customer Data Platform and SAP Analytics Cloud.

@ClaudiaFiess started a blog series about SAP extractors and SAP Datasphere with many great details on the different scenario options:
➡️ Exploring Integration Options in SAP Datasphere with the focus on using SAP extractors
➡️ Exploring Integration Options in SAP Datasphere with the focus on using SAP extractors - Part II

Another favorite in April is @jaigupta excellent deep dive into currency conversion and the derivation of variables
➡️ Using Variable derivation for currency conversion measures within Analytic Model

The last blogs I’d like to highlight is about the command line interface. @Yves_Kipfer checks on the new feature of CRUD operations
➡️ Use CRUD Operations with SAP Datasphere Command-Line-Interface

More blogs to check out

Find more information and related blog posts on the topic page for SAP Datasphere. You will find further product information on our Community with various subpages about Getting Started, Business Content, the SAP BW Bridge as well as content for Best Practices & Troubleshooting and the FAQ for SAP Datasphere.

Find out how to unleash the power of your business data with SAP’s free learning content on SAP Datasphere. It’s designed to help you enrich your data projects, simplify the data landscape, and make the most out of your investment. Check out even more role-based learning resources and opportunities to get certified in one place on SAP Learning site.

Create SAP S/4HANA External GL Account Hierarchy within SAP Datasphere using standard CDS Views

2024-05-07T09:51:03.951000+02:00

1. Introduction

This blogpost is inspired by this Blogpost (https://community.sap.com/t5/technology-blogs-by-sap/guide-create-sap-s-4hana-external-gl-account-hierarchy-within-sap/ba-p/13574648). It is described how you can create an SAP S/4HANA External GL Account Hierarchy within SAP Datasphere.

However, sometimes it might be not possible to use the community content package, therefore we want to hand you a guide which shows you how the hierarchy is being set up. If you don’t have any deviations, we still recommend using the community package, since it will save you a lot of time and effort. Please be aware that this logic can be used for differing use cases! Even with the data changed, the logic will remain the same.

First, you need to set up a connection between your S/4HANA Cloud System and your space in SAP Datasphere. There will soon follow a guide series provided by my colleague Simone and me. I will keep this blog post up to date and add this link as soon as our series started

You should know some basics of SAP Datasphere (build views in general and create an replication flow).

2. Basic Logic and Setup

We will build the hierarchy and the final view of a balance sheet with the G/L hierarchy applied to it. Please find this logic applied to it. It is the same from the community content. The purple arrows represent the associations.

Therefore, we will need the following CDS Views which will be pulled via Replication Flow. In this step we will create new target tables. These Local Tables remain with the same name as the original CDS views. You can choose existing target tables as well if you have them, but I assume you are starting from scratch.

For the views, we will use different business and technical names. We sticked with the naming convention of the original blogpost with the community content to not create confusion.

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTHIERARCHY	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHY	HRF: G/L Account Hierarchy Directory (DIM)	Local Table for GL Account Hierarchy View
I_GLACCOUNTHIERARCHYTEXT	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYTEXT	HRF: G/L Account Hierarchy Directory - Text (IL)	Local Table for GL Account Hierarchy Text
I_GLACCOUNTHIERARCHYNODE	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODE	HRF: G/L Account Hierarchy Node (IL)	Local Table for GL Account Hierarchy Node
I_GLACCOUNTHIERARCHYNODET	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODET	HRF: G/L Account Group (Hierarchy Node) - Text (IL)	Local Table for GL Account Hierarchy Node Text
I_GLACCOUNTINCHARTOFACCOUNTS	SAP_CC_FI_HRF_IL_I_GLACCOUNTINCHARTOFACCOUNTS	HRF: G/L Account (DIM)	Local Table for GL Account Master data Attributes
I_GLACCOUNTTEXTRAWDATA	SAP_CC_FI_HRF_IL_I_GLACCOUNTTEXTRAWDATA	HRF: G/L Account – Text (IL)	Local Table for GL Account Master data Text

In addition to these, you should have a view containing your transaction data. In our example we have a custom CDS View with Journal Entries. We are aware that there are Standard CDS Views that contain the same data, but in our case, we had to add some measures.

After deploying and running your replication flow, you should now have several Local Tables. Within the Data Builder you can filter after them using the top row.

Starting from the bottom, we will first create the hierarchy. Afterwards we will build the dimension to which the hierarchy is applied to. This dimension is our master data, which will be associated with our transaction data (Fact View). Based on the Fact View, we will create our Analytic Model to make our data available for consumption in SAP Analytics Cloud.

3. Building the Hierarchy with Directory

3.1 View (Text) for GL Account Hierarchy Text

We will first build this Text View:

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTHIERARCHYTEXT	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYTEXT	HRF: G/L Account Hierarchy Directory - Text (IL)	GL Account Hierarchy Text

Create the view and adjust the following Colums:

ValidityEndDate with the expression TO_DATE(ValidityEndDate)
ValidityStartDate with the expression TO_DATE(ValidityStartDate)

In the newly created View (Text) change the semantic type from “relational dataset” to “Text”.

Work on the Semantic Types for the Attributes accordingly.

There is no need to set a representative key in this step.

3.2 View (Dimension) for GL Account Hierarchy View

Next up is the related Dimension:

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTHIERARCHY	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHY	HRF: G/L Account Hierarchy Directory (DIM)	GL Account Hierarchy View

Create the view and adjust the following Colums:

ValidityEndDate with the expression TO_DATE(ValidityEndDate)
ValidityStartDate with the expression TO_DATE(ValidityStartDate)

In the newly created View (Dimension) change the semantic type from “relational dataset” to “Dimension”.

Create an Association between HRF: G/L Account Hierarchy Directory - Text (IL) (SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYTEXT) and the newly created Dimension.

Work on the Semantic Types for the Attributes accordingly. Please ignore the error warning.

There is no need to set a representative key in this step.

3.3 View (Text) for GL Account Hierarchy Node Text

We will first build this Text View:

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTHIERARCHYNODET	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODET	HRF: G/L Account Group (Hierarchy Node) - Text (IL)	GL Account Hierarchy Node Text

Create the view and adjust the following Colums:

ValidityEndDate with the expression TO_DATE(ValidityEndDate)
ValidityStartDate with the expression TO_DATE(ValidityStartDate)

In the newly created View (Text) change the semantic type from “relational dataset” to “Text”.

Work on the Semantic Types for the Attributes accordingly.

Set “Hierarchy Node” (HierarchyNode) representative key.

3.4 View (Relational Dataset) GL Account Hierarchy Node

We will now create the related Dimension:

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTHIERARCHYNODE	SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODE	HRF: G/L Account Hierarchy Node (IL)	GL Account Hierarchy Node

Create the view and adjust the following Colums:

ValidityEndDate with the expression TO_DATE(ValidityEndDate)
ValidityStartDate with the expression TO_DATE(ValidityStartDate)

Create two more calculated Columns:

Node ID (NODEID) with the expression:

CASE NodeType

WHEN 'L' then

CONCAT(ChartOfAccounts,CONCAT('/',GLAccount))

else HierarchyNode

end

Hierarchy Name (HIENAME) with the expression

GLAccountHierarchy

In the newly created View (relational Dataset) leave everything else as created by default.

3.5 View (Dimension) for GL Account Hierarchy Node

Next up is the Hierarchy with Directory View:

Relational Dataset

Technical Name View

Business Name View

Usage

HRF: G/L Account Hierarchy Node (IL)

(SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODE)

SAP_CC_FI_HRF_GLACCOUNTHIERARCHYNODE

HRF: G/L Account Hierarchy Node (IL)

GL Account Hierarchy Node

Create this view and project only the following four columns

Associate the Text View HRF: G/L Account Group (Hierarchy Node) -Text (IL) (SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODET) with the newly created view.

Adjust the semantics for the attributes

Choose "HierarchyNode" (HierarchyNode) as representative key.

3.6 View (Hierarchy with Directory) for GL Account Hierarchy View

Next up is the Hierarchy with Directory View:

Relational Dataset

Technical Name View

Business Name View

Usage

HRF: G/L Account Hierarchy Node (IL)

(SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODE)

SAP_CC_FI_HRF_GLAccountHierarchyView

HRF: G/L Account Hierarchy (Hierarchy with Directory)

GL Account Hierarchy

Like before, create the view based on the relational dataset HRF: G/L Account Hierarchy Node (IL) (SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODE).

In the newly created View (Hierarchy with Directory) change the semantic type from “relational dataset” to “Hierarchy with Directory”.

Create an Association between HRF: G/L Account Hierarchy Directory (DIM)(SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHY) and the newly created Hierarchy with Directory.

Create a second association between HRF: G/L Account Hierarchy Node (IL) (SAP_CC_FI_HRF_IL_I_GLACCOUNTHIERARCHYNODE ) and the newly created Hierarchy with Directory.

Work on the Semantic Types for the Attributes accordingly.

Change the keys to ValidityEndDate (ValidityEndDate), Node ID (NODEID) and Hierarchy Name (HIENAME). There is no need to set a representative key in this step.

Open the Hierarchy with Directory Settings and fill in the following information.

4. Apply the Hierarchy

You are almost there 🙂

4.1 View (Text) for GL Account Master Data Text

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTTEXTRAWDATA	SAP_CC_FI_HRF_IL_I_GLACCOUNTTEXTRAWDATA	HRF: G/L Account – Text (IL)	GL Account Master data Text

In the newly created View (Text) change the semantic type from “relational dataset” to “Text”.

Work on the Semantic Types for the Attributes accordingly.

Set GLAccount (GLAccount) as representative key.

4.2 View (Dimension) for GL Account Master Data Attributes

Next up is the related Dimension:

Name CDS View and DSP Local Table	Technical Name View	Business Name View	Usage
I_GLACCOUNTINCHARTOFACCOUNTS	SAP_CC_FI_HRF_IL_I_GLACCOUNTINCHARTOFACCOUNTS	HRF: G/L Account (DIM)	GL Account Master data Attributes

In the newly created View (Dimension) change the semantic type from “relational dataset” to “Dimension”.

Create an Association between HRF: G/L Account - Text (IL) (SAP_CC_FI_HRF_IL_I_GLACCOUNTTEXTRAWDATA) and the newly created Dimension.

Create a second association between HRF: G/L Account Hierarchy (Hierarchy with Directory) (SAP_CC_FI_HRF_GLAccountHierarchyView)

There is no need to work on the semantic types of the attributes.

Set “GLAccount” (GLAccount) as representative key.

5. Associate the Dimension

In our case we can now associate the Dimension with our Journal Entries, that are deployed as Fact View. We define the measures and create an Analytic Model. In the Data Preview we choose GL Account and can now apply several Hierarchies.

We hope that this guide helps. Feel free to reach out to us 🙂

Currency Translation in SAP Datasphere

2024-05-07T09:57:57.155000+02:00

Currency translation refers to the process of converting financial data from one currency to another.

This is particularly important for multinational companies that operate in multiple countries and use different currencies for their financial transactions.

Currency translation in SAP typically involves the following steps:

1. Setting up currency exchange rates: Before performing any currency translation, exchange rates need to be maintained in SAP. These rates can be manually entered or automatically updated from external sources.

2. Defining translation ratios: Companies may have specific rules for currency translation, such as using average rates, closing rates, or historical rates. These rules need to be defined within SAP.

3. Executing currency translation: Once the exchange rates and translation rules are set up, currency translation can be executed. SAP usually provides standard functionality and reports to perform this task.

4. Reviewing translated financial data: After translation, it's essential to review the translated financial data to ensure accuracy and compliance with accounting standards.

Currency translation in SAP is crucial for financial reporting, consolidation, and analysis purposes, allowing companies to view their financial data in a common currency, facilitating comparisons and decision-making across different regions.

To perform currency translation in SAP Datapshere, the following tables must be available in your space:

TCURV - Exchange rate types
TCURW - Exchange rate type text
TCURX - Decimal places in currencies
TCURN - Quotations
TCURR - Exchange rates
TCURF - Conversion factors
TCURC - Currency codes
TCURT - Currency text

Creating currency translation related tables in SAP Dataspher:

Login to SAP Datapshere, go to Data Builder menu, and click on create icon.
Select 'Currency Conversion View' from the drop-down.
Select source which can be manual or any SAP connection in your SAP Datasphere space which is compatible for replicating currency translation tables.
If source is manual, upload the data to all the required tables either manually or via data flows.
If SAP source is selected, respective remote tables and data flows will also get created.
After clinking on create option, 32 objects will get deployed including below 8 data flows
Create a task chain and add all these data flows for uploading the data to required TCUR* tables. You can also schedule the task chain to keep your data updated.
Execute the task chain to update the data.
Create 'currency conversion column' in required object (for instance in view)
Enter business name, technical name, data type, precision and scale.
In currency properties, select source amount column, source currency (fixed value or columns), target currency (fixed value, columns, or input parameter) and reference date (input parameters, columns, expressions, fixed date) for conversion.
Enter client, conversion type and error handling
Preview the data and verify the rate maintained:

Reference: https://youtu.be/m4m1TNJ5Ekw

Replication Flow Blog Part 6 – Confluent as Replication Target

2024-05-07T15:03:22.063000+02:00

This blog is part of a blog series from SAP Datasphere product management with the focus on the Replication Flow capabilities in SAP Datasphere: 

Replication Flow Blog Series Part 1 – Overview | SAP Blogs 

Replication Flow Blog Series Part 2 – Premium Outbound Integration | SAP Blogs 

Replication Flows Blog Series Part 3 – Integration with Kafka 

Replication Flows Blog Series Part 4 – Sizing   

Replication Flows Blog Series Part 5 – Integration between SAP Datasphere and Databricks 

Replication Flows Blog Series Part 6 – Confluent as a Replication Target

Data Integration is an essential topic in a Business Data Fabric like SAP Datasphere. Replication Flow is the cornerstone to fuel SAP Datasphere with data, especially from SAP ABAP sources. There is also a big need to move enriched data from SAP Datasphere into external environments to succeed certain use cases.

In this part of the Replication Flow Blog series we focus on the usage of Confluent as a target in a Replication Flow. We will explain in detail the new capabilities that have been introduced with SAP Datasphere release 2024.08. The content of this blog is structured as follows.

Introduction
Confluent as a new Connection Type in SAP Datasphere
Configuration options for Replication Flows
Details on Kafka Message and Schema creation
1. Schema & Message creation
2. Data Type mappings
Scenarios or what happens if...

1. Introduction

The purpose of the additional features that are described in this Blog, next to the generic Kafka integration that has been available since end of last year, is to provide tailor-made integration with the managed Kafka offerings Confluent Cloud and Confluent Platform of our dedicated partner Confluent.

The Confluent integration described in this blog is only usable in SAP Datasphere Replication Flows.

For the examples and step-by-step instructions in the blog, we assume that a properly configured SAP Datasphere tenant and a Confluent Cloud cluster are available. In addition, we assume that the reader is familiar with the basic concepts around Replication Flows and Connection Management in SAP Datasphere as well as with the Kafka capabilities of Confluent Cloud.

2. Confluent as a new Connection Type in SAP Datasphere

Beginning of this year a new connection type was introduced for Apache Kafka to offer basic integration with Apache Kafka based sinks. With the 2024.08 release of SAP Datasphere there is a new dedicated connection type introduced that is tailored to Confluent in order to support Confluent specific capabilities like Confluents built-in schema-registry.

Figure 1 - New connection type for Confluent

Here are the different configuration options for Confluent.

Connection Details

System Type
Possibility to choose between Confluent Cloud and Confluent Platform
Kafka Brokers
A comma-separated list of Kafka brokers in the format <host>:<port>.

Cloud Connector (only available if System Type Confluent Platform is chosen)

Use Cloud Connector
This setting configures whether a Cloud Connector is used

Authentication

The following SASL based authentication methods are supported.

For Confluent Cloud:

Authentication Type	SASL Authentication Type	Properties
No Authentication	n/a
API Key and Secret	PLAIN	API Key* API Secret*

*mandatory

For Confluent Platform:

Authentication Type	SASL Authentication Type	Properties
No Authentication	n/a
User Name and Password	PLAIN	Kafka SASL User Name* Kafka SASL Password*
Salted Challenge Response Authentication Mechanism (256)	SCRAM256	Kafka SASL User Name* Kafka SASL Password*
Salted Challenge Response Authentication Mechanism (512)	SCRAM 512	Kafka SASL User Name* Kafka SASL Password*
Kerberos with User Name and Password	GSSAPI	Kafka Kerberos Service Name* Kafka Kerberos Realm* Kafka Kerberos Config* User Name* Password*
Kerberos with Keytab File	GSSAPI	Kafka Kerberos Service Name* Kafka Kerberos Realm* Kafka Kerberos Config* User Name* Keytab File*

*mandatory

Security

Transport Layer Security (TLS) settings for encryption as well as server certification validation are supported for system type Confluent Cloud. In addition, Confluent Platform offers client certificate validation via mTLS.

Figure 2 - TLS configuration for Confluent Cloud and Confluent Platform

Schema Registry

Additional configuration options to specify schema registry endpoint and credentials.

URL
Endpoint of the schema registry in the format <host>:<port>
Authentication Type
Authentication mechanism that is supposed to be used for schema registry. User Name and Password and No Authentication are supported.
User Name + Password
User Name and Password in case the corresponding authentication type is chosen.

Remark: A schema registry configuration is mandatory when creating connections of type Confluent.

Connections of connection type Confluent can currently only be used as targets in Replication Flows.

Example: Creating a connection to Confluent Cloud in SAP Datasphere

We are now showcasing how to create a connection to Confluent Cloud including schema registry configuration. The case of Confluent Platform is similar assuming that a Cloud Connector has been configured properly.

Use the Bootstrap Server URL(s) of your Confluent cluster as an entry in the Kafka Brokers property.

Figure 3 - Specify Kafka Brokers in the SAP Datasphere connection creation wizard.

Choose User Name And Password as authentication type and use an API key and secret pair as Kafka SASL User Name and Kafka SASL Password.

Figure 4 - API Key and Secret as authentication credentials

Remark: It is assumed that the owner of the API key has sufficient rights to create/delete and access topics.

Use the Stream Governance API Endpoint as the Schema Registry URL.

Figure 5 - Specify schema registry URL in the SAP Datasphere connection creation wizard.

Choose User Name And Password as the authentication type for the Schema Registry and use an API key and secret pair as User Name and Password, respectively.

Figure 6 - Specify API Key Secret pair for Schema registry access.

Remark: It is assumed that the owner of the API credentials has sufficient rights to create and update schemas.

3. Configuration Options for Replication Flows

For the remainder of this blog post, we will describe the capabilities of the Confluent integration with Replication Flows alongside the following source data set that is assumed to be stored in a local table (Business Name: Demo Table) in SAP Datasphere.

Figure 7 - Example source dataset Demo Table

Let’s assume we want to replicate this table into a Confluent Cloud instance. We create a corresponding Replication Flow design time artifact, select SAP Datasphere and the local table Demo table, respectively, as source and chose the connection CONFLUENT_DEMO (see section 1) as the sink for the Replication Flow.

The following screenshots highlights the configuration options for the Confluent sink.

Figure 8 - Replication Flow configuration options for Confluent

Compared to the usage of the Apache Kafka connection type there are the following additional configuration options Use Schema Registry, Record Name, Subject Name Strategy, Compatibility Type and Clamp Decimal Floating Point Data Types. They are described in the following table together with the other properties.

Setting	Value	Explanation
Replication Thread Limit	Number	The number of parallel replication threads that can be executed during the replication process. Only available in Global configuration.
Number of Partitions	Number	The number of Kafka Partitions for the target Kafka topic. Only used for new topics that don’t yet exist in the Kafka Cluster, otherwise the setting is ignored.
Replication Factor	Number	The Kafka replication factor for the Kafka topic. Only used for new topics that don’t yet exist in the Kafka Cluster, otherwise the setting is ignored.
Message Encoder	AVRO or JSON	The message format for the Kafka topic.
Message Compression	No Compression Gzip Snappy LZ4 Zstandard	The compression method for the Kafka messages that are sent to a Kafka topic.
Use Schema Registry	True or False	True: Schema Registry is used False: Schema Registry is not used Schema registry is mandatory in case Message Encoder AVRO is chosen.
Topic Name	string	The name of the Kafka topic to be used as the target. The topic name is always based on the target object name. It can only be changed by renaming the target object.
Record Name	string	The record name that is used for the schema registry entry when the subject name strategy is applied. It is also referenced in the schema definition itself.
Subject Name Strategy	Topic Record Topic-Record	Choose the subject name strategy for the schema. Only available if Use Schema Registry is used/true.
Compatibility Type	Default Backward Backward Transitive Forward Forward Transitive Full Full Transitive None	Choose the compatibility type for the schema registry subject. Only available if Use Schema Registry is used/true.
Clamp Decimal Floating Point Data Types	True	Immutable setting (always True) Decimal values that do not fit into the target type (see section 4.2) are automatically clamped
Overwrite Target Settings at Object Level	True or False	True: The global configuration overwrites the configurations made at task level. Only available in global configuration

The settings on Replication Task level take precedence over the settings on Replication Flow level unless the Overwrite Target Settings at Object Level checkbox is checked in the settings on Replication Flow level.

After the Replication Flow has been deployed and run the Kafka topic was created and a corresponding schema registry entry was created (see section 4 for details on schema and message creation).

Figure 9 - Example of Target Kafka Topic

Remark:
As it was the case with the generic Kafka integration, for Confluent we support a Truncate flag. If set and in case the target Kafka topic for the replication is already existing in the Confluent cluster, it is deleted an recreated. This also deletes all messages that are assigned to the topic.

Figure 10 - Truncate flag configuration setting in Replication Flows

In the following section we will have a closer look on schema and message creation.

4. Details on Kafka Message and Schema creation

Before describing the details regarding schema and message creation, we start with a listing of the producer configurations that are used in SAP Datasphere Replication Flows. The parameter values are fixed and cannot be changed in SAP Datasphere.

Kafka Producer Configuration Parameter	Value used by Replication Flows	Remark
max.request.size	1048576 (1MB)	Confluent Default
request.timeout.ms	30 seconds	Confluent Default
max.in.flight.requests.per.connection	5	Confluent Default
retry.backoff.ms	100	Confluent Default
acks	all (-1)	Confluent Default
retries	3

4.1 Message & Schema Creation

The following section is only applicable in case the Use Schema registry toggle is activated (true) in the Replication Flow/Task properties panel. In this case schemas a written to the schema registry as described in the following paragraphs.

No schema definition for the Kafka message key (e.g. […]-key) is created in Confluent. Instead, the message key is a string consisting of the primary key values of the source dataset separated by underscores.

Figure 11 - Kafka message key generation

If, in case of our example above, we would have the two columns ID and First_Name together as a primary key, then the generated message keys would look as follows.

Figure 12 - Second example for Kafka message key generation

The schema definition for the Kafka message body is constructed and created based on the structure/schema of the source dataset, the configuration settings of the executed Replication Flow and the change data capture mechanisms of SAP Datasphere. The following screenshot shows how the different settings are fed into the Kafka schema.

Figure 13 - Schema creation for the Kafka message body

The corresponding Confluent schema and subject names, the serialization format and the compatibility mode are derived from the configuration values of the Replication Flow and Replication Task definition, respectively.

The schema definition for the message body itself is derived from the source dataset definition and the metadata for the delta information of the Replication Flow. In our example the schema of the Demo Table is translated into an AVRO schema that contains the four fields ID, First_Name, Second_Name and Age as well as the three delta columns that are always added and which contain the Replication Flow specific delta information. The AVRO type of the message body is always record.

In case the serialization format is set to JSON a corresponding JSON schema is registered in the Confluent schema registry that follows the JSON Schema standard. The following schema definition would have been written to the Confluent schema registry in case JSON serialization format would have been used in our example above:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "Age": {
      "anyOf": [
        {
          "maximum": 9223372036854776000,
          "minimum": -9223372036854776000,
          "type": "integer"
        },
        {
          "type": "null"
        }
      ]
    },
    "First_Name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ]
    },
    "ID": {
      "anyOf": [
        {
          "maximum": 9223372036854776000,
          "minimum": -9223372036854776000,
          "type": "integer"
        },
        {
          "type": "null"
        }
      ]
    },
    "Second_Name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ]
    },
    "__operation_type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ]
    },
    "__sequence_number": {
      "anyOf": [
        {
          "maximum": 18446744073709552000,
          "minimum": 0,
          "type": "integer"
        },
        {
          "type": "null"
        }
      ]
    },
    "__timestamp": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ]
    }
  },
  "title": "Demo_Table",
  "type": "object"
}

The topic itself is created and configured based on the Replication Flow/Task settings Number of Partitions, Replication Factor and Topic Name. All other topic specific configuration parameters are inherited from the Confluent Cluster settings.

For initial of delta loads during a Replication Flow run for each row in the source data set exactly one Kafka message is added to the target Kafka topic.

In case of our example (initial load of a SAP Datasphere table) the final Kafka messages look as follows (an overview of the Kafka topic is shown in figure 10 above).

Figure 14 - Example Kafka messages for Demo Table replication

The next section contains an overview of type mappings between selected source systems and JSON/AVRO schema types.

4.2 Data Type Mappings

The following two tables contain the data type mappings for the scenarios where SAP HANA/SAP Datasphere or ABAP artifacts are chosen as a source.

Scenario: SAP HANA/SAP Datasphere to Confluent

SAP HANA Type	JSON Type	AVRO Type
TINYINT	number	int
SMALLINT	number	int
INTEGER	number	int
BIGINT	number	long
REAL	number	float
DOUBLE	number	double
SMALLDECIMAL	number	type:"bytes",logical type:"decimal",scale:6,precision:28
DECIMAL	number	type:"bytes",logical type:"decimal",scale:6,precision:38
DECIMAL(p,s)	number	type:"bytes", logical type:"decimal",scale:s,precision:p
FLOAT	number	double
FLOAT(n), 1<=n<=24	number	float
FLOAT(n), 25<=n<=53	number	double
BOOLEAN	boolean	boolean
DATE	string ('YYYY-MM-DD')	type:"int",logical type:"date" (days from UNIX 0)
TIME	string ('HH:MM:SS.NNNNNNNNN')	type:"long",logical type:"time-micros"
CLOB/NCLOB	string	string
SECONDDATE	string ('YYYY-MM-DDTHH:MM:SS.NNNNNNNNNZ')	type:"long",logical type:"timestamp-micros" (microseconds after UNIX 0)
TIMESTAMP	string ('YYYY-MM-DDTHH:MM:SS.NNNNNNNNNZ')	type:"long",logical type:"timestamp-micros" (microseconds after UNIX 0)
VARCHAR(n)	string	string
NVARCHAR(n)	string	string
ALPHANUM(n)	string	string
SHORTTEXT(n)	string	string
VARBINARY(n)	string	bytes

Scenario: ABAP to Confluent

ABAP Data Dictionary Type (DDIC)	JSON Type	AVRO Type
INT1	number	int
INT2	number	int
INT4	number	int
INT8	number	long
DEC	number	type:"bytes", logical type:"decimal",scale:s,precision:d
DF16_DEC / D16D	number	type:"bytes", logical type:"decimal",scale:6,precision:28
DF34_DEC / D34D	number	type:"bytes",logical type:"decimal",scale:6,precision:38
FLTP	number	double
CURR	number	type:"bytes", logical type:"decimal",scale:s,precision:d
QUAN	number	type:"bytes", logical type:"decimal",scale:s,precision:d
DECFLOAT16 / D16N	number	type:"bytes", logical type:"decimal",scale:6,precision:28
DECFLOAT34 / D34N	number	type:"bytes",logical type:"decimal",scale:6,precision:38
DF16_RAW / D16R	number	type:"bytes", logical type:"decimal",scale:6,precision:28
DF34_RAW/D34R	number	type:"bytes",logical type:"decimal",scale:6,precision:38
RAW	string	bytes
LRAW	string	bytes
RAWSTRING / RSTR	string	bytes
SRAWSTRING / SRST	string	bytes
CHAR	string	string
LCHR	string	string
SSTRING / SSTR	string	string
DATS	string ('YYYY-MM-DD')	type:"int",logical type:"date" (days from UNIX 0)
DATN	string ('YYYY-MM-DD')	type:"int",logical type:"date" (days from UNIX 0)
TIMS	string ('HH:MM:SS.NNNNNNNNN')	type:"long",logical type:"time-micros"
ACCP	number	???
NUMC	string	string
CLNT	string	string
LANG	string	string
UTCLONG / UTCL	string ('YYYY-MM-DDTHH:MM:SS.NNNNNNNNNZ')	type:"long",logical type:"timestamp-micros" (microseconds after UNIX 0)
CUKY	string	string
UNIT	string	string
STRING / STRG	string	string
GEOM_EWKB / GGM1	string	string
TIMN	string ('HH:MM:SS.NNNNNNNNN')	type:"long",logical type:"time-micros"
Domain TZNTSTMPL	string ('YYYY-MM-DDTHH:MM:SS.NNNNNNNNNZ')	type:"long",logical type:"timestamp-micros" (microseconds after UNIX 0)
Domain TZNTSTMPS	string ('YYYY-MM-DDTHH:MM:SS.NNNNNNNNNZ')	type:"long",logical type:"timestamp-micros" (microseconds after UNIX 0)
Domain SYSUUID_X16 and SYSUUID	string	string
Domain SYSUUID_C22 SYSUUID_22	string	string
Domain SYSUUID_C26	string	string
Domain SYSUUID_C32 and SYSUUID_C	string	string
Domain SYSUUID_C36	string	string

In the final section we will have a look on the behavior of SAP Datasphere Replication Flows when they are scheduled and a Confluent cluster is selected as the target system.

5. Scenarios or what happens if...

In this section we assume that the Kafka topic a Replication Flow/Task is supposed to write to already exists in the target Confluent Cluster. By making different assumptions, we explain the behaviour of a Replication Flow/Task that is configured to use the already existing topic. We will again leverage our small Demo Table setup and we assume that the target Kafka topic Demo Table already exists.

In general, the concept of the Kafka schema registry are always applied when a new schema is registered during a Replication Flow run.

Scenario 1: A schema entry in the schema registry does not exist.

A schema is registered in the Schema registry using the subject name strategy that is specified in the Replication Flow. The messages are written by the Replication Flow into the already existing topic.

Scenario 2: A schema entry in the schema registry exists but with a different subject Naming strategy that is specified in the Replication Flow

A new schema is registered in the Schema registry using the subject name strategy that is specified in the Replication Flow. The messages are written by the Replication Flow into the already existing topic.

Figure 15 - Demo Table Example for Scenario 2

Scenario 3: A schema entry in the schema registry exists with the same subject name strategy that is specified in the Replication Flow

Assumption A: The compatibility type that is specified in the Replication Flow coincides with the compatibility type of the already existing schema/subject definition and the new schema definition is different from the already existing one but complies with the compatibility type.

A new schema version is registered based on the compatibility type that was specified for the existing schema/subject. The messages are written by the Replication Flow into the already existing topic.
Figure 16 - Demo Table Example for Scenario 3 Assumption A

Assumption B: The compatibility type that is specified in the Replication Flow is different from the compatibility type of the already existing Schema/Subject definition, but the schema definition is the same.

The compatibility type is overwritten, and no new schema is registered in the schema registry. The messages are written by the Replication Flow into the already existing topic.
Figure 17 - Demo Table Example for Scenario 3 Assumption B Part I
Watch out that such a change may produce schema version chains with changes that do not comply with the compatibility type that is specified in the schema definition.
Figure 18 - Demo Table Example for Scenario 3 Assumption B Part II

Assumption C: The compatibility type that is specified in the Replication Flow is different from the compatibility type of the already existing schema/subject definition and the schema definition is different but complies with the new compatibility type.

The old compatibility type is overwritten and a new schema version for the already existing subject is registered in the schema registry applying the new compatibility type
Figure 19 - Example: Compatibility type Backward is overwritten by Forward and a new schema version is registered that introduces a forward compatible change.

Assumption D : The compatibility type that is specified in the Replication Flow/Task is the same as the one that is specified in the already existing Schema/Subject definition but the schema in the Replication Flow is different or introduces a change that does not comply with the compatibility type.

The Replication Flow fails with an error message that indicated that the new message schema that is supposed to be registered in the schema registry is incompatible with the already existing one.
Figure 20 - Demo Table Example for Scenario C Assumption D

Summary

In this blog post we introduced new integration capabilities of SAP Datasphere with Confluent Cloud and Confluent Platform. The intention was to provide as many details as possible and provide step-by-step guides.

Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive

2024-05-08T17:00:00.007000+02:00

Introduction

This blog post is part of a series describing the usage of MLflow with HANA Machine Learning co-authored by @stojanm and @martinboeckling. In this blog post we provide a more technical deep dive on the setup of a MLFlow instance and provide a general introduction how Machine Learning models trained with HANA ML can be logged with MLflow. The first blog post of the blog series is called Tracking HANA Machine Learning experiments with MLflow: A Conceptual Guide for MLOps and gives an introduction to the topic of MLOps with MLflow.

Starting with the python HANA ML package version 2.13, HANA Machine Learning added support for tracking of experiments with the MLflow package, which makes the incorporation of models developed using HANA Machine Learning into a comprehensive MLOps pipeline easy to achieve.

In this blog post we will provide an overview how MLflow can be used together with HANA ML. MLflow, which manages the experiment tracking and artefact management can run as a managed service at a hyperscaler platform, deployed locally or on a remote infrastructure. In the following we describe how to deploy MLflow on SAP Business Technology Platform and how to track your HANA machine learning experiments with MLFlow. In addition, we present which methods and algorithms in the hana ml package currently support the experiment tracking feature. Finally, we touch on the possibility to use logged models in MLflow for prediction.

Prerequisites

In this blog post we solely focus on the technical integration of HANA ML and MLflow as a logging platform. Generally, we assume that Python is already installed together with an already established development environment. Furthermore, we will not completely explain all details of docker and Cloud Foundry, but simply focus on the essential parts for HANA ML and MLFlow within this blog post.

Set up MLFlow on BTP

MLFlow is leveraged and integrated in different solutions. For example Databricks as well Machine Learning in Microsoft Fabric integrate or provide a managed MLflow instance. In case MLFlow is not yet provided, we outline in this section a possibility to deploy MLflow in SAP BTP. We focus for simplicity reasons on the SQLite based deployment of MLflow. However, for productive environments it is recommended to separate the storage from the runtime of the MLFlow instance. A detailed explanation for setting up your own MLFlow server with alternatives to SQLite can be found under the following link: https://mlflow.org/docs/latest/tracking/server.html. In the following paragraphs, we focus on a step by step overview to set up your own MLFlow instance using SQLite in BTP.

As a first step we want to create a local docker file which can be used to upload it to our BTP environment. In the following code snippet, we provide the coding used to construct your own docker container locally. For that, paste the following code into a file called Dockerfile within your desired local folder.

# Retrieve a python version as a base runtime for our docker container
FROM python:3.10-slim
# Run the pip install command for the package of mlflow
RUN pip install mlflow
# Create a temporary folder within our docker container to store our artifact
RUN mkdir -p /mlflow
# Expose the port 7000 to make our application which runs within docker accessable over the defined port
EXPOSE 7000
# Define environment variables BACKEND_URI and ARTIFACT_ROOT to define the backend uri as also the 
# artifact root
ENV BACKEND_URI sqlite:///mlflow//mlflow.db
ENV ARTIFACT_ROOT /mlflow/artifacts
# run the shell comand to setup our mlflow server within our docker container
CMD mlflow server --backend-store-uri ${BACKEND_URI} --host 0.0.0.0 --port 7000

After creating successfully the Docker object, you can run docker build -t {tagname}. to construct your docker container. Afterwards, the local docker image is locally built. To expose the docker image, we push the image to a docker registry. In our example, we assume that you already have a docker registry set up where you can push your image to. For that step, you can run the following commands: docker tag {tagname} {dockerhub repository tag}, docker push {dockerhub repository tag }. After the successful run of the command, you see within your private docker hub the newly published docker container, which contains MLflow and all its dependencies inside of it.

After the successful publishing of your docker image to your registry, we can run the following command to create a BTP app based on the published docker image:

cf push APP-NAME --docker-image REPO/IMAGE:TAG

After successfully publishing the docker image on our BTP Cloud Foundry environment, we can find our published app within our BTP account and are able to access it under the published URL.

Set up tracking for MLflow

With MLflow users have the possibility to track their trained HANA ML models. In the following paragraph, we introduce the aspects that are needed to be able to log HANA ML models into MLflow itself.

To be able to use MLFlow together with HANA ML, we need to first install besides the HANA ML package also the MLFlow package. Therefore, you need to run the following command in your virtual environment, to be able to run the following scripts.

pip install mlflow hana-ml

As a general setup, we first need to run the following command to set up our tracking with MLFlow to our available MLFlow instance. Therefore, place into the following two lines first your personal MLFlow tracking URI and your own custom experiment. In case you do not want to create a separate experiment, the different runs together with the MLFlow model are stored under the default experiment.

The method that allows us to track HANA ML models is implemented in the HANA ML package and is called enable_mlflow_autologging(schema=None, meta=None, is_exported=False, registered_model_name=None). This method can be used for initialised HANA ML models that are under the following methods:

Within the method enable_mlflow_autologging the user has different keywords that can be filled that allows us to influence the behaviour of our MLFlow autologging in HANA ML.

schema: Defines the HANA database schema for MLFlow autologging where the MLflow logging table is stored
meta: Defines the name of the model storage table in HANA database
is_exported: Determines if the hana model binaries should be exported to MLflow
registered_model_name: Name of the model stored in MLflow

In the following section we provide an overview for the Unified Interface how the logging of MLflow can be used.

Run HANA ML Algorithms with MLflow

As we have explained and outlined in the sections above, we have created a MLFlow instance and have introduced the syntax that is needed for the logging of HANA ML models in MLflow. In the following sections we will provide based on an example how the logging of HANA ML models on MLflow is done.

Model training of HANA ML with MLFlow

For the training of HANA ML in combination with MLflow, we focus in this blog post on the Unified Method. We apply for the respective elements a Classification on the sample bank dataset which can be found under the HANA ML sample dataset folder on GitHub.

The dataset can either be uploaded directly to the SAP HANA database or you could also use SAP Datasphere as your starting point. Generally, to use HANA ML directly you would need to store the dataset in a HANA database. However, HANA ML also provides methods to integrate third party files/ data structures. This involves Pandas, Spark as also shapefiles. In addition also HANA Data Lake file tables can be integrated with HANA ML functionalities. An overview of the different methods can be found under the following page. In the following paragraphs, we will go through the sample code that we have created to combine HANA ML and MLFlow.

Connect to HANA database (Deployed under SAP Datasphere)

To be able to connect to the HANA database instance, we first need to build up a connection to the HANA database. In our example, we load the data from the data samples provided by HANA ML. During the time of this blog post, the OpenSQL schema of Datasphere only supports Basic Authentication. Therefore, in this blog post we only elaborate how the connection is done over basic authentication. SAP HANA standalone supports however non-basic authentication, which are also supported in the HANA ML package to connect certificate based to the SAP HANA instance.

To establish the connection to the HANA database, we make use of the implemented HANA ML dataframe class and call the method ConnectionContext. We store the instance of the connection in the variable conn. To now be able to establish the connection to the HANA database view or table, we will need to specify over the method table the connection. The beautiful aspect is that overall, the dataset is not going to be loaded to the Python runtime, but will only be represented with a proxy to the actual table in the HANA database. All transformations, if done over the methods of HANA ML, are then pushed down to the database itself and executed there if the training is executed. In our case, we load the sample dataset into our database by making use of the provided methods of the HANA ML package.

"""
This script provides a short example how HANA ML and MLFlow can be integrated together.
The credentials to the database are following the currently supported authentification (Basic) of the 
SAP Datasphere OpenSQL schema. Overall, HANA Cloud standalone is also able to support multiple other authentification
methods. We have used an abstraction python file (constants) where we retrieve the securely stored 
authentification properties. To get more details about the exact method structure needed, please
have a look at the documentation:
https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.07/en-US/hana_ml.dataframe.html#hana_ml.dataframe.ConnectionContext
"""
from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.algorithms.pal.auto_ml import AutomaticClassification
import mlflow
from hana_ml.algorithms.pal.auto_ml import Preprocessing
from hana_ml.algorithms.pal.partition import train_test_val_split
from constants import db_url, db_user, db_password
# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url, port=443, user=db_user, password=db_password)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

After connecting to the database, the user is able to use the preprocessing methods implemented in HANA ML. Generally, the different changes are pushed down to the HANA database and are not executed within the Python runtime. In our use case, we do not need to use the data preprocessing as we directly retrieve a sample dataset which we can directly use for our ML training.

After finishing the potentially needed transformations, we are now able to implement the tracking of our HANA ML runs with the possibility of MLFlow. Similar to the normal usage of MLFlow, we set up first our tracking uri under which we want to store our HANA ML runs and models. In your case you would need to change the keyword mlflow_tracking_uri with your respective MLflow tracking URL. Furthermore, we then are able to specify the experiment name under which the runs are tracked. If we do not specify a specific experiment, the runs are tracked under the Default experiment.

# set up MLFlow
mlflow.set_tracking_uri(mlflow_tracking_uri)
mlflow.set_experiment("HANA ML Experiment")

In the following chapters, we will provide an outline how the exact training is performed and what components are logged to MLflow.

Unified Method

For the example we use the implemented Hybrid Gradient Boosting Tree as a classification algorithm for our Classification. In order to perform the classification, we use the Unified Classification in order to be able to run our algorithm. On the defined variable, we then use the implemented enable_mlflow_autologging method. This allows us to directly log the model using implemented auto logging behaviour.

uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging() uc.fit(training_data, key="ID", label="responded")

We call the fit method once we have initiated the HANA ML model variable and the associated autologging for MLflow. For the fit method, we have in total two different options. Firstly, the non-partitioned training dataset where we only use the training dataset. If we decide to partition our training dataset, we allow to create a validation dataset for which we can log metrics automatically during training.

If we do not define for our fit function the partitioning, we will not log metrics within MLFlow. In the following image, you can see how a potential HANA ML tracked run looks like in MLFlow together with the stored HANA ML model in MLflow.

If we decide to partition our dataset, here for instance to partition the dataset along the defined primary key, we are able to directly log evaluation metrics relevant for the Classification we have used. This includes the following metrics: AUC, Recall, Precision, F1 Score, Accuracy, Kappa coefficient and the Mathews Correlation Coefficient (MCC). This would directly allow us to compare multiple runs within our MLFlow project to one another and measure the different performances.

In addition to the general run, HANA ML also logs the model to MLflow. What is logged to MLflow depends on the parameters set for the method enable_mlflow_autologging. If for instance everything is set to the default settings, we will see the following yaml file to be logged to MLflow.

If within the method enable_mlflow_autologging the parameter is set to is_exported, the model binaries stored in the model storage on HANA are exported to MLflow. This setting would allow us to retrieve the trained model from MLflow and use it in a different HANA database for prediction purposes. In addition to the yaml file containing the metadata we now can see a created subfolder called models which contains the necessary model artefacts normally stored in the HANA database now in MLflow.

After the training is finished, we have besides the auto logging capabilities of HANA ML for MLflow the possibility to track further artefacts in MLflow. In the following section we will outline a few possibilities that exist with the additional tracking.

Additional logging possibilities

Besides the outlined auto logging capabilities, we can track with MLFlow additional artefacts to the respective run. In the following chapters, we outline selected possibilities to further enrich the auto logging for HANA ML runs tracked in MLFlow.

Adding run and experiment description

The description in the experiment section can be handy once the number of your experiments grows in the repository. In addition, mlflow allows to also add individual description to each run of an experiment. Using the following methods you can set up both:

from mlflow.tracking import MlflowClient
current_experiment=dict(mlflow.get_experiment_by_name("HANA ML Experiment"))
experiment_id=current_experiment['experiment_id']
    
run = mlflow.active_run()
MlflowClient().set_experiment_tag(experiment_id,"mlflow.note.content",
"This experiment shows the automated methods of HANA machine learning and how to track them with MLFLOW")
MlflowClient().set_tag(run.info.run_id, "mlflow.note.content", "This is a run tracked with Unified Classification from HANA Machine Learning")

Logging input datasets

Sometimes it is important to keep the input dataset also as part of the tracking with MLflow. Since HANA machine learning datasets are located in HANA, they need to be converted to pandas DataFrames to be tracked as shown in the following code:

# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

This results in the change, that the respective state of the training data is logged to the current run. The logged dataset can be found in the associated MLflow run, where the schema of the dataset is provided together with some metadata information about the number of rows and number of elements. In addition, also the provided context is marked in the UI of MLflow.

Logging a model report

In addition to the logging of the dataset, it might also be important to add a model report to MLFlow. HANA ML generally provides different interactive visualisations for the trained model artefact, which can be stored as an HTML file. After the storing of the model report to your local repository, we can log the input of the model report to our current run. This allows us to interactively explore the model report automatically generated by HANA ML and make it accessible in MLFlow. To log the HANA ML model report, you can use the following code snippet.

# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")

After the Model report is stored successfully under the current run, we can see in the artefact tab in MLFlow the interactive model report:

The complete script used for this section can be found in the following code snippet:

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLflow
mlflow.set_tracking_uri(tracking_uri)
mlflow.set_experiment("HANA ML Experiment")
# set up classification
uc = UnifiedClassification(func="HybridGradientBoostingTree")
uc.enable_mlflow_autologging(is_exported=True)
# train model
uc.fit(training_data, key="ID", label="LABEL", partition_method="stratified", stratified_column="ID", partition_random_state=43, build_report=True)
# create additional model report in MLFlow
UnifiedReport(uc).display(save_html="UnifiedReport")
mlflow.log_artifact("UnifiedReport_unified_classification_model_report.html")
# Store training dataset in MLFlow itself
pandas_training_dataset = training_data.collect()
mlflow_dataset = mlflow.data.from_pandas(pandas_training_dataset, name="Customer data", targets="LABEL")
mlflow.log_input(mlflow_dataset, context='training')

Apply of trained model

After we have finished our training, we are able with HANA ML to retrieve the model from MLFLow and use it for our prediction purposes. For this purpose, we will create a separate Python script where we will provide an overview to retrieve the trained MLflow model.

Similar to our training script, we first set up our connection to the HANA database and establish the connection to our table. In our case, we simply use the sample dataset provided by HANA ML.

from hana_ml import dataframe
from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
from hana_ml.visualizers.unified_report import UnifiedReport
import mlflow
from hana_ml.algorithms.pal.utility import DataSets
from constants import db_url, db_user, db_password

# dataset retrieval
conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

Similar to our training script, we need to set the tracking url for MLflow and need to initiate the model storage of HANA. If we have decided to not export the HANA ML model to MLflow, we need to specify the same schema for the model storage where our HANA ML model is stored after the successful run. In case we have exported our model, we are able to specify a different schema. In the following, you can see the necessary script in order to retrieve the logged HANA ML model from MLflow.

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_url)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)

After the model storage has been initiated, we are able to retrieve the stored HANA ML model from MLflow. In order to select the correct model, you need to extract the correct run id associated to the model you would like to apply for your prediction dataset. In our case, this is the test dataset we have received from the sample dataset method. The model_uri needed for the model retrieval is consisting of the following pattern 'runs:/{run id}/model', in which you would need to exchange the run id with your respective run. For the actual retrieval of the model, we use the initiated model storage, in our case called model_storage and call the method load_mlflow_model to load the MLflow model to our HANA database and assign the respective proxy to our variable mymodel. The variable mymodel is then used to call the predict method in order to apply our model to our dataset. In the end we transform our prediction dataset into a Pandas DataFrame to look at the content of the created DataFrame. Normally, we could directly persist the created temporary table with the save method and therefore make the dataset available for further processing.

# load logged run from MLflow to HANA ML
logged_model = 'runs:/d8a763b7b81940598633605e447cd880/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

After running the script, you should be able to see the following terminal output, for which we can see the download of the artefact stored in MLflow and the created prediction dataset, which consists in our case of 4 columns: ID (primary key), SCORE (predicted label), CONFIDENCE (prediction confidence for applied row) and REASON_CODE (influence of individual variables to prediction output).

In case we have exported our model, the output of our terminal look slightly different indicating that we also download the respective model artefacts stored additionally to the yaml file. In the following you see the complete script used for applying the model to a new dataset.

from hana_ml import dataframe
from hana_ml.model_storage import ModelStorage
from hana_ml.algorithms.pal.utility import DataSets
import mlflow
from constants import db_url, db_user, db_password

conn = dataframe.ConnectionContext(address=db_url,
                                   port=443,
                                   user=db_user,
                                   password=db_password)

# full_set, diabetes_train, diabetes_test, _ = DataSets.load_diabetes_data(conn)
dataset_data, training_data, _, test_data = DataSets.load_bank_data(connection=conn, schema=schema_name, train_percentage=0.7, valid_percentage=0, test_percentage=0.3, seed=43)

# set up MLFlow and model storage
mlflow.set_tracking_uri(tracking_uri)
model_storage = ModelStorage(connection_context=conn, schema=schema_name)
# load logged run from MLflow to HANA ML
logged_model = 'runs:/ed7b8d4734cb42ca90c417f932957b40/model'
mymodel = model_storage.load_mlflow_model(connection_context=conn, model_uri=logged_model)
dataset_data_predict = mymodel.predict(data=test_data, key="ID")
# collect the predicted dataset to see content in dataframe
print(dataset_data_predict.collect())

Key take aways

In this blog post we have showcased an end to end example how MLflow can be integrated in your HANA ML workload by providing the possibility to share and compare multiple tracked runs in MLflow. If the data is already stored in HANA, this allows you to directly interact with MLflow while being able to run your Machine Learning algorithms on data stored in the HANA database without the need to transfer your data between multiple systems. This blog covered an essential part of the automated logging capabilities of HANA ML models into MLflow.

We highly appreciate your thoughts, comments and questions under this blog post. In case you want to reach out for general questions around HANA, or specifically HANA ML, don't hesitate to use the Q&A tool with the respective tags that describe your question.

SAP Datasphere catalog - Harvesting from SAP Datasphere, SAP BW bridge

2024-05-08T17:48:25.809000+02:00

In March 2023, we enhanced SAP Datasphere with catalog capabilities to democratize data access and strengthen data-driven decision-making (launch blog). This feature facilitates the curation and organization of metadata harvested from various SAP data sources, assists in discovering curated data assets, and enables search and classification of these assets by users and data stewards. The catalog further enriches assets with business metadata, aiding organizations in their data governance journey. It connects to sources such as SAP Datasphere and SAP Analytics Cloud from its inception, retrieving metadata from a variety of assets within these systems, including but not limited to stories, models, insights, predictive scenarios, local tables, remote tables, views, and data actions.

Today we are happy to announce that we recently delivered a new enhancement that extends the coverage of our metadata harvesting with the support to connect and retrieve technical metadata from SAP Datasphere, SAP BW bridge.

SAP Datasphere, SAP BW bridge provides a path that eases the transition from existing SAP BW systems to an innovative infrastructure supported by SAP Datasphere. This helps organizations to reuse existing investments in SAP BW by allowing you to transfer and enable the rich feature set of extractors and ABAP code for access to legacy SAP on-premises systems, and to seamlessly transfer existing ETL processes within a dedicated space.

Feature Description

You can add an SAP Datasphere, SAP BW bridge system to the catalog to as a source to harvest metadata of the following objects:

Advanced Data Store Objects
Info Areas
Composite Providers
Info Objects
Source Systems
Data Sources
Info Sources
Transformations

Once extracted, these new objects can be enriched with business metadata and published to the catalog like any other assets.

Create a Connection to an SAP Datasphere, SAP BW bridge system

In this section, we will briefly show how to connect to an SAP Datasphere, SAP BW bridge system to harvest the metadata for the catalog.

Connect to your SAP Datasphere tenant and access the Monitoring page under the Catalog section in the left menu.

On the monitoring page, click “+” to create a new entry for a system.

Select SAP Datasphere, SAP BW bridge.

Click Create.

You can now configure which Info Areas you want to select and then save the configuration for this connected system.

Now you can sync to request the harvesting of the metadata related to the selected Info Areas that define the connection to the SAP Datasphere, SAP BW bridge system.

Interact with the catalog

In this section, we will briefly show how to search and interact with assets in the catalog that are from the SAP Datasphere, SAP BW bridge connected system.

As usual, navigate to the Catalog, and search for any content, but also now you can leverage the Info Areas as well as the Source Systems associated to the SAP Datasphere, SAP BW bridge connected system as structure to narrow down your search result.

The support of metadata harvesting also introduces new asset types which can be used to filter and narrow down your discovery of the available content.

Once you find (or locate) the asset you were looking for, you can review its properties and the column details when . Column details, for example for attributes also allow to easily navigate to other referenced info objects in the catalog to further discover the data.

Like any other asset in the catalog, you can also look at the lineage and impact analysis graph to see the other assets associated to it.

The lineage and impact analysis graph allows to navigate to the different assets to further discover the data available and their relationships.

Any of the existing feature to enrich the assets with business metadata such as descriptions, business glossary terms, KPIs, and Classification Tags can be used to further enhance the metadata for the SAP Datasphere, SAP BW bridge harvested metadata as described in this blog.

What’s Next?

Soon, for the SAP Datasphere, SAP BW bridge metadata harvesting, we will enable scheduling of the metadata harvesting to remove the need to manually sync that process. Additionally, we will add column groupings for the column details associated to the assets.

We will continue to extend the coverage of the SAP metadata we can harvest to continuously support organizations on their data governance journey.

I also want to share some additional resources for you around SAP Datasphere:

Access to your free 30 day trial of SAP Datasphere
Join SAP Datasphere community to see what’s new, ask questions, share your feedback, and more.
SAP Datasphere product overview
Documentation for SAP Datasphere catalog

Tracking HANA Machine Learning experiments with MLflow: A conceptual guide for MLOps

2024-05-08T19:17:01.849000+02:00

Introduction

MLflow is an open-source platform, which is the de facto standard, when it comes to managing and streamlining machine learning lifecycles, including experimentation, reproducibility, and deployment. It offers a centralized repository to track experiments, share projects, and collaborate effectively, making it a common choice among data scientists and can be used with most open-source machine learning frameworks (e.g. scikit-learn, Tensorflow, etc.).

Some providers offer MLflow as a managed service (e.g. Databricks) or integrate it (e.g. Azure ML). In addition, it is possible for the user to deploy the service manually on their platform of choice (e.g. on SAP Business Technology Platform).

Starting with version 2.13 HANA Machine Learning added support for tracking of experiments with the mlflow package. This makes models, which were developed using hanaml, easily incorporated into an extensive MLOps pipeline.

This blog post is part of a series describing the usage of MLflow with HANA Machine Learning co-authored by @martinboeckling and @stojanm. In the first part we present an conceptual guide on how to use MLflow with SAP Datasphere and HANA Machine Learning (through the hanaml package). The objective is to provide to the reader a high level template for machine learning operations (MLOps) for HANA ML specifically with MLflow. In the second part of the series, called Tracking HANA Machine Learning experiments with MLflow: A technical Deep Dive, we provide a more technical deep dive on how to setup an MLflow instance and a general introduction on how Machine Learning models trained with HANA ML can be logged with MLflow.

It is important to mention that SAP offers an extensive MLOps platform for managing ML experiments, AI Core / AI Launchpad, which is out of the scope of this post. For more information on AI Core please refer to the blog post here.

Ok, let's start reviewing our example. We will work our way along a simplified Machine Learning pipeline as shown below and will comment on the architectural patterns for each task.

To simplify the use case, we will assume that the gravity of the data required for our model lies within SAP. This means that majority of the data used for model training is in an SAP application either on-premise or in the cloud.

Data Modeling

Typically data landscapes in enterprises are quite complex and data is distributed across numerous systems. So, even though the majority of the data for our example comes from an SAP source, it is realistic to assume that for the modeling a portion of that data could come from another system. It is the task of a Data Engineer to connect to the data and to prepare it for the algorithm training (e.g. feature engineering). As shown in the picture below SAP Datasphere can help unify data sources in a central repository either via federation or, where not supported, via replication.

In addition to the data modeling features, SAP Datasphere also offers a runtime for Machine Learning tasks thanks to the embedded HANA Cloud instance. This runtime can be utilized by Data Scientists and since it is embedded it allows to perform ML without the need for data movement and replication. This brings several benefits related to security, execution speed, business context preservation and compliance aspects. For more information about those benefits check out this blog post.

Ok, let's move to the data science tasks and model training.

Model Training

During this phase Data Scientists experiment and iteratively develop the ML model. Most Data Science experts have their preferred platform for ML prototyping. The HANA Machine Learning Python package, called hana-ml, can be used with any Python IDE available. The development environment can be either deployed manually by the Data Scientist or hosted centrally on a dedicated platform. The following blog posts show examples how HANA Machine Learning code can be developed using different platforms: Azure ML and Databricks.

Already during training and experimentation, MLflow plays an important role. It helps evaluate the progress and log the details of each experiment run for later reference. Several algorithms from the hana-ml package (e.g. Automated* or Unified* methods) support the automatic logging of model key performance indicators during training. This is seamless for the user and uses the same interface as open source frameworks. It allows to track hyper parameters, model performance KPIs and also log training activities with usernames and timestamps for auditibility. For more technical details about these features please refer to the second part of our blog post.

Model Deployment

Once a suitable model has been selected and trained, it needs to be deployed. For our example with HANA Machine Learning, we will do this in two steps. In the first step, hana-ml is used to store the artifacts into the built-in model repository of SAP HANA Cloud. In the second step the model is exposed via an API to be consumed by other applications. This can be achieved in several ways, but a lean approach is to use a deployed Flask application (e.g. on SAP Business Technology Platform). To see the details on this process please refer to this blog post.

Model Performance Tracking

In addition to being able to track experiments while training the model, also tracking of the model performance after deployment is important - e.g. in order to track prediction quality and also detect effects like data drift, etc. Some information about those concepts can be found in this blog post.

In our example we will achive the monitoring of the model during operations as follows: Since our model is deployed and exposed via a Flask application, as proposed above, in the application code we use the mlflow package to log incoming data as well as predictions. This allows us to run validation tests (compare actual vs. predicted) once validation data becomes available.

Let's now review how to detect deterioration in model performance and how to perform retraining.

Model Re-Training

Model retraining could be either scheduled or triggered based on a condition or an event. There are several ways how this can be achieved, including automation flows, like SAP Build Process Automation, AirFlow (https://airflow.apache.org/) or Kubeflow (https://www.kubeflow.org/), or by simple helper applications deployed by the user.

For the sake of simplicity in our case we use a simple application deployed on SAP Business Technology Platform (e.g. on Cloud Foundry), which can schedule model retraining runs (e.g. if new training data becomes available regularly). This same application can periodically run a check on the model performance via the APIs from mlflow as described in the previous section. And if there is any performance deterioration (e.g. there is a high deviation in predicted vs actuals) a new retraining run can be performed. And as discussed already, during the new training run the model parameters will be logged via hana-ml and mlflow. In addition, the model will be updated in the model repository in SAP HANA Cloud.

This closes the cycle for the example ML pipeline in the first section. Let's put all pieces together to see the big picture.

Key takeaways

In this blog post we showcased an example conceptual and architectural blueprint on how to realize MLOps pipelines using SAP HANA Machine Learning and the open-source framework MLflow. We discussed the end-to-end process and the advantages of integrating these tools to streamline the machine learning lifecycle, especially in the part of model lifecycle management. To see the technical details and example code to achieve the described steps please refer to the second part of the blog series here. Happy reading!

What’s New in SAP Datasphere Version 2024.10 — May 7, 2024

2024-05-09T15:55:33.407000+02:00

SAP has released version 2024.10 of SAP Datasphere. Enhancements in administration, data integration, data marketplace and data modeling.

There are 3 data integration, 2 data modeling, 1 administration, and 1 data marketplace enhancements in version 2024.10 of SAP Datasphere.

Administration

Photo by Patrick Tomasso on Unsplash

1 For resilience purposes, we are now limiting the maximum CPU resources for workload generated by spaces, user group users and agent users to 80%. The remaining 20% of CPU resources are reserved to ensure that the system can respond under heavy load.

You can now configure the total amount of threads that each space can consume up to a maximum of 100% of this 80% of the threads available in your tenant.

Also, statements are now queued when the CPU usage reaches 90% (instead of 80%) of the database capacity. This value, which you cannot change, applies to all spaces, including spaces created before this version. Set a Priority and Statement Limits for a Space

It’s all about how you manage resources in Space Management > Workload Management. Good luck!

The ADMISSION CONTROL QUEUE CPU THRESHOLD parameter is set to 90% and cannot be changed.

The TOTAL STATEMENT MEMORY LIMIT parameter is set to 80% by default. You can change it by entering the maximum number (or percentage) of GBs of memory that concurrently executing statements can consume in the space.

Data Integration

Photo by Manuel Mnvx on Unsplash

1 For transformation flows, if the source table is not a delta capture table, you can now switch delta capture on for the target table. The system adds the delta capture columns Change Date and Change Type to the target table. Add or Create a Target Table

It’s a handy addition to the transformation flows. Yes, if the source table is not a delta, then the default value of the Delta Capture property for the target table is Off. Now you can enable delta capture and add delta capture columns Change Date and Change Type to the target table.

2 If email notification is set up for a task chain, you’ll now receive email notifications if an error occurs during initialization or preparation to run the task chain, before the task chain run actually starts. Creating a Task Chain

You can now set up email notifications when task chains are completed. These options are available:

Send email notification only if the run failed.
Send email notification only when the run completes successfully.
Send email notification when the run completes.

SAP Datasphere — Task Chain Email Notification

3 Make your SAP Analytics Cloud stories available to the world by translating metadata from SAP Datasphere’s Translation tool. With the Translation privilege, you can access the dashboard and translate metadata such as business names and column names for dimensions and analytic models, and hierarchy dimension labels for stories to a wide range of languages. You can translate manually or via an XLIFF file, but also manage already existing translations, and update them. Translating Metadata for SAP Analytics Cloud

It’s another handy feature for multilingual environments when using SAP Datasphere with SAP Analytics Cloud stories and reports. Use the translation tool in Datasphere and translate object metadata, and your SAP Analytics Cloud story can be viewed in the language of your choice.

Data Marketplace

Photo by Raul Gonzalez Escobar on Unsplash

1 You can now duplicate existing data products. Duplicating a Data Product

It’s simple. My Data Products > Duplicate Product and make necessary changes and Save.

Data Modeling

Photo by Josh Olalde on Unsplash

1 We have improved the user experience:

You can now copy existing measures to create a new measure.
For measures, attributes and associated dimensions there is now a Select All/Unselect All checkbox.

Thanks again for handy features!

2 The Data Preview shows only the first 1,000 rows to increase performance. You can use filters to display the data relevant to you.

Okay, that works.

References

What’s New in SAP Datasphere May 7, 2024. Version 2024.10
What’s New in SAP Datasphere Version 2024.10— May 7, 2024 at medium.com/@tncykarc

Datasphere – Delta Extraction

2024-05-13T13:54:30.330000+02:00

Optimizing Dataloads requires delta functionality.

Here a How-To for a Generic Delta-like extractor within DSP.

Prerequisite is a ‘last change date/timestamp’ or a ‘unique document counter’ in the source view, based on which the delta will be determined.

In DSP two components will be used:

The Delta table, which contains the ‘Extractor’ and ‘Last Extraction’ Date. This date provides the info to the Extractor itself on what to load – the Delta. It always loads data from the Last Extraction Date till Today. The Delta Table is general for all Extractors.
Actual Extractor, from where we load the Delta data. The Extractor has several steps which include the Extraction View, Datapshere transparent table and Extractor Delta updater. Each Extractor has its own Task Chain and own 3 Task Chain steps.

Objects Used:

Type	Usage
Table	Extractors and Last Extraction Dates
Extractor	Task Chain controls the actual Extractor
SDA Table	Remote Table
View	Data Load View used for the Delta Extraction
DSP Table	Local Table used to persist all data
Data Flow	Flow pushes the Delta data into Local table
Transformation Flow	Flow updates the Last Extraction Date

How To:

Add new entry into the Delta Table

Create Delta View with selection using the entry from Delta Table

Create Local Table (type FACT so it can be used in Queries, Analytical Models) with all Dimensions as Key

Create Task Chain with 3 Steps:

First Step is the Data Load View persisting the Delta Data (using objects from #2 and #3)
Second Step is Data Flow loading data from Delta View to Local Table
Third Step is Transformation Flow updating the Delta Table

#1 Delta Table

Use: Extractors and Last Extraction Dates

Type: Local Table

Semantic Usage: Dimension

Key: Extractor

Fields: Date, Extractor Description, Extractor Type, Upper and Lower Safety Interval and a Last Changed timestamp

#2 Delta View with selection using the entry from Delta Table

Use: Loading the Delta Data

Type: View

Semantic Usage: Relational Dataset

Key: Not relevant / none as used for the Delta Data only

View uses an SDA/Remote source

Example: SELECT … FROM "[SDA/Remote table with change date/timestamp]"

AND "[delta_field]" >= ( SELECT "DATE" FROM "[DELTA Table]" WHERE "EXTRACTOR" = '[extractor_name]' )

The Remote Table is used as the interface to Source System. The table has to have a Delta pointer. Either a Date, TimeStamp or Document Counter, what can be used to calculate the Delta.

Type: SDA Table

Semantic Usage: Relational Dataset

Key: Not relevant / none as used for the Delta Data only

#3 Local Table (type FACT so it can be used in Queries, Analytical Models) with all Dimensions as Key

Local table is holding all extracted data. To be used for Query-like Views or Analytical Models. Important is to set the Key to all inbound fields / dimensions including texts.

Type: Local Table

Semantic Usage: Fact

Key: All Fields/Dimensions (including Texts, …) except Key Figures

#4 Task Chain:

The Extractor is controlled via a Task Chain:

View providing the Delta Data from the Extractor is based on the Delta table (View is persisted)

The Delta Data (Persisted data) are then moved to a local (FACT) table with UPSERT (Data Flow)

Delta Date is updated after success (Transformation Flow)

Data Flow

Pushing the Delta Data from Delta View into the Local Table holding all the data

Type: Data Flow

Uses:

Delta view persisting the Delta data

Local Table holding all data

Transformation flow

Updates the Last Extraction Date of the Extractor

Type: Transformation Flow

Uses (same object as source and as a target): Delta Table for the Delta Extraction

Technovation Summit 2024: Expand your AI and BTP Horizons

2024-05-13T15:21:27.786000+02:00

This week the SAPinsider Technovation Summit will commence its inaugural event, with the topics of Artificial Intelligence (AI) and SAP Business Technology Platform (BTP) the stars of the show. And following on just a few short months from the last SAPinsider EMEA conference, it will be great to see if the buzz and excitement around these two topics is just as palpable in Barcelona as it was in Copenhagen.

Technovation Summit 2024 is a 1.5-day event designed for anyone interested in AI or SAP BTP and offers an agenda rich in informational sessions and learning opportunities for any technology enthusiast seeking to expand their knowledge and understanding of the development, application and best practices in these two hot topics. And in addition to hearing from a wide range of speakers at the event, there will also be ample opportunity to network and dive into deeper technology discussions with technology experts and industry peers.

Barcelona of course is a wonderful city location for an event like this. After spending a long day inside, absorbing all the sessions that one can fit into a day, building your knowledge and expanding your horizons of the potential that AI and BTP has to offer your business, what better place could there be to then relax and reflect on the day by taking the time to visit the beach, enjoy a city walk, or unwind and eat Tapas in one of the many tavernas on Las Ramblas, while marveling at the diverse and unique architecture.

6 of the Best for Barcelona

The trouble with attending any event, is selecting which sessions to see that will prove to be of most value to you, especially when there are so many opportunities to learn new things. My advice is simple, try and plan as much in advance, and create your personal agenda as early as possible, so that when you get to the event you can spend less time planning and more time networking and seeking answers to the burning questions that are most important to you and your business.

So why don’t I help a little and start you off with some suggestions for your agenda. Let’s simply assume that everyone will plan to attend the day-1 and day-2 keynotes, to be delivered by Iver van de Zand (14 May, 9:30am) and Prof. Marek Kowalkiewicz (15 May, 9:30am) who will share their own insights and experiences of AI, and in the case of Iver BTP too, setting the scene and the tone for each day of the summit. But after the keynotes, what next? Here are my personal 6 picks for Barcelona:

Generating the Future: SAP Business AI and the Generative AI Roadmap, 14 May, 10:45am to 11:30am
AI Ethics – A Primer to Guide your Organization on the Opportunities and Risks, 14 May, 11:45am to 12:30pm
How to Automate Document Streams using SAP BTP and Machine Learning, 14 May, 1:30pm to 2:30pm
Case Study: Moving From On-Premises to The Cloud with SAP Integration Suite, 14 May, 3:00pm to 4:00pm
What's on your Mind? Maximize the Potential of SAP BTP & AI, 14 May, 4:15pm to 5:00pm
What's on your Mind? Unveiling the Magic of Digital Transformation, 15 May, 11:45am to 12:30pm

And, if you don’t mind, I’ll take a liberty and add one more to see. This isn’t strictly a session, but it will cover many topics that are BTP and AI related, and this is delivered in the form of a BTP: Technology News Update, which will take place at the SAP booth during the networking break at 10:15 on 14 May. Please head to the booth to hear from our presenter Carles, and then network with the SAP onsite team to ask them any questions that spring to mind. Don’t be afraid to challenge them – they’ll appreciate you spending the time to talk with them!

Of course, these are just a few suggestions, and there are many more sessions available for you to select from the agenda. Why not take a look at the SAPinsider Technovation Summit website and look at the event sessions list, or even better download the SAPinsider2024 event app to your smart device (available on the App Store and Google Play) and to see a full list of available sessions, and to create your own personal agenda.

Continue Your Learning After the Event

While I’m sure you’ll enjoy your experience and learning opportunities in Barcelona, I expect that you’ll also want to continue your own personal learning journey after the event too. In this case why not explore other learning opportunities from SAP. For example, find out how to build software applications, side-by-side extensions, and integrations to and from cloud applications by exploring SAP’s free learning content on SAP BTP. It is made for both integration designers and extension developers from all levels of expertise and will help you stay up to date with the latest SAP BTP innovations. And check out even more role-based learning resources and opportunities to get certified in one place on SAP Learning site.

For those of you travelling to Barcelona this week, I do hope that you find the SAPinsider Technovation Summit to be an informative and stimulating experience. My best wishes for safe travels, and I hope that you enjoy the event!

Chris Grundy

Product Marketing, SAP BTP

Watch the SAP BW Modernization Webinar Series

2024-05-13T15:49:24.460000+02:00

Step into the future with our SAP Business Warehouse Modernization Webinar Series. Designed with the goal of revolutionizing the way you approach your SAP Business Warehouse (SAP BW) with SAP Datasphere, this series provides you with an unparalleled opportunity to learn, grow, and take your SAP BW environment to the next level.

Each session comes packed with valuable information, transformative guidance, and real-life insights from seasoned SAP BW and SAP Datasphere experts. The series includes:

Industry-Expert Guidance: Our sessions are hosted by technical experts from SAP, offering you the unique opportunity to learn from those shaping the future of business data technology.
Practical Tools and Applications: We'll demonstrate the SAP Readiness Check, the SAP Expert Evaluation Tool, and SAP BW Bridge to ensure you have a robust toolbox.
In-Depth Case Studies and Demonstrations: We’ll share how other customers have successfully modernized their SAP BW environment. Each session concludes with thorough demonstrations that you can apply in your practice.

Access the webinar series

The overall length of the seven units is close to 7 hours. We have divided the recordings into several parts of 2 to 20 minutes for ease of use. Find the materials below:

Unit 1: Modernizing SAP Business Warehouse, Explore Your Options

Lesson 1: Exploring Transition Option to SAP Datasphere – part 1 , PDF |part 2 , PDF
Lesson 2: Embedded Analytics in SAP S/4HANA Cloud - video , PDF
Lesson 3: Modernization Customer Stories - video , PDF

Unit 2: Preparation for Modernization – Your Source System

Lesson 1: Preparation for Modernization - video , PDF
Lesson 2: Exploring the Transition from BW7.x and BW4 HANA Systems to BW Bridge Conversion - video , PDF
Lesson 3: System Analysis Tooling - video , PDF
Lesson 4: Additional Resources - video , PDF

Unit 3: Using SAP Tooling for a Smooth Transition

Lesson 1: Exploring Tools for Evaluating SAP BW System Readiness for Transition to SAP Datasphere - video , PDF
Lesson 2: Understanding the Conversion Paths to SAP Datasphere and SAP Datasphere, BW Bridge - video , PDF
Lesson 3: Leveraging SAP Services for Successful Deployment of SAP Datasphere - video , PDF
Lesson 4: Expert Evaluation Report - video
Lesson 5: Assessment Tools by SAP Partners - video , PDF

Unit 4: Sizing Matters for the Cloud

Lesson 1: tools and methods for sizing of SAP Datasphere - video , PDF

Unit 5: Preparation for Modernization: Your target system

Lesson 1: SAP Datasphere, BW Bridge as your target system - video , PDF
Lesson 2: Strategic Directions and Options for Data Warehousing Solutions - video , PDF
Lesson 3: Exploring the Value and Features of SAP Datasphere - video , PDF
Lesson 4: Additional Resources - video , PDF

Unit 6: A Deep Dive on Transition Approaches

Lesson 1: Remote conversion and shell conversion options - video , PDF
Lesson 2: Metadata and Business Data Transfers in SAP Datasphere BW Bridge - video , PDF
Lesson 3: Transition Approaches - Demo - part 1 | part 2
Lesson 4: Additional resources - video , PDF

Unit 7: BW Modernization Series - Working with SAP Business Warehouse Elements in SAP Datasphere

Lesson 1: Best practices for working with SAP Business Warehouse elements in SAP Datasphere - video , PDF

Get Started

We’ve designed the series to offer a comprehensive overview, from the basics of starting your SAP BW modernization journey to successfully implementing the transition.

Don’t miss the opportunity to prepare your operations, strategies, and mindset for the future with the SAP BW Modernization Webinar series. Take the first step towards empowering your business' digital transformation journey with the first lesson.

S/4HANA Public Cloud Integration with SAP Datasphere

2024-05-14T05:54:48.426000+02:00

Create a Connection between SAP S/4HANA Public Cloud and SAP Datasphere

1. Introduction

In the fast-paced digital landscape, businesses seek synergy between technology solutions to thrive. SAP S/4HANA Public Cloud offers robust ERP capabilities, while Datasphere empowers data-driven insights. Together, they form a dynamic partnership, driving innovation and efficiency.

In this blog, we unravel the connection between SAP S/4HANA Public Cloud and Datasphere. From integration strategies to real-world applications, discover how this convergence revolutionizes operational excellence and strategic decision-making.

Join us on a journey towards unlocking the full potential of this transformative partnership, where data-driven insights fuel unprecedented success in the digital era.

To make the same steps you must have enough permissions on the S4 side and on the Datasphere side.

2. Basic Logic and Setup

We will setup the Connection to work with the S4 data in the Datasphere. To make sure that you have enough permissions to do that connection check the following steps:

On the Datasphere side you must have the permissions to setup a Connection with a System.

On the S4 side make sure that you have access to these apps.

If you have all these requirements, then you are to go with the blog. 🙂

3. Setup the Communication Settings in the S4

To build up the Connection between the S4 and Datasphere we must enable on the S4 side the communication between the S4 and other Systems. The first app that we we need to open is the App “Communication Arrangements.”

After that you can click on “New” to create a new Communication Arrangement. Then a pop up will appear. There you can search for the scenario “SAP Datasphere - ABAP CDS Extraction - WebSocket Integration”

After that you will jump in the Communication Arrangement. Then you can create a new communication System. This will be needed to declare which system will be take the data from the S4. For that click on new and enter a Name for your system and then click create.

Now you are in the Communication System. There you must enter the URL of your Datasphere tenant but make sure to remove the https:// of the URL. Your URL should be something like this: “your-datasphere-tenant.location.hcs.cloud.sap”

Then you can scroll down until you reach the options for the Inbound Communication User. There you can click on the + and press the button to create a new User.

You now can create your own user. Enter there a Name and Description for the User and then you can enter your own password, or you can propose own. After you have done it you can click on Create.

After you have created your Inbound Communication User you will be send back to the Communication System App. You will see that you User has been selected and you can confirm it by clicking “OK”.

Now you can scroll down to the Outbound Communication User. There you can click on the + to create a new User. In the Pop up you can select the Authentication Method of the User. There you can select “None” and then you can create the User.

Now you can save your communication System.

Now you are there where you have started. Select on the Authorization Group ID the Value “SAP_DSP_ALL” to have access to all Views from the S4.

Click on save and then you have setup everything on the S4 side.

4. Make the Connection to S4 in Datasphere

Now that we are back in the Datasphere we can create the connection to the S4. Go to the Connections and click on “Create” there you select the Connection “SAP ABAP.”

Then you can enter your credentials:

Protocol: Web Socket RFC

SAP Logon Connection Type: Application Server

Application Server: your-s4-tenant-api.s4hana.cloud.sap

Port: 443

Client: 100

Username: YourUsernamefromtheInboundConnection

Password: YourPasswordfromtheInboundConnection

Then you can click on “Next Step”

Enter a Name for the Connection and click on “Create Connection”.

The connection is created. If you want to check, if the connection has worked. You can select the Connection and click on the Validate button. There your connection will be validated.

If it has worked, you will see this message:

Now you can work with the S4 Views in the Datasphere. 🙂

SAP Datasphere - Space Management

2024-05-14T10:37:35.395000+02:00

This blog helps you getting started with SAP Datasphere. After getting Datasphere tenant, we need defining space for team to define connection, Data access mode, Model and views.

Navigate to given Icon / Space management:

Choose the Menu option ‘Create’:

Provide the Name of the space:

Allocate Disk and Memory Storage:

Workload Management:

Set the appropriate priority based on space usage.

Users:

Add new users to space by clicking on the Menu option ‘ Add’:

*Scope need to defined for the users before they can be added to a space.

Database Access:

Check the tick box for ‘Expose for Consumption by Default’ , if you want to expose data stored on this space for reporting.

Database Users:

Create space database user to connect to space at DB layer or for external tool like PowerBI:

provide username and select the option as required.

Auditing:

We can keep Logs for Read and Change operation by default 7 days as required by the Audit team.

Multiple space can be created based on the requirement.

SAP BW bridge – SAP Cloud connector configuration

2024-05-14T10:40:01.329000+02:00

Get the sub account and location ID created on SAP DataSphere. Sub-account can be created by following the documentation at below link: https://help.sap.com/docs/SAP_DATASPHERE/9f804b8efa8043539289f42f372c4862/6de74f7731c54ce88f2883df8f8671a8.html

(SAP DataSphere –> System –> Administration-> Data Source Configuration)

On SAP Cloud connector, add the sub account and location ID created in SAP DataSphere:

fill the required information and save.

Next, get the information of BW Bridge server from SAP DataSphere (Connections -> BW Bridge)

On SAP Cloud connector, add SAP BW Bridge as ‘On-Premise To Cloud’ by select option as below.

Click on '+' Symbol to add new server:

Provide the input as shown below: (Type - ABAP Cloud System and Description about server)

In next screen, input Host address, Instance No., and No# of connections and Click on Finish.

BW Bridge is added to SAP Cloud connector and we can see status as 'GREEN' as shown below:

SAP BW Bridge – Integrating with On-Premises ECC / BW servers

2024-05-14T10:47:37.351000+02:00

To begin with, first we need to add on-Premises server to SAP Cloud connector.

Go to the path: Sub-account -> Cloud To On-Premise

Click on ‘+’ symbol to add new server.

In Add System Mapping, select Back-end Type as ‘ABAP System’

Select protocol as ‘RFC’.

Choose the option with / without load balancing, since mine is single instance server, I have chosen second option.

Enter the application name as FQDN “<host>.<domain>.com” and server instance no.,

Give a virtual name of the server, that will be used as reference in SAP BW Bridge.

Provide meaningful description.

Check the box ‘Check internal Host’ and click on finish.

Check the server status, it will show as reachable.

Download the resources for R/3 On-premises server from below snote: 3112568

Upload the resources for newly added server by clicking on up arrow.

In the Import Scenario, browse and select the zip file for ODP resources.

Once imported status will be ‘Green’ and function name will be like below.

In Communication Management select ‘Maintain communication Users’

Select ‘New’ communication user to be added.

Provide the Name and description of the user. For password either you can provide complex password or select random generated password with option of ‘Propose Password’. Click on create.

A new user ID will be created for the given user.

Return the Communication Management page and click on Communication Systems.

Select the ‘New’ option to create new communication server

Provide the reference name and system ID and click on create.

For the new system, under Technical Data under general, provide the server’s name and port no., as created in SAP Cloud connector.

Switch ON the cloud connector and provide the location ID.

For RFC settings, provide target host name as virtual host name in SAP cloud connector, for Client – 100 and Instance no# 00.

Add the user for Inbound communication

Select the user created as communication user and click OK.

Similarly for the outbound communication provide the user ID created in R/3 server.

Save the configuration.

Return to Communication Management and select “Communication Arrangements”.

Select new communication arrangement.

Select the Scenario as ‘SAP_COM_0692’ (SAP BW Bridge – ODP RFC Source system Integration) and give a name and click on ‘Create’.

Select the newly created Communication system.

Select the backend system ID as created in eclipse tool as source system.

Validate the Inbound and Outbound user and click on save.

Create the RFC connection of type ‘3 – ABAP ‘ as shown below.

RFC Destination : ‘Own SAP Cloud system’ name of BW bridge.

Target host: SAP Cloud connector FQDN (without https and port no#).

Instance no# 00

Under logon & Security enter the Inbound User ID & Password created for Communication user and save the settings.

Click on Test RFC connection and it will show connected.

We have successfully added backed R/3 server to SAP BW Bridge.

SAP DataSphere – SAP HANA Database Monitoring

2024-05-14T10:49:02.161000+02:00

Login to SAP DataSphere and navigate to path System -> Configuration. Select the Menu option 'Database Access' -> Database Analysis Users.

Select the option Create for new DB user.

Provide the Name of the user along with No# of days the user will be active ( after which the user ID will get deleted) and click on 'Create'.

HAND DB Analysis User will be created and access details will pop up in next screen. Please make not of User Name, Host, Port and Password.

We can see the newly created user under 'Database Analysis Users' along with its status and Validity date.

To Access the SAP HANA Cockpit, check the User Name and click on 'Open SAP HANA Cockpit'

This will take us to new webpage/ tab for HANA DB Cockpit and prompt us to enter the Login details.

After successful login we land on the Database Overview page of SAP DataSphere HANA DB.

Here we have option to select different view from the drop down menu.

Select Monitoring the see the overall status of the SAP HANA DB. Now, we will explore some key monitoring artifacts like Storage, CPU, Threads, Transactions, etc.,.

1. Storage Utilization will provide statistic of used disk space against total allocated space of the SAP DataSphere tenant.

Analyze Workloads will provide details view of 'Top SQL Statements', 'Background Jobs', 'Timeline' & active 'Threads'.

We also have option to select the time period and Filter the view.

Background jobs will show the status of background job, which were schedule for the selected period.

Under 'Timeline', we can monitor different application threads along with duration and status.

In Menu option 'Threads', we can select the different monitoring option for the process threads and see the overall performance. Here I've selected option as 'Thread State', we can also select different option for more detailed view.

It also has option to select 'Secondary Dimension' and monitor the statistics based on selected criteria.

2. Manage Services, will show the overall status of core HANA DB services for the SAP DataSphere tenant.

we can see the service status, Database SID and overall status. Below image shows HANA DB services are running with one high priority alert with in the 'indexserver'. We can also access 'indexserver' trace file by selecting option ' Go to Trace file'.

3. Memory Usage: This menu option provide us high level view of Memory utilization.

To access detailed information on memory utilization, click on option ' Monitoring Performance', This will show us detailed statistical view. we can see memory allocation to different DB components and services.

with in the performance Monitor, we also have option to view performance metrics CPU & Memory as well.

More detailed view of Memory usage with respect to Workloads, Buffer Cache and Paging can also be accessed by selecting the option for menu (...).

4. Monitoring Table usage: We can access table space usage statistics from 'Monitoring'

we can monitor the tables and its space utilization and statistics with different spaces.

5. Blocked Transactions: Performance of HANA DB get severely impacted if we have any blocked transaction at the DB level.

We can see the active transaction in blocked status, if any.

6. CPU Usage: We can monitoring overview of CPU utilization and details view can be access with options like 'Monitoring Performance' and 'Analyze Workloads'.

In case we observe any issue, which requires SAP attention than we need to open a support ticket with SAP to get the required help.

SAP Datasphere – Architecture and Security Concept

2024-05-15T11:52:30.219000+02:00

SAP Datasphere (DSP) design differs from classic BW and therefore, a new approach is needed.

Space is the point of entry. Consider Spaces as Info Areas. Creating spaces and assigning users to specific spaces allows easier security maintenance and more straight forwards consumption. Row-Level data security is still applied however.

Roles Used:

Role	Description	Space
System Architect	System and Cloud Administration	All
Data Architect	Knows inbound data from all sources	Central
Solution Architect	Works on specific reporting requirements	Functional

Spaces Used:

Space	Description
Central	Connections, Data Persistency, DACs, Time and Conversion Tables, Private-Like Views and Reusable-like Views for sharing with Functional spaces
Security	Security Tables
Functional	Reusable and Query-like views for consumption
CSV (optional)	For CSV Uploads (if necessary for key users / like BW-Workspace)

Access Used:

Role	Space	Access Type
System Architect	All Spaces	Full
Data Architect	Central	Full
Data Architect	Functional	Read
Solution Architect	Central	Read
Solution Architect	Functional	Full
Security Team	Security	Full
Support Team	Central	DIM only

Security:

Security Tables will be placed in a separate Space to ensure, that only specific users have access to them, but the DACs are built and assigned in the Central Space.

Central:

All connections – full control of all connections to all source systems
Persist the data (with Task Chains) – full control of the quota
Assign DAC (based on shared tables from the Security Space) to the persisted data / views / remote tables
Conversion (TCUR*, T006*) tables will be shared as Views
Time Tables can be shared directly or as Views
Central Space shares Reusable-Like views to Functional spaces.

Functional:

Only functional relevant (virtual) object
No data persisted
Possible to expand to Branche/Country Spaces. Proper technical name and self-explaining business name is necessary
Functional Spaces use shared reusable views from the Central space.

CSV:

Purely optional and only used if it is necessary to upload any CSV data (BW-Workspace like)
Separate space as the functional spaces should not persist any data

Transformation Flow partition generation with SAP Datasphere CLI

2024-05-15T17:16:21.756000+02:00

You want to partition a Transformation Flow in SAP Datasphere, to reduce the memory consumption? That’s what we discuss in this blog post and provide the Python code to realize this. It makes use of the SAP Datasphere Command Line Interface (CLI) to read and write Transformation Flows, and to write a Task Chain that holds them together as one.

Recently I tried loading a large amount of data from the SAP Datasphere BW Bridge into SAP Datasphere Core, using a Transformation Flow in Delta Mode. As you might know, the Bridge runs its own SAP HANA database, separate from the SAP HANA database that SAP Datasphere “Core” runs. When using a Transformation Flow to load from Bridge to Core, the data transfer goes over SDA. Both Core and Bridge were minimally sized, so I needed to be careful with the memory consumption of the operation. And that is what you can usually achieve with partitioning, so that the data is transferred in smaller packets using a pre-defined set of criteria. However, Transformation Flows do not support partitioning yet.

The alternative? Creating multiple Transformation Flows, each with a WHERE clause in the SQL definition, effectively creating your own partitioned Flows. And since I didn’t feel like creating 20 of this manually, I thought this would be nice to give the CLI another swing. And some python code of course.

The original Transformation Flow

The original Transformation Flow is rather straightforward: we read from one Delta (remote) Table and write it into another Delta (local) Table. In this case we are writing from and to a Local Delta Table, which means that the Transformation Flow is in Delta Mode. Below you see the screenshots of the main work area and the source SQL definition.

Figure 1: The original un-partitioned Transformation Flow

Figure 2: The SQL definition of the source node

SELECT "Change_Type",
   "Change_Date",
   "ID",
   "SOMESTRING"
FROM "UC2_DLT_SRC_Delta"

What we want to accomplish is to copy this Transformation Flow for each partition that we want to create, add a WHERE clause to the SQL definition, and bind these Flows together in one Task Chain.

The code (and running it)

I’ve uploaded the code into this Git repo: https://github.com/SAP-samples/btp-global-center-of-excellence-samples/tree/main/DSP_object_generation. The repo is shared with a few other code samples, only the folder “DSP_object_generation” is needed for what we achieve in this blog. The folder also contains the code for Remote Table to View generation, which is described in a previous blog.

Requirements:

Python to run this. I developed this using 3.10.6.
The latest SAP Datasphere CLI. I used 2024.9.0.

To run it:

Create a dsp_logon_data.json file in the root of the folder. You can use “dsp_logon_data.json.template” as a template. The hdb* parameters do not have to be set for transformation flow generation, they only apply to the previous blog where we read remote table definitions from the database.
Edit the parameters in main.py:
- Technical space name
- Technical transformation flow name (this one should have been created already)
- Column to partition on (in my example this is a String column)
- Partition definition (in my example these are Strings)
Run main.py

The output

For each partition, a Transformation Flow is generated, as you can see in below screenshot. Also depicted is the WHERE statement added to one of the generated Transformation Flows. Besides that, a Task Chain is generated which runs all generated Transformation Flows in sequence.

Figure 3: An overview of the generated objects

Figure 4: The WHERE statement added to the generated Transformation Flow SQL definition

Figure 5: The generated Task Chain that binds all generated Transformation Flows together

Figure 6: Running the generated Task Chain

Conclusion

This blog post describes how to use the SAP Datasphere CLI and object generation using Python to partition a Transformation Flow. This is merely another example of what you can achieve with the CLI and some coding. This example might help you directly but applies to a rather straightforward Transformation Flow where we only want to add a WHERE clause, repeat this for each partition, and bind them together with a Task Chain. In anyway, I hope you enjoyed the read and are inspired to build something yourself.

SAP Datasphere Multilingual Support - Translation Dashboard

2024-05-16T09:39:56.336000+02:00

Introduction

The SAP Datasphere Multilingual Support Series is intended to provide you with useful guidance on how to utilize the new multilingual translation dashboard. The translation dashboard allows you to translate large amounts of metadata in SAP Datasphere that can be leveraged by the SAP Analytics Cloud to display the stories or dashboard in several available languages.

This article is the second in the blog post series on multilingual capabilities in SAP Datasphere.

Blog Post #1: Master Data Translations in SAP Datasphere

Blog Post #2: Meta Data Translations using the Translation dashboard in SAP Datasphere

Prerequisites

To manage translation, you must be assigned to a scoped role that inherits a template (such as DW Space Administrator), which grants the Translation privilege.

Understading the Translations Workflow

The below diagram clearly represents the activities that needs to be performed by different roles to enable, create and consume translations

Steps to setup and consume Translations

Enable the Space for Translation

In the Main Menu, click (Space Management) and Select a space.
In the Overview tab, in the General Settings section, check the Enabling Translation box. Select a language as Source Language. For more information on available languages, see Supported Data Access Languages in SAP Analytics Cloud

Note:

Translation toggle cannot be disabled and the source language cannot be changed after the space is deployed.

2. Select Analytic Models to Translate in the Translation Dashboard

The user with the Translator privilege is able to launch the Translation Dashboard. All the spaces enabled with the Translation will be visible in the translation dashboard. Select the required space and you would be able to add the analytic model

Click + (Add). The Add Objects dialog opens. Select the objects you want to translate. Click Add Objects.

The selected Analytic Models and their dependencies are added to the tool.

Note

Only Analytic Models can be selected in the Add Objects dialog, but the dimensions and hierarchies attached to added Analytic Models are automatically selected to the Translation tool.

3. Provide Translations for required languages.

The Translator can translate the metadata using two methods:

a. Translate Metadata via XLIFF Files

You can translate large amounts of metadata with the help of XLIFF files.

Download

In the Translation tool, select at least one entity that you want to translate. Click (Download) to download the XLIFF files.

You can choose to select All Strings or Outstanding Strings while exporting the XLIFF file(s). The Outstanding Strings option is enabled only for partially translated objects. The XLIFF file is downloaded to the default location in your system.

The downloaded XLIFF file contains the source locale and the list of text that needs translation. You can use this XLIFF file in any external translation tool to generate the translated content as a separate XLIFF file per locale.

Upload

Once the translation is done, click (Upload) to open the Import Translations dialog. Click Upload Files to upload the translated XLIFF files back into the Translation tool and Click Import to upload the XLIFF files.

You can import multiple XLIFF files. Each XLIFF file contains the translated content in one language, which is automatically fetched from the language code embedded in the name of the uploaded XLIFF files. These XLIFF files can be uploaded together since they belong to one object.

Note: Name the file in the format <Analytical Model Name>_<Language (2 char)> as shown below,

b. Translate Metadata Manually

You can view the source and the translated text in the Translation tool, and add or edit the translated text in-line.

Select the object you want to translate manually and click (Edit).

The list of source text appears. Choose the Target Language and start adding the translated text in the empty field and save.

4. Consume Model with Data Access Language

Once the translations have been maintained for the Analytic Model in the Translation Dashboard in SAP Datasphere, you can see that the Analytical Model Preview displays the data in the Data Access Language.

Similarly, users can access the SAP Analytic Cloud stories built on top of this Analytic Model in one of the available Data Access Languages defined in Translation Dashboard.

Managing Translations

Different Status of Translations per object - To see the status of the translation of an object per language, click on the object. A dialog appears with the list of translated languages and corresponding status in which the object was translated.

Translated: Once all strings are translated completely, the status is shown as Translated.
Not Translated: For a new object, or for objects whose translation is expired or deleted completely, the status is shown as Not Translated.
Partially Translated: When the strings in an object are not translated completely, the status is shown as Partially Translated. Once it is translated again, the status changes back to Translated.

Analytical Model Updated with New Columns - If the model has changed, new strings will automatically appear in the translation tool as Not Translated. Expired strings will still be available in the Translation tool. You can upload the XLIFF files even if there is a mismatch in the HTML tags between the source and target language strings.
Deleting Translations - To delete translations, select one or more objects in the Translation tool and click (Delete). Deleting translations removes the object and attached translated strings from the Translation tool. You'll be able to add the object back to the tool, but no translation will be attached to it any longer.

Summary

Now you should be able to create and consume Translations for metadata in Analytic Models in SAP Analytic Cloud Stories.

Please like, comment or post a question!

SAP Datasphere Multilingual Support - Master Data Translation

2024-05-16T10:22:57.119000+02:00

In this blog we will provide you the overview of Master data translation. We will walk you through each case and how it can be visualized in the Analytical Model Preview as well.

This article is the first in the blog post series on multilingual capabilities in SAP Datasphere.

Blog Post #1: SAP Datasphere Multilingual Support - Master Data Translations (this blog)

Blog Post #2: Translating Metadata using the Translation Dashboard in SAP Datasphere

Introduction

Master data text is the descriptive information associated to the respective master data. For example, customers, products, vendors, materials and their hierarchies can have their independent description in their respective text table. In general, the master data or fact data entity has only IDs. In the analytical representation, if the textual data gets fetched it adds more value and additional context to the better understanding.

In SAC stories, displaying the product description rather than their id's gives a better understanding to the visuals.

Master Data Translation within SAP Datasphere can be accomplished through various methods.

Semantic mapping from an attribute within entity
Text Association
Dimension Association
Hierarchy Association

Figure1: Dependent entities of the Analytical Model

Figure2: Text/Dimension Associations in the entity – SALES (fact)

1.Semantic mapping from an attribute within entity

Here Product Id attribute fetching the corresponding description from the attribute Description of the same fact. When an attribute is marked as semantic type Text, it can be referenced in another attribute for Text/Association. Description with Semantic type Text is referenced at Product ID for Text/Association as mentioned in Figure2.

Figure3: Analytical Model Preview - Fetching the description from the same entity.

2. Text Association

Customer ID attribute has the Text association to table Description.

Figure4: SALES – Description Text Association

Based on the Data Access Language the description will be retrieved. Here I have German (Deutsch), and the texts are fetched for the same.

Figure5: Data Access Language Settings

To get the descriptions, we need to enable the ID and Description or Description from Presentations.

Figure6: Analytical Model Preview - Retrieving the description via Direct text association.

3. Dimension Association

Purchase Date is associated to the Time Dimension – Day.

Figure7: Purchase Date – Time Dimension - Day Association

Figure8: Time Dimension – Day, Here the associated Text tables are highlighted.

Since the day Dimension has a text association for months, the description for months is fetched accordingly (refer Figure10) based on the data access language settings.

Select the Hierarchy like below and enable the display of ID and Description from Presentations settings.

Figure9: Hierarchy Selection

Figure10: Analytical Model Preview - Retrieving the description via Dimension text association.

4. Hierarchy Association

SAP datasphere has released the external hierarchy with directory support. Here the language and texts for Hierarchy label and the Node can be retrieved as well. This blog has already addressed the same here - An Introduction to Hierarchy with Directory in SAP Datasphere

Summary

Text tables serve as the core of master data translation. While importing master data, ensure that if you are using the ABAP language codes like E or D, as compared to i18n language codes like de or en, the string length for the language column is set to 1. And this ensures the effective translation of master data if it has ABAP language codes. A cleaner data without any text duplications is recommended for the effective translation.