SAP Community - Big Data

5 Steps to a Business Data Fabric

2023-10-17T19:31:56+02:00

Managing and leveraging data can be a daunting task. Businesses grapple with complex datasets from many different and unconnected sources, including operations, finance, marketing, customer success, and more. Plus, a lot of organizations are geographically dispersed and have complicated use cases or specific needs, like storing data across cloud, hybrid, multi-cloud, and on-premises devices.

A business data fabric offers a solution.

This data management architecture provides an integrated, semantically-rich data layer over underlying data landscapes to deliver scalable access to data without duplication. In other data platforms, when data is extracted from core systems, much of its original context is lost. A business data fabric preserves this context, helping ensure the data remains meaningful and relevant for decision-making, regardless of its origin.

This approach offers a number of benefits including enhanced data accessibility, improved data governance, and accelerated insights.

But how do you get started?

Implementation framework

Let’s look at a high-level framework for implementing a business data fabric architecture in your organization.

1. Data Ingestion

The first step is to ensure that all your data, whether it’s structured or unstructured, can be easily ingested into the system. A business data fabric, with its open data ecosystem, allows for simple data ingestion, regardless of the source or the format of the data.

2. Data Integration

Data from various sources must be integrated and transformed into a unified format easily consumed by data users. The interoperability of a business data fabric enables data from different sources to be combined and connected rather than being moved around.

3. Data Governance

With the growing complexity and volume of data, governance becomes an increasingly important topic. This includes ensuring data quality, privacy, and compliance with various regulations. A business data fabric ensures effective governance by maintaining metadata, lineage, and control measures.

4. Data Cataloging

This involves creating an inventory of data assets and their metadata. The catalog serves as a single source of truth for users to find, understand, and trust the data they need. It’s a critical component of the business data fabric that allows data consumers to understand the business semantics.

5. Data Consumption

This is about delivering the right data, in the right format, at the right time, to the right people. The business data fabric supports data federation, which enables unified and consistent access to data across diverse sources, reducing redundancy. It ensures data is presented in business-friendly terms and contexts, making it simple for data consumers to interpret and use the data for their specific use cases.

This is where SAP provides the foundation for a business data fabric: SAP Datasphere.

Transform your organization with SAP Datasphere

SAP Datasphere is a comprehensive data service that empowers users to provide seamless and scalable access to mission-critical business data.

It makes it easy for organizations to deliver meaningful data to every data consumer with business context and logic intact. As organizations need accurate data that is quickly available and described with business-friendly terms, this approach enables data professionals to permeate the clarity that business semantics provide throughout every use case.

In a major moment for our industry and customers, SAP is partnering with other open data partners — Databricks, Collibra, Confluent, DataRobot, and Google Cloud — to radically simplify customers’ data landscapes. By closely integrating their data and AI platforms with SAP Datasphere, organizations can access their mission-critical business data across any cloud infrastructure.

SAP Datasphere, and its open data ecosystem, is the technology foundation that enables a business data fabric.

Learn more

Read the new e-book to learn more about the practical applications of a business data fabric, including:

Why you need a business data fabric

How to implement a business data fabric

Five business data fabric use cases

Get started with the Five Steps to a Business Data Fabric Architecture e-book today.

Why Business Data is Fundamental to Artificial Intelligence

2023-10-30T23:09:56+01:00

The introduction of cloud computing has enabled organisations all over the world to store vast amounts of data in a cost-effective way as they digitally transform their business operations. Data has commonly been referred to as the 'new oil' and is where companies are looking to help increase their productivity going forward. However, in order to harness this technology the data needs to be relevant, reliable and responsible. As the adage goes, garbage in, garbage out.

AI serves as a powerful tool for extracting actionable insights from the vast amount of reliable data generated and stored within SAP systems. Combining AI with SAP BTP, advanced data analytics and machine learning algorithms becomes possible, allowing organisations to tap into the potential of their SAP data along with the AI technologies available in the market.

Five pillars support SAP BTP: App development, automation, integration, data analytics, and AI. These pillars interplay with embedded intelligent technologies like situation handling, machine learning, and analytics, all fully integrated within the SAP S/4HANA Cloud. Furthermore, side-by-side capabilities through SAP BTP offer additional intelligent industry functionalities like Intelligent Situation Automation, SAP Build Process Automation, and chatbot technology.

By harnessing artificial intelligence, SAP BTP combines business data from S4 with external data, enabling the creation of increasingly precise models in real-time. This ensures a versatile and agile platform that propels innovation while retaining a clean digital core. This demonstrates a shift from traditional systems of record to systems of intelligence.

AI-Powered Capabilities

SAP has embedded AI into its products for many years, from journal reconciliations in S4 to AI-powered writing assistants aimed to streamline HR-related tasks in Success Factors.

These innovative functionalities are not merely theoretical but are practically applicable, ensuring HR admins, managers, and employees can operate more efficiently. In fact, these innovations exist across all the functions in your SAP landscape such as procurement, finance, and human resources.

How to get started?

Automation is pivotal for managing manual and repetitive tasks, especially those involving the consolidation and manipulation of data from diverse sources like MS Excel, vendor portals, and SAP systems. High-volume processes, often exceeding 1000 steps a day—such as data migrations and approvals—and those requiring access to multiple applications, can be streamlined, ensuring seamless operation across your SAP environment.

SAP has provided templates across all the business functions to accelerate these initiatives, as shown below.

A More Advanced AI Use Case

The transition from a rules-based approach to an AI-empowered, data-driven model is illustrated through an example case study of an Australian customer. A decade ago, an employee scripted manual “if:then” statements for road upgrades; a process that has now been revolutionised by AI. AI can now analyse these rules and infuse them with real-time data like weather, road usage, and vehicle types. As a part of their operations, this customer assesses road conditions using specialised trucks called profilometers, generating colossal data volumes that outpace their storage capacities. SAP BTP, however, can house this data in expansive lakes, giving AI the agility to model exponentially precise “if:then” statements.

The shift will allow this customer to manage large datasets from disparate sources seamlessly, scaling memory and compute capabilities to handle big data without losing granularity. Moreover, unlike fixed rules, the AI algorithms continually evolve based on data, thereby ensuring maintenance and road upgrade strategies that are timely, relevant, and efficient.

In the realm of road maintenance, AI’s practical application is manifest, where even a small percentage improvement can result in significant savings for this customer. This financial efficacy, combined with the potential to extend the useful life of assets, underscores the tangible, impactful benefits of combining AI with SAP BTP.

In Summation

Early AI integration can offer businesses a decisive advantage. SAP’s AI vision isn’t just about pioneering technology; it's about tangible, real-world applications. From simple tools deployable within days to intricate endeavours with broad impact.

If you’d like to find out about the value AI can bring to businesses through automation and explore other use cases, then visit the SAP Business AI website.

SAP and DXC team plan to deliver RISE with SAP S/4HANA Cloud, in customer data centers and co-location facilities, creating a new and powerful platform for digital transformation

2023-11-06T18:39:34+01:00

SAP and DXC aim to deliver RISE with SAP S/4HANA Cloud, private edition, customer data center option as a turn-key service delivered by DXC. The new service is ideally suited for Private Cloud customers and other managed services customers who wish to run SAP either from their own data center or a DXC managed data center and get the transformational benefits of RISE with SAP.

In this blog, DXC reaffirms its commitment as a Partner Managed Cloud (PMC) service provider of RISE with SAP by expanding its distinct deployment capabilities already announced with DXC Hyperscaler solutions to support the customer data center option.

This partnership empowers DXC and SAP to offer a comprehensive catalog of managed services and extraordinary opportunities that surpass what each entity could achieve independently, benefiting our mutual customers.

RISE with SAP S/4HANA Cloud, private edition, customer data center option or “CDC” represents the hybridization of SAP’s strategic RISE cloud solution, where a customer can run S/4HANA as a cloud service from their data center; while accessing SAP BTP and SAP Signavio, and all the other innovative RISE components from the public cloud – aka a mixture of internal and external cloud services. For more information about CDC, please visit this dedicated web page: Customer data center option | RISE with SAP

Similar to SAP’s cloud solutions, CDC provides diverse deployment options on Lenovo, HPE and Dell infrastructure. DXC enhances this offering further by enabling delivery of SAP and non-SAP Infrastructure as a Service (IaaS), platform as a service (PaaS), and software as a service (SaaS), all backed by a top service level agreement.

With DXC, SAP’s strategic RISE Cloud offering deployable in Customer Data Centers, is specifically designed to:

Enhance RISE with SAP by harnessing DXC’s expertise in delivering SAP and non-SAP managed services and enabling mutual customers to become more agile and alleviate the challenges often associated with digital transformation projects. This is achieved through best-of-breed solutions and a wealth of experience and skills.

Establish a secure infrastructure and platform for SAP within the customer data centers, backed, managed, designed to perform by SAP and DXC.

Empower customers who want to run in their data center (on-premise) while staying aligned with SAP’s cloud innovation agenda, including ML/AI (Machine Learning / Artificial Intelligence), LLMS (large language models), etc.

Meet the needs of industries with strict regulatory compliance requirements that may prevent running SAP in a public shared Hyperscaler such as utilities, public sector, healthcare, pharmaceuticals, aerospace & defense, etc.

Provide extended services to address specific data sovereignty needs of customers, governments, and industry stakeholders by keeping sensitive data within their national boundaries, governed by local laws.

Offer an innovative approach for those seeking a cloud OpEx model while benefiting from a high-performance dedicated onPrem system with minimal latency.

Benefit from a dedicated on-premise setup, without the data center environment having to be managed by the customer.

To learn more about the DXC & SAP cloud solutions and the DXC Premier Services for RISE with SAP, please visit the following link: DXC Premier Services for RISE with SAP

All thoughts and questions are welcome, please share your comments below to contribute to this discussion.

Joseph Zarb
Head of RISE with SAP – Customer Data Center
SAP RISE Global GTM Execution
10 Hudson Yards, 51st Floor, New York NY 10001 USA

j.zarb@sap.com

How to connect SAP ECC + S/4HANA on-prem and private cloud with Confluent Cloud or Confluent Platform (1)

2023-12-01T15:27:08+01:00

This blog post explains how to connect SAP ECC or S/4HANA on-prem and private cloud editions with Confluent Cloud or Confluent Platform.
Whether you're using SAP NetWeaver Event-enablement Add-on or ASAPIO Integration Add-on, this step-by-step guide provides all you need as a first step to enable SAP systems for communication with a Confluent broker.

Architecture / Connection types

When connecting SAP with Confluent, it is important to understand that there are two very different approaches in terms of connection architecture.

Using a REST Proxy

"Confluent REST Proxy for Kafka" or a similar product is the standard approach currently and therefore can be considered mandatory for the connectivity.
Please see https://github.com/confluentinc/kafka-rest for details.

Reason for using REST instead of AMQP is, that SAP ECC does not support streaming protocols and 3rd-party-libraries cannot be used in SAP-certified Add-ons.

SAP-to-Confluent Architecture

Direct connect/V3:

A Connector for Confluent Cloud direct connect (V3 REST API) is available as pre-release in the download section for registered ASAPIO customers. This connector can only handle outbound connectivity at the time of this blog being published (November 2023). For inbound connectivity, the REST proxy approach above is still required.

AMQP support:

AMQP support is planned to be released by ASAPIO in 2024, for S/4HANA systems.

System prerequisites

Before diving into the integration process, make sure you have the following components available:

Software components required on your SAP system

SAP NetWeaver Event-enablement Add-on (SAP Event Mesh Edition)

or, alternatively, ASAPIO Integration Add-on - Framework (full version)

ASAPIO Connector for Confluent (using REST proxy)

For direct connect/V3 REST API, a pre-release Connector for Confluent Cloud is available for registered ASAPIO customers.

Non-SAP components

Please make sure you have endpoint URI and authorization data at hand for:

Confluent Components:
- Confluent REST Proxy for Kafka (More info)
- Confluent Cloud
- Or, alternatively Confluent Platform

Licensing

All software above requires the purchase of appropriate licenses.

Set-up Connectivity

1. Create RFC Destinations to Confluent REST Proxy

Transaction: SM59

Type: "G" (HTTP Connection to External Server)

Target Host: Endpoint of Confluent REST Proxy

Save and perform a "Connection Test" to ensure HTTP status code 200.

2. Set-up Authentication to REST Proxy

Pre-requisites: Obtain user and password for the REST proxy or exchange certificates with the SAP system.

Transaction: SM59

Choose the correct RFC destination, go to "Logon & Security," and select authentication method:

"Basic Authentication" with username and password

SSL certificate-based authentication

3. Set-up Basic Settings

Activate BC-Sets: Use SCPR20 to activate BC-Sets for cloud adapter and codepages.

Configure Cloud Adapter: In SPRO, go to ASAPIO Cloud Integrator, Maintain Cloud Adapter, and add an entry for the Confluent connector.

4. Set-up Connection Instance

Transaction: SPRO or /ASADEV/68000202

Add a new entry specifying connection details, RFC destination, ISO code, and cloud type.

5. Set-up Error Type Mapping

Create an entry mapping response codes to message types.

6. Set-up Connection Values

Maintain default values for the connection to Confluent in Connections -> Default values.

Default Attribute	Default Attribute Value
KAFKA_ACCEPT	application/vnd.kafka.v2+json
KAFKA_CALL_METHOD	POST
KAFKA_CONTENT_TYPE	application/vnd.kafka.json.v2+json (or application/vnd.kafka.jsonschema.v2+json)

Set-up Outbound Messaging

1. Create Message Type

Transaction: WE81

Add a new entry specifying a unique name and description for the integration.

2. Activate Message Type

Transaction: BD50

Activate the created message type.

3. Set-up additional settings in 'Header Attributes'

Configure the topic, fields for the key, and schema IDs for key/value schemas.

4. Set up 'Business Object Event Linkage'

Link the configuration of the outbound object to a Business Object event.

Send a "Simple Notifications" event for testing

1. Create Outbound Object Configuration

Transaction: SPRO or /ASADEV/68000202

Select the created connection and go to Outbound Objects.

Add a new entry specifying the object, extraction function module, message type, load type, and response function.

2. Test Outbound Event Creation

In the example above, please pick any test sales order in transaction /nVA02 and force a change event, e.g., by changing the requested delivery date on header level.

3. Check monitor transaction for actual message and payload

Access to monitor application
User must have PFCG role /ASADEV/ACI_ADMIN_ROLE to access Add-On monitor.

Use transaction /n/ASADEV/ACI_MONITOR to start the monitor.
You will see the entry screen with a selection form on top.

Congrats, you are now able to send data out to Confluent.

In the next blog, we will create a custom payload for the event.

Event-driven architecture: Simplifying Payload Creation with Payload Designer

2023-12-11T08:42:28+01:00

Overview

Configuring payloads for the SAP NetWeaver Add-On for Event enablement has become remarkably straightforward, all thanks to the Payload Designer.

This powerful tool enables you to effortlessly add tables, define relationships with inner joins, left outer joins, rename tables and fields, all through simple configuration.

In this blog post, we will guide you through the process of configuring your payload with just a few easy steps.

If you you are new to SAP Enterprise Messaging in SAP ERP systems and the Integration Add-on, you can have a look at following Blog-Posts:

Event-driven architecture – now available for SAP ECC users

SAP Enterprise Messaging for SAP ERP: HowTo-Guide (Part 1 - Connectivity)

SAP Enterprise Messaging for SAP ERP: HowTo-Guide (Part 2 - First use case)

Data Events scenario With SAP Event Enablement Add-on for SAP S/4HANA, SAP Event Mesh and SAP Cloud Integration: Step-by-Step Guide

Emit Data Events from SAP S/4HANA or SAP ECC through SAP NetWeaver Add-On for Event Enablement

How to connect SAP ECC + S/4HANA on-prem and private cloud with Confluent Cloud or Confluent Platform (1)

System Prerequisite

One of the Software components needs to be availabvle on your system:

SAP NetWeaver Event-enablement Add-on (SAP Event Mesh Edition)

or, alternatively, ASAPIO Integration Add-on – Framework (full version)

Licensing

All software above requires the purchase of appropriate licenses.

Creating Custom Payloads with Payload Designer

Step 1: Creating Custom Payloads with Payload Designer

1. Navigate to transaction /n/ASADEV/DESIGN.

Click the "Create Payload Designer" button on the main screen and fill in the necessary fields. This action creates the initial version of the payload and takes you to the main screen.

Creating a Payload Designer version in transaction /n/ASADEV/DESIGN

2. Use the join builder to establish table joins.

Insert new tables or custom views.

Adjust table joins through field connections.

Return to the main screen.

Note: Parent relationships in the Table section and key fields in the Field section are automatically determined based on hierarchical sorting.

3. Add additional payload fields from the tables:

Double click on the preferred table.

Select one or multiple fields.

Fields can be reordered using sequence numbers.

Step 2: Outbound Configuration Using Payload Designer

To configure outbound objects using Payload Designer, follow these steps:

Access transaction SPRO.

Navigate to IMG > Cloud Integrator – Connection and Replication Object Customizing or directly to transaction: /ASADEV/68000202.

Select the created connection.

Go to the "Outbound Objects" section.

Add a new entry and specify the following:
- Object: Name of the outbound configuration.
- Extraction Func. Module: /ASADEV/ACI_GEN_PDVIEW_EXTRACT.
- Load Type: Incremental Load.
- Trace: Activate for testing purposes.
- Formatting Func.: /ASADEV/ACI_GEN_VIEW_FORM_CB.
- Field Payload View Name: Payload Name.
- Field Payload View Version: Payload Version.

Step 3: See your Payload in the ACI_Monitor

Navigate to transaction /n/ASADEV/ACI_MONITOR

Conclusion

Payload Designer simplifies SAP interface configuration by providing a user-friendly, code-free approach to defining payloads. With its intuitive interface and powerful features, it enables organizations to streamline their data integration processes and improve efficiency in managing payloads for event messages sent to the SAP Event Mesh.

SAP HANA Cloud, data lake Files への最初のアクセス設定

2023-12-19T07:37:51+01:00

このブログは、2022 年 11 月 15 日に SAP ジャパン公式ブログに掲載されたものを SAP ジャパン公式ブログ閉鎖に伴い転載したものです。

このブログは、jason.hinsperger が執筆したブログ「Setting Up Initial Access to HANA Cloud data lake Files（2021 年 8 月 5 日）の抄訳です。最新の情報は、SAP Community の最新ブログやマニュアルを参照してください。

SAP HANA Cloud, data lake は、フォーマットのあらゆるタイプのデータのネイティブ形式でのストレージをサポートしています。

マネージドファイルストレージは、外部のハイパースケーラーアカウントでストレージを設定することなく、あらゆるタイプのファイルをセキュアに格納するストレージを提供します。

これは、高速 SQL 分析を行う目的で SAP HANA Cloud, data lake にデータを高速投入する必要がある場合や、何等かの目的でデータを extract する場合にとても便利です。

SAP HANA Cloud, data lake Files への初回のアクセス設定は、特にデータベースのバックグラウンドを持ち、オブジェクトストレージや REST API に詳しくない場合には少し難しいプロセスかもしれません。

以下は、私が SAP HANA Cloud, data lake files をテストするのに使用したプロセスです。

SAP HANA Cloud, data lake files はユーザーセキュリティーやアクセスを認証経由で管理するため、ユーザーアクセスの設定には署名付きの証明書の生成が必要です。

認証局へのアクセスがない場合には、OpenSSL を利用する以下のプロセスを使用してCAと署名付きのクライアント証明書を作成してSAP HANA Cloud, data lake files 設定を更新することができます。
私はこれまで何度もこれでテストしたことがあるので読者の方でも同様に行えるでしょう。

最初に、CA バンドルを作成してアップロードする必要があります。

以下の OpenSSL コマンドを使用して CA を生成できます。

openssl genrsa -out ca.key 2048

次に、CA の公開証明書（この場合は 200 日間有効）を作成します。共通名を最低限入力し、他のフィールドを必要に応じて入力します。

openssl req -x509 -new -key ca.key -days 200 -out ca.crt

クライアント証明書の署名リクエストを作成する必要があります。共通名を最低限提供し、他のフィールドを必要に応じて入力します。

openssl req -new -nodes -newkey rsa:2048 -out client.csr -keyout client.key

最後に、クライアント証明書を作成します（この場合は 100 日有効）。

openssl x509 -days 100 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt

*備考 – CA とクライアント証明書のフィールドがすべて全く同じにならないようにしてください。さもないと、自己署名証明書とみなされ、以下の証明書の認証が失敗します。

証明書が任意の CA によって署名されたことを認証するには（SAP HANA Cloud, data lake に CA 証明書をアップロードしたときにクライアント証明書を認証するために使用できるとわかるように）

openssl verify -CAfile ca.crt client.crt

次に、SAP HANA Cloud Central でインスタンスを開き、「Manage File Container」を選択し、SAP HANA Cloud, data lake files ユーザーを設定します。

設定を編集し、「Trusts」セクションの「Add」を選択します。前に生成した ca.crt をコピーまたはアップロードし、「Apply」をクリックします。すぐには「Manage File Container」スクリーンはクローズしないでください。

これで、管理されたファイルストレージにアクセスできるようにユーザーを構成できるようになりました。

「Authorizations」セクションをスクロールダウンして、「Add」を選択します。新しい入力欄が表示されます。

ユーザーのロールをドロップダウンリストから選択します（デフォルトでは admin とユーザーロールがあります）。

ここからが少し難しいところです。

リクエストしたときに、ストレージゲートウェイ（SAP HANA Cloud, data lake files へのエントリーポイント）がどのユーザーに対して認証するのか決定できるようクライアント証明書からパターン文字列を追加する必要があります。

パターン文字列を生成するにあたり、2 つのオプションがあります。
以下の OpenSSL コマンドを使用して、パターン文字列を生成することができます(アウトプットに表示される「subject= 」　プレフィックスは省略します) 。

openssl x509 -in client.crt -in client.crt -nameopt RFC2253 -subject -noout

あるいは、スクリーンにある「generate pattern」オプションを使用することもできます。
これは、ダイアログボックスを開き、クライアント証明書をアップロード/貼り付けて、自動でパターンを生成します。
証明書は保存せず、パターン文字列だけを保存することに注意してください。

「Apply」をクリックして、権限入力欄にパターン文字列を追加します。

パターン文字列は、ワイルドカードも使用可能なため、特定のロールの証明書のクラスを認証できることに注意してください。証明書のパターンが複数の認証と一致する場合、使用する認証は、特定の認証エントリーに設定された「Rank」値セットによって制御されます。

これで、REST api 経由で SAP HANA Cloud, data lake files にアクセスして使用することができます。

私のテストではうまくいったcurl コマンドのサンプルがあります。接続が成功しているかどうかvalicate できます。(インスタンス ID とファイル REST API エンドポイントは HANA Cloud Centralのインスタンス詳細からコピーすることができます)。

上記で生成して認証の作成に使用したクライアント証明書とキーを使用してください。

curl は少し tricky なことに注意してください。Windows で試していましたが、Windows 10 バージョン用の curl を動作させることができませんでした。最終的に新しい curl version (7.75.0) をダウンロードしたところ、機能しましたが、Windows で curl から証明書ストアへどうアクセスするのかわからなかったため、SAP HANA Cloud サーバー証明書の認証をスキップするために –insecure’ オプションを使用しなければなりませんでした。

curl --insecure -H "x-sap-filecontainer: <instance_id>" --cert ./client.crt --key ./client.key "https://<Files REST API endpoint>/webhdfs/v1/?op=LISTSTATUS" -X GET

上記のコマンドは、以下を返します (空のSAP HANA Cloud, data lake)。

{"FileStatuses":{"FileStatus":[]}}

これで、SAP HANA Cloud, data lake files を使用して、あらゆるタイプのファイルを SAP HANA Cloud に格納するための設定は終了です。

ファイルの管理でサポートされている REST API と引数のフルセットについては、マニュアルを参照ください。

オリジナルのブログはここまでです。

SAP HANA Cloud, HANA データベースから SAP HANA Cloud, data lake リレーショナルエンジンへの最速のデータ移動方法と移動速度テスト結果

2023-12-19T07:47:55+01:00

このブログは、2022 年 11 月 17 日に SAP ジャパン公式ブログに掲載されたものを SAP ジャパン公式ブログ閉鎖に伴い転載したものです。

このブログは、douglas.hoover が執筆したブログ「The fastest way to load data from HANA Cloud, HANA into HANA Cloud, HANA Data Lake（2022 年 3 月 8 日）の抄訳です。オリジナルのブログページでのコメントのやりとりなどもぜひご参照ください。

最新の情報は、SAP Community の最新ブログやマニュアルを参照してください。

このブログは、SAP HANA データ戦略ブログシリーズの1つです。
https://blogs.sap.com/2019/10/14/sap-hana-data-strategy/

概要

SAP HANA Cloud の HANA データベースから SAP HANA Cloud, data lake リレーショナルエンジンにより大きなテーブルを移動するお客様が増えるにつれ、SAP HANA Cloud の HANA データベースから SAP HANA Cloud, data lake リレーショナルエンジンへのデータ移動の最速の方法について聞かれるようになりました。

より正確にいうと、SAP HANA Cloud, data lake リレーショナルエンジン仮想テーブルに対してシンプルに HANA INSERT を実行するよりも高速な方法があるのか聞かれるようになりました。

なぜ SAP HANA Cloud の HANA データベースから SAP HANA Cloud, data lake リレーショナルエンジンに大きなテーブルを移動するお客様がいるのか疑問に思う方もいるかもしれません。

最もよくある利用ケースは、大きなデータベースの最初のマテリアライズあるいは古いデータをSAP HANA Cloud, data lake リレーショナルエンジンにアーカイブするためです。

これらのお客様の大半は、通常 SAP HANA Smart Data Integration (SDI) を使用してこのマテリアライゼーションを行っており、これらのテーブルを最新の状態にキープするために SDI の Flowgraphs や SDI のリアルタイムレプリケーションを使用した Change Data Capture を同じインターフェースを使用して行っています。

SAP HANA SDI の詳細については以下のブログを参照してください:

SAP HANA データ戦略: リアルタイム Change Data Capture を含む高速データ投入（英語）

https://blogs.sap.com/2020/06/18/hana-data-strategy-data-ingestion-including-real-time-change-data-capture/

SAP HANA データ戦略: 高速データ投入 – 仮想化（英語）

https://blogs.sap.com/2020/03/09/hana-data-strategy-data-ingestion-virtualization/

ここで実験するデータ移動に関するシンプルな方法は以下の 3 種です:

シンプルに SAP HANA Cloud, data lake リレーショナルエンジン仮想テーブルへの HANA INSERT

HANA 仮想テーブルにアクセスし、シンプルに SAP HANA Cloud, data lake リレーショナルエンジンから data lake INSERT

HANA エクスポートと data lake LOAD

こう質問する人もいるかもいれません：

「なぜ SAP HANA Cloud, HANA データベース経由で行うのか?」

「なぜ SAP HANA Cloud, data lake リレーショナルエンジンに直接データをロードしないのか?」

繰り返しますが、これらのお客様はターゲットとして HANAオブジェクト（ローカルまたは仮想）を必要とする HANA Enterprise Information Management （EIM）ツールを使用しています。

将来のブログでは、SAP IQ クライアントサイドロード、Data Services、Data Intelligence 経由の SAP HANA Cloud, data lake リレーショナルエンジンへの直接のデータロードについて説明したいと思います。

SAP HANA Cloud, HANA データベースから SAP HANA Cloud, data lake リレーショナルエンジンへの最速のデータロード方法は、SAP HANA Cloud, data lake リレーショナルエンジンから、SAP HANA Cloud, HANA データベースの物理テーブルを指定するプロキシテーブルを作成するために「create existing local temporary table」を使用して HANA テーブルから SELECT で INSERT 文を実行する方法です。
（詳細は以下のテーブル参照）

方法	行	データサイズ	時間（秒）
HANA Cloud, data lake/IQ INSERT..SELECT	28,565,809	3.3 GB	52.86
*HANA Cloud, data lake/IQ LOAD Azure ファイルシステム	28,565,809	3.3 GB	116 (1分56秒)
*HANA Cloud, data lake/IQ LOAD Data Lake ファイルシステム	28,565,809	3.3 GB	510 (8分30秒)
HANA INSERT..SELECT	28,565,809	3.3 GB	1277 (21分7秒)

* HANA データベースからファイルシステムへのデータエクスポート時間は含めていません。

28,565,809 行、約 3.3 GB の TPC-D ORDERS テーブルを使用し、SAP HANA Cloud, data lake リレーショナルエンジンの小さめの設定でロードしています。

以下の SAP HANA Cloud 設定を使用してテストしました

SAP HANA Cloud, HANA データベース：60 GB / 200 GB、4 vCPU

SAP HANA Cloud, data lake リレーショナルエンジン：16 TB、ワーカー 8 vCPU /コーディネーター 8 vCPU

SAP HANA Cloud, data lake リレーショナルエンジンでは、より多くの並列処理を実行するには（特により大きなテーブルの場合）より多くの vCPU 数を設定します。

より多くの TB を SAP HANA Cloud, data lake リレーショナルエンジンに追加することで、より大きなディスク I/O スループットを得ることができます。

テストで使用した詳細設定と構文

SAP HANA Cockpit をスタートして SAP HANA Cloud を管理します。

SAP HANA Cloud, data lake リレーショナルエンジンから「Open in SAP HANA Database Explorer」を選択します。

もしこれが初回であれば、SAP HANA Cloud, data lake リレーショナルエンジンの ADMIN パスワードを求められます。

SQL コマンドを入力し、クリックして実行します。

以下を作成するための SAP HANA Cloud, data lake コマンド

SAP HANA Cloud, data lake リレーショナルエンジンから、SAP HANA Cloud HANA データベースへ接続しているサーバー

データをロードして作成するためのローカルの SAP HANA Cloud, data lake リレーショナルエンジンテーブル

SAP HANA Cloud インスタンスのテーブルを指定するローカルのテンポラリープロキシーテーブル

CREATE SERVER

–DROP SERVER DRHHC2_HDB

CREATE SERVER DRHHC2_HDB CLASS ‘HANAODBC’ USING ‘Driver=libodbcHDB.so;ConnectTimeout=60000;ServerNode=xyxy.hana.prod-us10.hanacloud.ondemand.com:443;ENCRYPT=TRUE;ssltruststore=xyxy.hana.prod-us10.hanacloud.ondemand.com;ssltrustcert=Yes;UID=DBADMIN;PWD=xyxyx;’

CREATE TARGET TABLE

CREATE TABLE REGIONPULL (

R_REGIONKEY   bigint                  not null,

R_NAME            varchar(25)        not null,

R_COMMENT    varchar(152)      not null,

primary key (R_REGIONKEY)

);

CREATE local temporary PROXY

create existing local temporary table REGION_PROXY (

R_REGIONKEY   bigint                  not null,

R_NAME                          varchar(25)        not null,

R_COMMENT    varchar(152)      not null,

primary key (R_REGIONKEY)

)

at ‘DRHHC2_HDB..TPCD.REGION’;

INSERT DATA

INSERT into REGIONPULL SELECT * from REGION_PROXY;

Commit;

–1.9s

ORDERS テーブルテストコマンド

–DROP TABLE ORDERSPULL;

create table ORDERSPULL (

O_ORDERKEY           BIGINT               not null,

O_CUSTKEY            BIGINT               not null,

O_ORDERSTATUS        VARCHAR(1)           not null,

O_TOTALPRICE         DECIMAL(12,2)        not null,

O_ORDERDATE          DATE                 not null,

O_ORDERPRIORITY      VARCHAR(15)          not null,

O_CLERK              VARCHAR(15)          not null,

O_SHIPPRIORITY       INTEGER              not null,

O_COMMENT            VARCHAR(79)          not null,

primary key (O_ORDERKEY)

);

create existing local temporary table ORDERS_PROXY (

O_ORDERKEY           BIGINT               not null,

O_CUSTKEY            BIGINT               not null,

O_ORDERSTATUS        VARCHAR(1)           not null,

O_TOTALPRICE         DECIMAL(12,2)        not null,

O_ORDERDATE          DATE                 not null,

O_ORDERPRIORITY      VARCHAR(15)          not null,

O_CLERK              VARCHAR(15)          not null,

O_SHIPPRIORITY       INTEGER              not null,

O_COMMENT            VARCHAR(79)          not null

)

at ‘DRHHC2_HDB..TPCD.ORDERS’;

INSERT into ORDERSPULL SELECT * from ORDERS_PROXY;

Commit;

–59s

–52.86 s

SELECT COUNT(*) FROM ORDERSPULL;

–28,565,809

LINEITEM テーブルテストコマンド

create table LINEITEM (

L_ORDERKEY           BIGINT               not null,

L_PARTKEY            BIGINT               not null,

L_SUPPKEY            BIGINT               not null,

L_LINENUMBER         INTEGER              not null,

L_QUANTITY           DECIMAL(12,2)        not null,

L_EXTENDEDPRICE      DECIMAL(12,2)        not null,

L_DISCOUNT           DECIMAL(12,2)        not null,

L_TAX                DECIMAL(12,2)        not null,

L_RETURNFLAG        VARCHAR(1)              not null,

L_LINESTATUS         VARCHAR(1)              not null,

L_SHIPDATE           DATE                 not null,

L_COMMITDATE         DATE                 not null,

L_RECEIPTDATE        DATE                 not null,

L_SHIPINSTRUCT       VARCHAR(25)          not null,

L_SHIPMODE           VARCHAR(10)          not null,

L_COMMENT            VARCHAR(44)          not null,

primary key (L_ORDERKEY,L_LINENUMBER)

);

create existing local temporary table LINEITEM_PROXY (

L_ORDERKEY           BIGINT               not null,

L_PARTKEY            BIGINT               not null,

L_SUPPKEY            BIGINT               not null,

L_LINENUMBER         INTEGER              not null,

L_QUANTITY           DECIMAL(12,2)        not null,

L_EXTENDEDPRICE      DECIMAL(12,2)        not null,

L_DISCOUNT           DECIMAL(12,2)        not null,

L_TAX                DECIMAL(12,2)        not null,

L_RETURNFLAG         VARCHAR(1)              not null,

L_LINESTATUS         VARCHAR(1)              not null,

L_SHIPDATE           DATE                 not null,

L_COMMITDATE         DATE                 not null,

L_RECEIPTDATE        DATE                 not null,

L_SHIPINSTRUCT       VARCHAR(25)          not null,

L_SHIPMODE           VARCHAR(10)          not null,

L_COMMENT            VARCHAR(44)          not null

)

at ‘DRHHC2_HDB..TPCD.LINEITEM’;

INSERT into LINEITEM SELECT * from LINEITEM_PROXY;

Commit;

— Rows affected:       114,129,863

— Client elapsed time: 4 m 52 s

まとめ

SAP HANA Cloud, HANA データベースから SAP HANA Cloud, data lake リレーショナルエンジンへのデータの最速のロード方法は、SAP HANA Cloud, data lake リレーショナルエンジンから、SAP HANA Cloud, HANA データベースの物理テーブルを指定するプロキシテーブルを作成するために「create existing local temporary table」を使用してHANA テーブルから SELECT で INSERT 文を実行する方法です。

これは、このブログで紹介しているコマンドを使用することで、とても容易に行うことができます。

あるいは、これらのコマンドを生成するプロシージャーを作成すると、さらに容易になります。（下の Daniel のブログを参照してください。）

以下も参考にしてください

Jason Hinsperger の「SAP HANA Cloud, data lakeへのデータロード」のブログでは、SAP HANA Cloud, data lake リレーショナルエンジンの vCPU 数やデータベースサイズを増やすとロードのパフォーマンスにどのような影響があるか説明しています。

https://blogs.sap.com/?p=1866471

Daniel Utvich の「SAP HANA Cloud, HANA データベースから SAP HANA Cloud, data lake へのデータの高速移動」のブログでは、システムテーブル情報をベースにした SQL コードを生成するプロシージャーの例を紹介しています。

https://blogs.sap.com/?p=1867099

SAP HANA データ戦略ブログインデックス

SAP HANA Data Strategy

SAP HANA Data Strategy: HANA Data Modeling a Detailed Overview

HANA Data Strategy: Data Ingestion including Real-Time Change Data Capture

HANA Data Strategy: Data Ingestion – Virtualization

HANA Data Strategy: HANA Data Tiering

オリジナルのブログはここまでです。

Using Data lake and SQL to create custom reporting models

2023-12-20T10:29:19+01:00

Overview: Through a series of blogs, would like to share scripts that utilize data lakes built for SAP tables, to create reporting models that represent certain sections of SAP screens/transactions or areas of analysis. Hopefully, these scripts serve as an accelerator to cater multiple use cases.For this first script we'll look at building User Status using JCDS and JEST.

Background: Most structured reporting tools (eg:BW) or ETL processes don’t bring in all fields available in source systems, these are deployed using a predefined datamodel (dimensions/measures) that collects fields from different tables and limit what’s initially available for reporting, restricting the ability of Analysts to explore additional fields.

Eg: Financial reporting models built using ACDOCA or BSEG or FAGLFLEXA tables- Irrespective of the approach(CDS views or BW models), these don’t bring all fields from the source as they mostly focus on meeting initial requirements from primary stakeholders.

Additional fields maybe available in SAP transaction systems and to make them available for reporting, multiple cycles of enhancements are implemented, reflecting a dependency on different support teams and time involved to meet these requirements.

Solution With a data lake that replicates tables from SAP, Analysts working with functional resources can build models that meet their specific needs. If replications are managed through SAP SLT, then it enables near realtime (possible delay of a few seconds) reporting. Review must be done with functional consultants to ensure that tables being replicated dont have confidential content.

As part of this blog series, we shall see some models that reflect SAP transactions or commonly used reporting metrics.

Factors that are not addressed in this blog but must be considered:

Organization of reporting models and data lake tables, if not using similar reference as SAP Application components. This becomes Important for managing confidentiality and ensuring personal information of customers, employees and vendors is only available to those that need it as part of their business roles.

Security models needed for

Here's the first script:

Script for Plant maintenance object status

Need: Near real time availability of object status’ for Plant maintenance, eg: an emergency order created for addressing critical equipment failure, the status and progress of investigation needs to be communicated through the manufacturing channels for them to manage bottlnecks in production.

Solution: Below layout provides a simplified overview of how different tables are joined together with their respective fields.

Tables used:

JEST-Individual Object Status

JCDS-Change Documents for System/User Statuses (Table JEST)

JSTO- Status object information

TJ02-System status

TJ02T - System status texts

TJ04- Status control for object type

TJ30- User Status

TJ30T- Texts for User Status

Object status tables relationship overview

Script below provides active status’ for all Plant maintenance objects . To view all instances of status changes remove the JEST.INACT is NULL clause/restriction. Each table and the filter condition starts with a comment(begins with --) to show what it represents. May have to tweak formatting based on tool being used, especially the comments section.

SELECT



JEST.OBJNR AS OBJECT_NUMBER,



JSTO.OBTYP AS OBJECT_CATEGORY,



SUBSTR(JEST.OBJNR, 3) AS OBJECT,



JEST.STAT AS OBJECT_STATUS,



(CASE WHEN LEFT(JEST.STAT, 1) = ‘I’ THEN ‘SYSTEM’ ELSE ‘USER’ END) ASSTATUS_TYPE,



(CASE WHEN LEFT(JEST.STAT, 1) = ‘I’ THEN TJ02T.TXT04



ELSE TJ30T.TXT04 END) AS STATUS_SHORT_TEXT,



(CASE WHEN LEFT(JEST.STAT, 1) = ‘I’ THEN TJ02T.TXT30



ELSE TJ30T.TXT30 END) AS STATUS_LONG_TEXT,



JSTO.STSMA AS STATUS_PROFILE,



JCDS.USNAM AS STATUS_CHANGED_BY,



JCDS.UDATE AS STATUS_CHANGED_DATE,



JCDS.UTIME AS STATUS_CHANGED_TIME,



JCDS.CHIND AS STATUS_CHANGED_TYPE,



TJ04.INIST AS SYSTEM_STATUS_INITIAL_STATUS_FLAG,



TJ04.STATP AS SYSTEM_STATUS_DISPLAY_PRIORITY,



TJ04.LINEP AS SYSTEM_STATUS_LINE_POSITION,



TJ02.NODIS AS SYSTEM_STATUS_NO_DISPLAY_INDICATOR,



TJ02.SETONLY AS SYSTEM_STATUS_SET_ONLY_INDICATOR,



TJ30.STONR AS USER_STATUS_WITH_NUMBER,



TJ30.INIST AS USER_STATUS_INITIAL_STATUS_FLAG_INDICATOR,



TJ30.STATP AS USER_STATUS_DISPLAY_PRIORITY,



TJ30.LINEP AS USER_STATUS_LINE_POSITION,



CASE WHEN TJ30.LINEP = ’01’ THEN TJ30T.TXT04 END ASPOSITION1_USER_STATUS



FROM JEST --Individual object status



INNER JOIN JCDS -- Change Documents for System/User Statuses (Table JEST)



ON JEST.OBJNR = JCDS.OBJNR



AND JEST.STAT = JCDS.STAT



AND JEST.CHGNR = JCDS.CHGNR



LEFT JOIN JSTO -- Status profile information for objects



ON JEST.OBJNR = JSTO.OBJNR



LEFT JOIN TJ02T --System status texts



ON JEST.STAT = TJ02T.ISTAT



AND TJ02T.SPRAS = ‘E’



LEFT JOIN TJ04 -- System status control config table 2



ON JEST.STAT = TJ04.ISTAT



and TJ04.OBTYP = JSTO.OBTYP



LEFT JOIN TJ30T -- User status texts



ON JSTO.STSMA = TJ30T.STSMA



AND JEST.STAT = TJ30T.ESTAT



AND TJ30T.SPRAS = ‘E’



LEFT JOIN TJ02 ”System status config table 1



ON JEST.STAT = TJ02.ISTAT



LEFT JOIN TJ30 -- User status config table 1



ON JSTO.STSMA = TJ30.STSMA



AND JEST.STAT = TJ30.ESTAT



WHERE JEST.INACT is NULL -- remove this to see when a status was set inactive or to get timelines for all status

Conclusion : Using the above code we can active status' and their respective times for all operational objects that have been configured for status tracking. Similar approach can be used to get status' for CRM using table CRM_JEST and CRM_JCDS. Remove the inactive filter to get status' that are currently not active (depending on the values are mapped in data lake i.e default value of blanks as NULLs, NULL may need to be replaced with '')

Possible variations based on need:

To plot timeline of how the operational object moved between status' use JCDS

Restrict to certain Status profile(s) in table JSTO when requirement is to focus on certain types of objects or group

Restrict using change date and time if the need is to focus of recent changes within the hour or day(s)

Next blog will look at details of combining details of orders and related operational tasks

Empowering Businesses with New Insights: The Google Cloud and SAP Analytics Partnership

2023-12-28T12:35:19+01:00

A year ago, the tech giants Google Cloud and SAP embarked on a journey to revolutionize data analytics for businesses. Their goal: to bring together SAP systems and data with Google’s data cloud, offering customers better insights for decision making and innovation. The new SAP Datasphere Replication Flow connector for Google BigQuery is now available.

From the outset, customers have been excited about the potential of this partnership: integrating SAP's robust data models and real-time processes with Google BigQuery's comprehensive real-time data streams, including search engine data, weather data, marketing data, and customer event data, to inspire new and better ways to do business.

Prior to this collaboration, businesses found it challenging to merge data models from either side due to cost, effort, and time. This partnership aims to eliminate these hurdles, providing real-time data streams that adjust dynamically to changes from Google Cloud and SAP. Haridas Nair, the Head of Cross Product Management for Database and Analytics at SAP, stated, "Customers using SAP Business Technology Platform can now extend the reach of curated and modeled SAP business data for downstream consumption with SAP Datasphere Replication Flow. The integration with BigQuery now enables customers to combine SAP business data with Google BigQuery data. This enables new use cases that can unleash significant business value."

For example, while enterprises rely on SAP S/4HANA for their financial planning, reporting, and budgeting there are many that also have finance data coming in other systems. Join ventures, new acquisitions or decentralized business models are common cases where finance data will reside in non-SAP S/4HANA systems. Early adopters of such companies are leveraging the Google Cloud and the SAP Datasphere Replication Flow connector to unify accounting data insights into SAP Datasphere to get a single financial dashboard across all their financial sources and thereby enable secure, self-service access to reusable data models, and streamline financial reporting. The result is enhanced analytics that enable new market correlations to transpire, as well as reporting efficiency and reduced data management costs. As such finance and operations experts can receive new insights that improve their business planning.

The other common use case relates to Consumer Products and Retail companies. A prime example is a North American consumer products company selling through retailers as well as their online platform. As many other companies in this space, they're investing in brand loyalty and scaling their product portfolio to target different customer segments. The company strives to corelate online customer trends with their retail channel sales using demographics and other consumer data.

Their business goals involve improving channel inventory turns, trade promotion management, shelf-availability, SKU margins, and overall understanding of customer buying behavior. Simultaneously, they aim to reduce the cost, effort, and time for accessing SAP data and enable richer SAP ERP data in real-time.

To achieve these goals, they have connected Google Cloud and SAP data to gain better insight into their retail channels and to improve their demand forecasting and supply chain algorithms. The more real-time the connectivity between their SAP data and Google BigQuery data, the more confident they'll be in the predictive algorithms they adopt for their supply chain.

But this is just the beginning. The Google Cloud and SAP Analytics partnership opens the door to a wide range of strategic customer and supply chain programs that leverage data for advanced predictive demand models. Early examples of customer innovations include true customer 360 insight, improved sales performance, yield-driven pricing, new product introductions, unifying external accounting with SAP, manufacturing automation, and operationalizing sustainability.

Both of the Google Cloud and SAP development organizations are excited to see that their work is making a difference. Future releases plan to include more advanced features for managing enterprise scale federation, replication and data catalogs between their respective data platforms. Honza Fedak, Director of BigQuery Engineering at Google Cloud stated, "The combination of Google Cloud's data and AI expertise and SAP's deep understanding of business data is a powerful force that can help businesses unlock the full potential of their data."

As more enterprises utilize this partnership, we are confident that the Google Cloud and SAP analytics partnership can provide better insights, enable better decisions, and foster innovation. This partnership is a significant step towards creating more Intelligent Enterprises, and we hope your enterprise will be one of them.

Integration Options for moving data from SAP into Databricks

2024-08-13T19:58:39.925000+02:00

Background

This blog delves into the various methods for integrating data from your SAP systems into Databricks. This exploration is particularly relevant given SAP's recent announcement of SAP Datasphere in March 2023. This collaboration aims to provide businesses with the power of federated AI-driven analytics. This will allow them to effortlessly analyze structured and unstructured data from both SAP and non-SAP sources within a single, unified platform.

However, I am not going to discuss how we integrate SAP systems into Databricks via Datasphere or BW4 HANA as we already have great blogs on datasphere and BW4HANA.

This blog will explore options for migrating data from SAP to Databricks without relying on Datasphere or BW4HANA, even though licensing for transferring SAP data to non-SAP systems might still be necessary.

Integration Options

In this blog, I am discussing 4 different options to move data from SAP into data bricks.

SAP Data Services ETL Integration

We can leverage the popular ETL tool SAP Data Services to move data between SAP and Databricks.

While a direct integration between SAP Data Services and Databricks might not be readily available, you can establish a connection using intermediary stages and leveraging data transfer mechanisms. Here are a few approaches:

File-Based Integration: Initiate the integration by designing and running data extraction jobs within SAP Data Services. These jobs should be configured to export your SAP data in formats readily consumable by Databricks, such as CSV, Parquet, or Avro. Once exported, these files can be seamlessly transferred to a storage service[Ex: Azure Blob Storage or AWS S3, as well as shared file systems] accessible by Databricks.

Database Staging: Optimize your data pipeline by using SAP Data Services to efficiently load extracted and transformed data directly into a staging database readily accessible by Databricks. Suitable options for this staging database include Azure SQL Database, Amazon Redshift, or similar platforms. Once the data is in the staging area, establish a connection between Databricks and the database using Spark JDBC connectors or Azure Synapse native connectors and map the respective tables.

Custom Integration using APIs: Investigate the availability of APIs or SDKs provided by both SAP Data Services and Databricks. Develop custom scripts or applications using languages like Python or Java to extract data from SAP Data Services and transfer it to Databricks using their respective APIs.

SAP SLT Integration

Replicating SAP data to external systems using SAP SLT can be complex, but leveraging HANA as a staging area provides a pathway for efficient real-time replication. By establishing connectivity through JDBC Spark or SDI HANA connectors, you can move data into Databricks for AI based predictive analytics.

Event Based Messaging

Set up SAP BTP Integration Platform to capture real-time data changes from your SAP system, leveraging Change Data Capture (CDC) mechanisms or APIs for seamless data extraction. Then, integrate SAP BTP Integration Platform with a message queue or streaming platform like Apache Kafka or Azure Event Hubs to reliably publish these captured data changes. Databricks can then tap into these data streams using its robust streaming capabilities, subscribing to and consuming the data from the message queue.

This approach empowers you with near real-time data ingestion and analysis capabilities within Databricks. For additional flexibility, consider incorporating HANA Cloud as an optional staging area to further transform and prepare your data before it's loaded into Databricks.

SNP GLUE

SNP Glue is another product that can be used to replicate data from SAP platforms into cloud platforms. While that particular product might have limitations in terms of advanced transformation capabilities, it's essential to investigate its compatibility with other cloud solutions like SuccessFactors and Ariba to ensure a comprehensive integration strategy.

Key Considerations

We need consider the following factors when choosing the right tool :

Data Volume and Frequency: The chosen integration method should align with the volume of data being transferred and the desired frequency of updates.

Data Transformation: Determine whether data transformations are necessary before loading into Databricks and whether these transformations are best performed within SAP Data Services or using Databricks' data manipulation capabilities.

Security and Access Control:Implement appropriate security measures to protect data during transfer and storage, ensuring secure access to both SAP Data Services and Databricks.

Data Latency Requirements: Determine the acceptable latency for data availability in Databricks. The streaming approach offers near real-time capabilities, while the intermediate database approach might involve some delay

As you embark on your SAP-Databricks integration journey, carefully consider your specific needs, data characteristics, and latency requirements to select the optimal approach for your business. With a well-planned strategy and the right tools in place, you can harness the combined power of SAP and Databricks for AI powered federated analytics.

Consuming Data from Datasphere to Azure Data Factory via ODBC

2024-09-18T13:43:15.587000+02:00

Prerequisites:

Access: Access to ADF and Datasphere.
Credentials: Datasphere and ADF credential details.

Connect Datasphere to Azure Data Factory

DATASPHERE PART:

Log in to Datasphere -> Space Management -> Choose the space and select Edit
Click Create and Make sure that you have enabled Expose for consumption by default

Copy the Database Username, Hostname, Port, Password
Go to System-> Configuration-> IP Allowlist-> Trusted Ips
EXTERNAL IPV4 ADDRESS should be added here, not Internal IPV4
To get an External IPV4 Address, use this URL: What Is My IP Address - See Your Public Address - IPv4 & IPv6
Add and Save the External ipv4 address in the Datasphere’s IP Allowlist.

ODBC PART:

Need to install SAP HDODBC driver SAP Development Tools (ondemand.com) in the system.
Open ODBC in the system
Click Add
Select HDODBC

Give any meaningful name to Data source name, description.

Database type: SAP HANA Cloud or SAP HANA Single tenant (both will work fine).

Already copied Host URL in datasphere space, Paste the copied Host URL.

Click Test connection

Paste the Database username in the Username and Password

AZURE DATA FACTORY PART:

Open Azure Data Factory:

Go to your Azure Data Factory instance via the Azure Portal.

Create a Linked Service:

On the left pane, go to Manage > Linked services.
Click New to create a new Linked Service.
In the search box, search for ODBC or SAP HANA.
Select ODBC

Enter Connection Information:

In the Connection String field, enter the connection string:

Driver={HDBODBC};ServerNODE=XXXXXXXXXX:443;UID=SAP_CONTENT#XXXX;PWD=XXXXXXXXXXXX;SCHEMA=SAP_CONTENT;

Breakdown of Components:

Driver: Specifies the SAP HANA ODBC driver needed to connect (e.g., {HDBODBC}).
ServerNODE: Indicates the SAP HANA server address and port to connect to (e.g., rw2922...443).
UID: The username used to authenticate to SAP HANA (e.g., SAP_CONTENT#XXXX).
PWD: The password associated with the provided username for authentication.
SCHEMA: The specific database schema where the data is located (e.g., SAP_CONTENT).
Encrypt: Ensures the connection is encrypted for secure communication (e.g., Encrypt=1).

Authentication Type:

Choose Basic Authentication since you’re using a username and password.
Enter the copied Datasphere’s Space username and password.

Test the Connection:

Click on the Test Connection button to make sure the connection is successful.

To check the connection, create a copy pipeline and choose the ODBC as the connector we used to get data from Datasphere and the corresponding Destination environment.

In this case, I have chosen the Azure Data Lake Storage Gen 2 as the Destination environment.

Hence the connection is established successfully from Datasphere to Azure Data Factory.

HINTS: 😀🤫

If your connection failed, then you have to check these two things,

Make sure your current IPV4 is added to Datasphere's IP Allowlist.
Ensure you have entered the correct Datasphere's Space credentials in your systems ODBC and test connection.

Thank you!

SAP Community Question: In your opinion, what's the biggest use-case for Blockchain at SAP Customers

2024-11-13T09:09:19.281000+01:00

Good Morning SAP Community,

in your opinion what is _the_ biggest use case for Enterprise Blockchain / Distributed Ledger Technology at SAP Customers ?

Anybody following my blogs will know I like Enterprise Blockchain, and am interested in positioning Enterprise Blockchain as a Technology Standard for use-cases and business demands at SAP Customers enabling them to run Enterprise Blockchain.

I talk to a lot of people about this and regularly get asked the same question, what is _the_ BIG use case for Blockchain in the Enterprise and at SAP Customers ?

So, the floor is open, the comments are open, there is no wrong answer, and everybody is invited to give their opinion on,

for SAP Customers, what is the biggest use case for Blockchain ?

Over to you, feel welcome to put your thoughts in the comments and let's see what we discover, and, there is no wrong answer 🙂

Andy Silvey.

Beyond the Data Silo: The Convergence of SAP Systems and Modern Data Lake Technologies

2025-02-10T04:53:39.311000+01:00

In today's enterprise landscape, a fascinating transformation is underway. While SAP systems continue to serve as the backbone of business operations, organizations are discovering that traditional approaches to data management no longer meet the demands of our rapidly evolving digital economy. The challenge isn't just about managing data – it's about turning it into actionable intelligence at the speed of business.

The Evolution of SAP Data Architecture

The SAP ecosystem has long been the backbone of enterprise operations, housing critical business data across its various modules. However, today's digital economy demands more than just robust transaction processing – it requires seamless integration of internal and external data sources, real-time analytics, and AI-driven insights.

The current state of enterprise data presents an intriguing paradox. Consider a typical global manufacturing Company X: Multiple SAP instances span continents, each running critical modules like FI/CO, MM, SD, and PP. In Europe, teams work with one set of customizations and configurations, while their Americas counterparts operate with different chart of accounts and business rules. What seems like a simple question about global inventory levels can spiral into a days-long exercise involving multiple teams, countless Excel sheets, and lengthy email threads.

Traditional-architecture

These challenges extend beyond just SAP systems. The integration of external data sources – IoT sensors, market data feeds, social media analytics – adds layers of complexity to an already intricate landscape. The time lag between data creation and insight generation has become a critical bottleneck for business agility.

The Promise of Modern Data Lakes

Modern data lake technologies represent a paradigm shift in how enterprises can manage and utilize their data. These platforms bring transformative capabilities through their support for industry-standard formats, enabling seamless data exchange across the enterprise. Their flexible schema management adapts to changing business needs, while enterprise-grade reliability is ensured through ACID transactions. The ability to access and restore historical data states, combined with unified processing for both batch and streaming workloads, creates a robust foundation for next-generation enterprise data management.

The Convergence: A New Enterprise Data Architecture

The future lies in architectures that bring together the best of both worlds – SAP's robust business processes and the flexibility of modern data lakes. This next-generation architecture reimagines how enterprise data flows and interacts:

Future-architecture

This convergence enables a new approach to data governance, where metadata management, data quality rules, and lineage tracking are centralized across both SAP and non-SAP data. Organizations can achieve zero-latency access to operational data with instant synchronization across systems, while performing real-time analytics without impacting transaction systems.

Perhaps most importantly, this convergence creates an AI-ready data foundation. By standardizing data formats for AI/ML workloads and enabling the integration of structured and unstructured data, organizations can fully leverage the power of large language models and generative AI across their enterprise data landscape.

Practical Implementation Strategies

Organizations looking to modernize their SAP data architecture should begin with a focus on high-value scenarios that require integrated data, particularly in areas where real-time insights drive business value. Projects that combine SAP and external data often yield the most significant returns and provide valuable learning experiences for teams. The foundation of this modernization requires implementing modern data lake capabilities alongside existing systems, while establishing unified governance frameworks. Creating semantic layers that abstract technical complexity enables broader adoption across the organization. Organizations must also prepare their data structures for AI workloads, implementing robust data quality measures and building pipelines for continuous data refreshes.

Conclusion: Preparing for Tomorrow

The convergence of SAP systems with modern data lake technologies isn't just a technical evolution – it's a business imperative. Organizations that successfully navigate this transformation will find themselves better positioned to accelerate innovation through integrated data access, improve decision-making with real-time insights, enable AI-driven business processes, and maintain competitive advantage in the digital economy.

The question isn't whether to embrace this transformation, but how to implement it in a way that maximizes business value while minimizing disruption. As technology leaders, our role is to guide our organizations through this evolution, ensuring we build data architectures that are not just modern, but future-ready.

What steps is your organization taking to modernize its enterprise data architecture? The journey toward unified, AI-ready data platforms is just beginning, and the decisions we make today will shape our ability to compete tomorrow.

Announcing the Water Track at the SAP for Energy and Utilities Conference 2025

2025-02-10T16:01:41.866000+01:00

For the third consecutive year, we are happy to announce a dedicated track for water and wastewater companies in the SAP for Energy and Utilities Conference agenda for this year.

It will take place on Wednesday afternoon April 9th with the aim to recognize the success of this section in previous conference editions and continue giving floor to a selection of exceptional professionals not only presenting, but building relationships and developing partnerships among this industry group.

Shifting from an agenda designed last year to show the concept architecture built by SAP to support water management processes to help organizations leveraging their software investments to tackle processes such as smart meter data management, named the “Smart Platform for Water”, current year be focused on cases where SAP portfolio of applications and technologies can help organizations confront their challenges and obtain operational benefits.

Mrs Inge Opreel, Farys CIO

I am delighted to inform you about the presentions of the Water Track:

SAP for Water and Wastewater Services: Building the Resilient Water Company (SAP Keynote)
ESRI ArcGIS on HANA: Building Next Generation of Integrated Asset Management at FARYS
Streaming Service Operations: SAP Field Service Management (FSM) Integration at Vitens

But the agenda of water-specific items will not be restricted to this track but will be highly recommended water-specific or water-relevant conferences before and after the Water Track which will configure an entire agenda specifically for delegates of this industry:

SAP Business Data Cloud

Wednesday morning, April 9th (before the Water Track): suggested slots unveil the large ongoing updates in the portfolio of Asset Management, including an interesting use case of Asset Performance Management as well as an excellent presentation by SAP Product Development of the evolution of SAP analytical portfolio, this is Business Data Cloud:

Latest Innovations in SAP Asset Management (Asset Management track)
Business Data Cloud – Unlocking the Power of Unified Data
SAP Asset Performance Management (APM) at AkerBP: From Test Environment to Production

Thursday morning, April 10th will be a continuation of the water-specific cases, presenting the application of AI with SAP technology platform at FARYS, a live demonstration at the Pop-up Campus (which will require a pre-booking)

Inside of SAP Pop-Up Campus

A very interesting use case of RISE conversion, from a perspective very close to the reality of many water companies'.

Unlocking the Potential of AI with SAP Business Technology Platform (BTP) and Datasphere (SAP Microforum at 10:30AM
A Day in a Life of a Water Drop: Deep Dive into the Benefits of Asset Management at the Lifecycle of a Waterdrop (session needs to be pre-booked via the app)
Transformation from OnPrem to SAP RISE: Lessons from Andel’s Journey (AI Track)

Overall, the complete conference is designed for an era based in networks, relationships and reputation, where collaboration and developing partnerships is the key of success. But it is known that such relationships cannot be automated and the soul of this track, and the overall water community formation, is the inheritance of the Executive Value Network for Water and Waste Water Companies workshops. This is a regular meeting of water companies taking place in Europe (already six editions), Latin America (three) and now Africa. Participants highlight common challenges and go along with case studies of how SAP customers are leveraging SAP solutions to address them. Take this unique opportunity to connect with your peers, gain insights into industry trends, and chart your development plan with the latest technologies of 2025!

Looking forward to seeing you in Rotterdam!

🚀 Unlock the Power of Cloud, AI, and Business Transformation at SAP Sapphire 2025 in Madrid

2025-04-12T09:23:10.254000+02:00

From May 26 to 28, SAP Sapphire in Madrid will bring together the brightest minds, boldest innovations, and most impactful customer stories across the SAP ecosystem. Whether you're embarking on your RISE with SAP journey, transforming your support strategy with AI, or streamlining operations with SAP Cloud ALM, this is your chance to connect, learn, and get inspired.

If you're planning your agenda, make sure to explore the powerful lineup of sessions hosted by our SAP Customer Support & Cloud Lifecycle Management team. These sessions dive deep into real customer use cases, cutting-edge tools, and strategic insights that can help you unlock value, improve resilience, and stay ahead in today’s rapidly evolving business landscape.

✨Don't miss these highlights from our team at SAP Sapphire Madrid 2025

📍Main Theater & Breakout Sessions – Tuesday, May 27

SER1211 | 01:00 PM | Room 10.17
Achieving a clean core with your RISE with SAP journey
Wieland Schreiner
Learn how RISE with SAP enables a clean core in your transition to cloud ERP. See how SAP Cloud ALM supports transformation across five key dimensions and discover proven strategies for sustainable business success.

SER1213 | 01:30 PM | Room 10.17
Accelerating success: Unlocking value with smooth customer onboarding
Lee Evans
Discover how structured onboarding can fast-track value realization. This session shares strategies to drive adoption, increase customer lifetime value, and build a solid foundation for ongoing success.

SER1212 | 02:00 PM | Room 10.19
Streamline your RISE with SAP journey with the power of SAP Cloud ALM
Wieland Schreiner
Explore how SAP Cloud ALM works with SAP LeanIX and SAP Signavio to simplify landscape discovery, optimize processes, and enhance visibility. Hear success stories and practical takeaways for smarter transformation.

SER1372 | 02:30 PM | Room 10.15
Fast, Smart, and Scalable – How Bosch Leverages SAP S/4HANA Public Cloud
Stefan Steinle, Bastian Kloiber (Bosch)
Get an inside look at Bosch’s two-tier ERP strategy using SAP S/4HANA Public Cloud. Learn how automation, SAP Best Practices, and close collaboration with SAP helped ensure smooth rollouts and operational success.

SER1215 | 03:00 PM | Room 10.18
Elevating your GROW with SAP journey with SAP Cloud ALM
Tonja Kehrer
Discover how SAP Cloud ALM can accelerate your GROW with SAP journey with real-time insights, demo-driven learning, and innovation strategies for scalable growth.

SER1214 | 03:30 PM | Room 10.18
Thriving in holiday peaks: How AI transforms system resilience
Stefan Steinle, Suhaim Mohamed (Douglas)
Explore how AI-driven holiday readiness safeguarded €11 billion in GMV for 143 customers. Learn how this initiative transforms system resilience and prepares your business for peak performance.

📍Ask the Expert Sessions – Tuesday, May 27 | Room 10.201

SER1218 | 01:30 PM
Maximizing business value with AI-driven support tools
Wilhelm Juette
Explore how SAP Cloud ALM, Built-In Support, and the ITSM connector help you tailor support and drive value through intelligent automation.

SER1219 | 02:00 PM
AI – From A to Impact: A case study to measure business value with AI
Wilhelm Juette
See how SAP uses process mining and data insights to create a value measurement framework in the world of agentic AI.

SER1220 | 02:30 PM
Streamlining migration to SAP S/4HANA with lean selective data transition
Thorsten Spihlmann
Get hands-on strategies using the SAP Business Transformation Center to simplify and automate your SAP S/4HANA migration while focusing on what matters most.

SER1221 | 03:00 PM
Safeguarding operations during the smooth transition to SAP S/4HANA Cloud
Mirja Kempin, Markus Winter
Learn how to keep business continuity front and center during your move to SAP S/4HANA Cloud Private Edition.

SER1222 | 03:30 PM
Using AI and data-driven insights for a smooth customer support experience
Vivian Luechau-de la Roche
Explore how AI and tools like SAP for Me are streamlining support by surfacing relevant case history, resolution paths, and customer insights.

🎯Why You Should Attend

From onboarding to migration, support to system resilience—these sessions are packed with real-world insight, customer voices, and practical frameworks to drive your transformation forward. They also highlight SAP’s ongoing commitment to intelligent, cloud-based innovation that delivers measurable outcomes.

🗓️ Join us in Madrid!

We can’t wait to meet you onsite—stop by our sessions, ask your toughest questions, and connect with our experts to explore how AI, automation, and cloud transformation can take your business to the next level.

Let’s build the future together at SAP Sapphire Madrid 2025.

#SAPSapphire #SAPCommunity #CustomerSupport #SAPCloudALM #AI #RISEwithSAP #GROWWITHSAP

Lets Talk to the Database - CAP & SQLite Powered by Google Gemini-2.5-Pro

2025-04-16T14:08:37.441000+02:00

I am sure everyone is doing great !!!

I am glad and excited to share my learning on Talk to Database , I mean just talk to the database using Natural Language powered by Google Gemini-2.5-pro-exp handled by SAP CAP programming.

So what all we have here to achieve this end-to-end ?

We have access to Gemini-2.5-pro exprimetnal version available ( PAID ) Get your API Keys to interact through API Call
A CAP Application where we have deployed Sales Order header / Line Item / Deliveries Entities Deployed
Install SQLITE DB For Data Storage for all the 3 Tables
Add some Dummy data into your DB tables So that later we can validate data and interact with the database for fetching requested information and run select statement on top of that.

Test out Gemini API using POSTMAN before you jump into this.

Data Prep:

Extracted data from Sales Order Header (VBAK)
Extracted data from Sales Line Item (VBAP)
Extracted data from Delivery (LIPS)

Tasks Break Down:

Prompt Prep :

Prepare the prompt and provide enough context to generate SQL
Example Prompt – Give me number of Sales Order in System
Prompt Specifies -- A Count is required from Sales Order Header table

Context -

Sales Order Header = SalesOrderH Table
Sales Order Item = SalesOrderItem

Instruction –

`You are a specialized SQL query assistant for a computer store database. Your primary goal is to answer user questions by retrieving data using the available tools.

Database Context:

The key tables you will interact with are:

'sp_prompts_Delivery': Contains Delivery Data.

'sp_prompts_SalesOrderHeader': Contains Sales Order Header Data (often referred to as just Sales Orders).

'sp_prompts_SalesOrderItem': Contains detailed Sales Order Item Data. `

Identify Tables :

Develop NodeJS API to extract tables names Specified
Develop NodeJS API to list all the tables:

Example - http://localhost:4004/odata/v4/my/listTables

Benefit – this will help in finding out extract matching table while building the select query , will help in preparing the right context for model input

Schema Extraction : Develop Nodejs API to extract schema of tables ( Entities ) specified

Extract Schema for any give table input

Example –

http://localhost:4004/odata/v4/my/describeDbTable

Content-Type: application/json

{ "tableName":"sp_prompts_SalesOrderHeader" }

Benefit: this will help in preparing schema for tables incase we need only few columns or specific column like Sales Order Types / Delivery Types etc.

Execute DB Query: This is the most important one once Model has generated the select statement for the input , this will execute the query and provide the response

Example – Prompt - SELECT * FROM sp_prompts_SalesOrderItem

Response :

Benefit: - It will run the DB Statements and will return the results

Prepare Response:

This will come into picture in case we are requesting some information and expect the response in a plain english Language

Example – Give me no of Sales Order and Sales Item from System

If you notice the generated sql then you will get the idea what happening behind the scene.

Generated SQL

CAP Structure:

This is how my CAP Structure look like.

QueryEngine Contains the Main API Call to Gemini which subsequently calls all the functions required to get data back from Database like below.

AI Model Used : Gemini-2.5-pro-exp

Initialize of GenAI Model like below.

If you see we have passed set of instructions that contains information about the database and tools are nothing the list of function which will give information about Tables and their Schema and later the final function for executing the generated query.

Dbquery.cds

This action will be responsible for prompt handing and it will return object of Information.

Dbquery.JS

This will handle all the incoming requests ---QueryDatabase is the one which will be triggered from outside rest of functions will be called internally , these are listed for standalone testing.

Note

I am also attaching a small Video Clip where you can see the execution of various prompts for technical implementation DM me directly , i can help you in building this POC.

#KeepLearningKeepSharing

SAP TechEd 2025: 👟 Slip On, Speed Up: The Next Decade of Business

2025-09-02T14:16:17.305000+02:00

The world is in sprint mode ...

.... AI cycles flip every quarter, supply chains get bendy, climate shocks rewrite plans. As Lenin put it, "there are weeks when decades happen." Winning 2030 isn’t about hoarding data or demo sparkle. It’s about cutting friction, staying focused, and moving as one.

SAP’s 2030 play:

AI that removes grind - copilots, automation, anomaly detection.
Data you can trust - clean, connected, decision-ready.
Networks that sync the ecosystem - suppliers, partners, customers moving together

The metaphor? Lock laces. Triathletes don’t stop to tie shoes. They slip on and go. In finance, AI is the lock lace: auto-recs, instant cleanup, fewer errors. Hours back. Focus up. Strategy on.

This is the shift: from tying shoes to breaking records. From reconciling ledgers to reshaping the business.

Join Paul Saunders and Laura Marwood Ph.D. at SAP TechEd 2025 to see how we’re building for that future - because in 2030, every second (and every decision) will count.
__________________________________

Can't make it to Berlin? Dial in for SAP TechEd Virtual!

If you can’t make it to Berlin, we’ve still got you covered. SAP TechEd Virtual is a free online event that’ll help you broaden your technical expertise and explore SAP’s latest innovations

Optimizing Dynamic Group Creation and simplifying rule for New Hire Data Review

2025-10-17T21:34:56.085000+02:00

Optimizing Dynamic Group Creation and New Hire Data Review

My client had a large number of offices and training locations—around 250+—each responsible for reviewing new hire data for incoming employees. This created a need for multiple Dynamic Groups and a complex set of rules to select the correct locations based on predefined criteria.

To simplify this, we implemented a structured solution, which I will explain step by step below.

Step 1: Configuring the Training Location Field in Recruiting

At the application stage, where the correct location for the employee is selected during the recruiting process, we created a Training Location field with the same picklist values as in Employee Central (EC).

This field carries the value over to EC, which becomes the **key deciding factor** for determining which group will be selected for reviewing new hire data.

Note: This blog does not cover how to create a field in the RCM application or how to map that field to ONB.

Step 2: Creating and Configuring the Object

Once the field was set up, we created an object as shown below.

Please ensure that the following fields are configured as indicated:

Object: cust_ONB2ResponsibilityDetailConfig

* The second field must match the value from Recruiting. It should have the same data type and field configuration as in Recruiting to enable proper mapping and data transfer to Onboarding.

Step 3: Setting Permissions

It’s essential to assign the correct permissions to the object before entering data.

Once the permissions are set, the object should appear as shown below:

---

Step 4: Creating the Business Rule

Next, we designed a **business rule** to control the group selection logic. This rule determines which dynamic group should be picked based on the training location value passed from Recruiting.

---

Data Collection Object Configuration

The second part of the requirement involved handling **various data collection UIs**, which the customer wanted to display based on specific conditions.

To achieve this, we created a **new object definition** as illustrated below:

We added the **foundation object: Location Group**, since each combination of **Location Group + Company + Employee Group** determines the type of object to be displayed.

Additionally, the **Object UI** must include the following configuration:

Once the object is uploaded, it should look like this:

Finally, we designed a Rule to pick the appropriate lookup table configuration**. The rule should resemble the structure shown below:

Conclusion

By standardizing the location field, creating a well-structured object, and designing targeted business rules, we were able to eliminate the need for lengthy line items in dynamic group creation. This approach not only improves system performance but also provides flexibility in data collection UI configuration based on location and organizational attributes.

SAP RPT-1 Context Model vs. Training Classical Models: The Models Battle (Python Hands-on)

2025-11-20T07:50:27.670000+01:00

💥The Models Battle

Predictive modeling is becoming a built-in capability across SAP, improving how teams handle forecasting, pricing, and planning. Many SAP professionals, however, aren’t machine-learning specialists, and traditional models often demand extensive setup, tuning, and repeated training, which slows down new ideas.

SAP RPT-1 offers a simpler path. It’s a pretrained model from SAP, also available in an OSS version, that lets developers and consultants produce predictions with far less technical effort, no deep ML background required.

I've explored SAP RPT-1 hands-on, comparing it with traditional regressors using Python and a real public vehicles price dataset.

Goal: To see (as a non Data Scientist) how SAP RPT-1 behaves in practice, what advantages and limits it shows, and when it could make sense in a predictive scenario.

Usually for real-world scenario, the right approach would be consume the SAP RPT-1 though the available and simplified API, but for studies proposal and fair comparision over othe traditional ML models, the OSS fits perfectly for it:

🤔 SAP RPT-1 vs Traditional Machine Learning - Core Differences

Before diving into the code, let’s quickly revisit how traditional ML models work:

Training-based models like Random Forest, LightGBM, and Linear Regression learn patterns directly from data.
They require hundreds or thousands of examples to tune their internal parameters.
Their performance depends heavily on data quantity and quality.
The more relevant examples they see, the smarter they get.

On the other hand, SAP RPT-1 follows a different philosophy. It’s part of the RPT (Representational Predictive Transformer) family, pretrained on a wide variety of business and contextual data. This means:

You don’t "train" it in the traditional sense. Instead, it uses context embeddings to predict outcomes.
It can be used immediately, even with smaller datasets.
The OSS version allows developers to experiment directly in Python.
No special SAP backend required.

Outcome: Traditional ML models learn from high amount of data. SAP RPT-1 already knows how to deal with small context amount of data.

🖥 The Experiment - Setup & Dataset

Spoiler

Don't worry on "playing puzzles" copying + pasting code below. The full version is available for download at end!

To make this comparison tangible, I built a simple yet realistic Python experiment to predict vehicle selling prices using a public dataset containing car attributes like make, model, year, transmission, and mileage.

Why vehicle pricing? Because it’s an intuitive example where both traditional machine learning and pretrained AI models can be applie, and it helps visualize how prediction quality evolves as the sample size grows.

This entire analysis runs on a local Python environment with the following stack:

import os
import gc
import warnings
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from datetime import datetime
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LinearRegression
from sap_rpt_oss import SAP_RPT_OSS_Regressor
import lightgbm as lgb

pandas and numpy for data manipulation
scikit-learn for classical ML regressors (Random Forest, Linear Regression)
LightGBM for gradient boosting comparison
sap_rpt_oss — the open-source Python version of SAP’s RPT-1 model
matplotlib for all visualizations

SAP RPT-1 OSS can be downloaded installed following official Hugging Face: https://huggingface.co/SAP/sap-rpt-1-oss?library=sap-rpt-1-oss . Python can be installed with executable download on Windows, or via Home Brew for Mac and apt commands for Linux. Libraries dependencies can be downloaded with pip commands. Googling it may not be a road blocker.

We use a sample vehicle sales dataset. The complete file is about to 88Mb but for such experiment a restricted sample of 20k as it's more than enough to prove our the concept, still it's faster and consuming less computing resources.

Feature	Description
`year`	Vehicle model year
`make`	Brand (e.g., Toyota, Ford, BMW)
`model`	Specific model name
`body`	Type (SUV, Sedan, etc.)
`transmission`	Gear type
`odometer`	Vehicle mileage
`color`, `interior`	Visual attributes
`sellingprice`	The target variable to predict

📊 Dataset Download: https://www.kaggle.com/datasets/syedanwarafridi/vehicle-sales-data?resource=download

The dataset is loaded and preprocessed in a few simple steps:

df = pd.read_csv("car_prices.csv").sample(n=20000, random_state=42)

# Fill missing values for categorical columns
fill_defaults = {
    'make': 'Other', 'model': 'Other', 'color': 'Other',
    'interior': 'Unknown', 'body': 'Unknown', 'transmission': 'Unknown'
}
for col, val in fill_defaults.items():
    df[col] = df[col].fillna(val)

X = df[["year", "make", "model", "body", "transmission", "odometer", "color", "interior"]]
y = df["sellingprice"]

At this point, the stage is set:

The data is clean.
The environment is ready.
All models, traditionals and SAP RPT-1, are ready to be tested under identical conditions.

🤖 Training the Models - Three different ones

With the dataset ready, the next step is to run each model under the same conditions: same features, same target, same train/test split and same random seed. This ensures the comparison is fair and repeatable.

We evaluate prediction performance using R² (coefficient of determination), which indicates how much of the price variation the model can explain (1.0 = perfect prediction).

Training Model #1 - Random Forest

Random Forest is often the first model used in tabular ML. It works by creating many decision trees and averaging their predictions. Before training, categorical variables need to be label-encoded into numbers, a common requirement for classical ML models:

def train_random_forest(X, y):
    X = X.copy()
    cat_cols = ["make", "model", "body", "transmission", "color", "interior"]
    le = LabelEncoder()

    for col in cat_cols:
        X[col] = le.fit_transform(X[col].astype(str).fillna("Unknown"))

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=default_test_size, random_state=42
    )

    model = RandomForestRegressor(
        n_estimators=150, max_depth=20, random_state=42, n_jobs=-1
    )

    try:
        model.fit(X_train, y_train)
        preds = model.predict(X_test)
        r2 = r2_score(y_test, preds)
    except Exception as e:
        preds, r2 = np.zeros_like(y_test), 0

    return [preds, r2, y_test]

Up to 50 rows:

Up to 7067 rows:

Live view

Training Model #2 - LightGBM

LightGBM is one of the most powerful models for tabular data. Unlike Random Forest (many independent trees), LightGBM builds trees sequentially, each correcting the errors of the previous one. It supports categorical features natively, which simplifies preprocessing.

def train_lightgbm(X, y):
    X = X.copy()
    cat_cols = ["make", "model", "body", "transmission", "color", "interior"]
    for col in cat_cols:
        X[col] = X[col].astype(str).fillna("Unknown").astype("category")

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=default_test_size, random_state=42
    )

    model = lgb.LGBMRegressor(
        n_estimators=500, learning_rate=0.05, num_leaves=31,
        subsample=0.8, colsample_bytree=0.8, random_state=42
    )

    try:
        model.fit(X_train, y_train, categorical_feature=cat_cols)
        preds = model.predict(X_test)
        r2 = r2_score(y_test, preds)
    except Exception:
        preds, r2 = np.zeros_like(y_test), 0

    return [preds, r2, y_test]

Up to 50 rows:

Up to 7067 rows:

Live view

Training Model #3 - Linear Regression

Not fancy and even not complex, Linear Regression provides a baseline that shows: “If the relationship between attributes and price is roughly linear, how well can a simple model perform?”

def train_linear_model(X, y):
    X = X.copy()
    cat_cols = ["make", "model", "body", "transmission", "color", "interior"]
    for col in cat_cols:
        X[col] = LabelEncoder().fit_transform(X[col].astype(str).fillna("Unknown"))

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=default_test_size, random_state=42
    )

    model = LinearRegression()
    X_train = X_train.fillna(X_train.mean(numeric_only=True))
    X_test = X_test.fillna(X_test.mean(numeric_only=True))

    try:
        model.fit(X_train, y_train)
        preds = model.predict(X_test)
        r2 = r2_score(y_test, preds)
    except Exception:
        preds, r2 = np.zeros_like(y_test), 0

    return [preds, r2, y_test]

Up to 50 rows:

Up to 7067 rows:

Live view

🏁 SAP RPT-1 OSS: Context Model

This is where things get interesting. SAP RPT-1 does not rely on learning patterns from the dataset. Instead, it uses a pretrained transformer architecture to infer relationships directly through context embeddings. Lean and simple, "for non-Data Science PhD":

def train_sap_rpt1(X, y):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=default_test_size, random_state=42
    )

    model = SAP_RPT_OSS_Regressor(max_context_size=8192, bagging=8)
    model.fit(X_train, y_train)

    preds = model.predict(X_test)
    r2 = r2_score(y_test, preds)

    return [preds, r2, y_test]

Up to 50 rows:

Up to 2055 rows:

Live view

🔎 Running Experiments at Multiple Sample Sizes

This section breaks down how the iterative experiment loop works, why the SAP RPT-1 OSS model has a max-context limit, and how performance changes as we scale up the dataset. By running the same models across several sample sizes, we can see where traditional ML shines, where RPT-1 stays competitive, and how both behave as the data grows.

sample_sizes = np.linspace(50, len(X), 200, dtype=int)
results, max_r2_rpt1, max_sample_rpt1 = [], 0, 0

for n in sample_sizes:
    idx = np.random.choice(len(X), n, replace=False)
    X_sample, y_sample = X.iloc[idx], y.iloc[idx]


    # SAP RPT-1 OSS (limited sample size)
    if n <= rpt1_limit:
        rpt_res = train_sap_rpt1(X_sample, y_sample)
        fn = plot_predictions(rpt_res[2], rpt_res[0], rpt_res[1], "SAP_RPT1", n)
        video_frames["SAP_RPT1"].append(fn)
        r2_rpt1 = rpt_res[1]
        max_r2_rpt1 = max(max_r2_rpt1, r2_rpt1)
    else:
        r2_rpt1 = max_r2_rpt1
        if max_sample_rpt1 == 0:
            max_sample_rpt1 = n

    # Train and plot models
    rf_res = train_random_forest(X_sample, y_sample)
    fn = plot_predictions(rf_res[2], rf_res[0], rf_res[1], "RandomForest", n)
    video_frames["RandomForest"].append(fn)

    lgb_res = train_lightgbm(X_sample, y_sample)
    fn = plot_predictions(lgb_res[2], lgb_res[0], lgb_res[1], "LightGBM", n)
    video_frames["LightGBM"].append(fn)

    lin_res = train_linear_model(X_sample, y_sample)
    fn = plot_predictions(lin_res[2], lin_res[0], lin_res[1], "LinearModel", n)
    video_frames["LinearModel"].append(fn)

    results.append((n, rf_res[1], r2_rpt1, lgb_res[1], lin_res[1]))

    # Early stop if traditional model reaches SAP RPT-1
    if rf_res[1] >= max_r2_rpt1 or lgb_res[1] >= max_r2_rpt1 or lin_res[1] >= max_r2_rpt1:
        break
    gc.collect()

This loop compares SAP RPT-1 OSS with traditional ML models as sample sizes increase. Each iteration randomly selects a subset of the data and trains all models on the same slice for a fair comparison. SAP RPT-1 can only run up to its max-context limit, so once the sample size exceeds that threshold, it stops retraining and simply carries forward its best R². The traditional models continue training at every step. The loop ends early when any traditional model matches or surpasses RPT-1’s best score, making the experiment efficient while showing how performance evolves as data grows.

🔚 Conclusion and Final Thoughts

SAP RPT-1 OSS stands out because it performs well with small datasets, requires minimal code, and can generate useful predictions with just an API call and a bit of context. This makes it ideal for jump-starting predictive use cases early on, delivering fast business value without a full ML pipeline. Traditional models, however, still shine when projects mature, data grows, and fine-tuned control becomes important. It’s not about choosing one over the other, but understanding where each approach brings the most value.

Aspect	SAP RPT-1 OSS	Traditional ML (RF, LGBM, Linear)
Data Requirements	Low (performs well with small samples)	Medium/High (performance scales with data
Setup Effort	Minimal (API call + context)	Higher (preprocessing, encoding, tuning)
Training Process	None (pretrained context model)	Full training pipeline required
Speed to Insights	Very fast	Moderate to slow
Best Use Case	Early-stage predictive cases, quick baselines	Mature pipelines, high control and customization
Flexibility	Limited tuning / plug-and-play	Highly customizable
Business Value	Immediate, fast, accessible	Strong when optimized and scaled

This experiment highlights a simple truth: SAP RPT-1 isn’t here to replace traditional ML, it jump-starts it. With a pretrained, context-driven approach, RPT-1 delivers fast, reliable insights with very little data and almost no setup. Traditional models still excel in mature, data-rich scenarios, but RPT-1 shines as a rapid accelerator and early-value generator inside SAP landscapes.

💬Open for Exchange

If you're testing RPT-1, exploring predictive cases, or want the full code, feel free to reach out.
Happy to connect, compare experiences, and push this topic forward together.

SAP BTP: Data Fabric

2025-12-12T12:50:49.920000+01:00

Irrespective of where your source system resides, SAP provides the capabilities to ingest, harmonize and unlock real-time and predictive insights on your target system with confidence. This is supported by resilient, enterprise-grade frameworks, tools, and methodologies that safeguard data quality, integrity, and consistency throughout the entire data lifecycle.

SAP. (2023). Figure:1

Source Systems for Data Provisioning in SAP

URL: https://learning.sap.com/courses/implementing-data-provisioning-in-sap-bw-4hana/identifying-sap-bw-4hana-source-systems-and-their-use-cases

The endless possibilities of raw, intrinsic data ingestion from e.g. Enterprise warehouses, data lakes, rational & in-memory databases, SaaS applications even social media into SAP HANA (On-premises and/ or Cloud) opens a world of opportunity for data engineers to acquire rich, accurate, and complete master and transactional data from all possible sources imaginable. Responsibility for managing the data acquisition processes depends on the provisioning type being used. Common approaches include:

Provisioning Type	Data Flow Responsibility	Metadata Management	Typical Use Cases
Application-Controlled	Source application pushes data to target using APIs or connectors.	Managed in application layer (schema, mappings, transformations).	Custom logic, application-driven workflows, SaaS integrations.
Change Data Capture (CDC)	CDC engine streams changes (insert/update/delete) from source to target in near real-time.	Managed by CDC tool (change logs, timestamps, transaction markers).	Real-time replication, incremental updates, event-driven architectures.
Database-Controlled	Target database pulls data from source using adapters or SDI.	Centralized in database layer (schemas, lineage, governance).	Governance-heavy environments, centralized data lakes, enterprise ETL.

Figure:2

Provisioning Types: Data Flow & Metadata Management

The utilization of high-performance ETL/ELT tools ensure the consistence, quality and reliability of the data is upheld and thus often used to unify the data to best meet the objectives of the stakeholder’s request either being executives, decision makers, marketing or product. A well-thought-out data provisioning strategy should always take precedence as this ensure a smooth flowing data process. Depending on the usage of the data, a data engineer may explorer the below different data provisioning approaches:

Data federation – Data is not loaded nor persisted into the target system (SAP HANA) but rather read directly from the source system only when the data is needed for use.
Data replication – Data from the source system is loaded, persisted into the target system and kept in sync using various mechanisms
- Real-time (Synchronous) replication
- Delta load (asynchronous)replication
Data transformation – The raw data is enhanced with additional features e.g. calculation columns, renamed fields or new fields derived from other exiting fields

SAP provides but not limited to the below high-performance native data integration technologies:

Tool	Primary Function	Position in Landscape Architecture
SLT (SAP Landscape Transformation Replication)	Real-time replication (CDC)	Integration layer between SAP sources and HANA DB
SDA (Smart Data Access)	Virtual access to remote sources (federation)	Integration layer enabling query pushdown to the HANA DB
SDI (Smart Data Integration)	ETL/ELT pipelines, adapters, dataflows	Integration layer moving and transforming data before loading into the HANA DB

Figure 3:

SAP Native ETL/ELT tools

Furthermore, for Data quality, cleansing and enrichment before persisting into the HANA DB, SAP provides the below:

Smart Data quality (SDQ) for SAP HANA on-premises
Data Quality Management Microservices (DQMm) as a subscription-based microservice for SAP HANA Cloud

SAP Hana provides a variation of consolidated tools both on premise and in the cloud, all providing for various functionality for different use cases such as:

Supporting data provisioning
Developing Database artefacts
Monitoring the performance of the develop artefacts
Monitoring the Database and its system performance
Administering and managing the Database

One of the key strengths of the SAP ecosystem is the ability to seamlessly combine cloud functionalities together with on-premises and still import specific/ needed applications as micro services into common IDE’s such as Microsoft Visual Studio code is commendable because it:

Provides business with a comprehensive 360-degree view of
- Database health overview.
- Data life cycle from end to end.
- Application life cycle from end to end.

Empowers the end users, data engineers, developers, data scientists, data analyst and support engineers with greater power and flexibility over the data being used.

Below are native tools used by SAP for various purposes:

SAP Business Application studio – Cloud development tool.
Web IDE – On-premises development tool
SAP Hana cockpit - a web-based tool for monitoring, administering, and managing SAP HANA databases, offering a centralized, browser-accessible interface with tile-based dashboards for performance, resource usage (CPU, memory, disk), alerts, and security. It allows tasks like starting/stopping services, managing users, viewing alerts, analyzing performance, and running SQL queries through its integrated Database Explorer.

SAP SaaS applications are imported into common IDE, s such as Microsoft Visual Studio code via extensions, libraries etc. for a specific purpose e.g.:

Supportability tools for SAP HANA - For both online and offline performance analysis and troubleshooting. With a focus on the needs of support engineers, the performance analysis tools provide enhanced analysis options for trace files and runtime dump files, as well as allowing in-depth analysis on extensive datasets.
The SQL analyzer tool for SAP HANA - A performance analysis tool that helps developers visualize and understand SQL execution plans. Designed for query tuning and optimization, it offers detailed insights into how queries run within the SAP HANA database.
SAP HANA Database Explorer - Provides functionality for accessing SAP HANA databases, browsing the database catalog, and executing SQL from a SQL console.

Figure 4:

SAP Development and Administration Tools Overview

The consolidated toolset further enhances productivity and convenience for end users, data engineers, developers, data scientists, data analyst and support engineers by enabling them to access all essential capabilities from a single unified platform. The choice and flexibility to work with the desired programming languages ranging from but not limited to: SQL, SQL Script, Python, Node.js, JavaScript and react native etc. Enable a cross-functional team with diverse skill set to collaborate effectively while striving towards the same common goal.

Figure 5:

Core Programming Languages and Frameworks for SAP HANA Cloud and On-Premises

After successfully completing the Explore phase, with a solid fit -to-standard analysis of proposed use cases, Extrinsic data pipelines are then implemented for reliable and consistent data flow. These pipelines support a wide range of business needs such as decision making, go-to-market activities or ML/AI use cases requested by various stakeholders. The process is typically executed and maintained by data engineers, developers, data scientists, data analyst and/or support engineers.

The harmonized data is then governed, analyzed, and visualized through multiple front-end applications. These applications are powered by various integration frameworks e.g. REST APIs, OData v4, JDBC/ODBC, each serving different user needs and functional objectives:

Figure: 6

SAP-Compatible Applications and Their Integration Frameworks

Mutatis mutandis, Data is the inception of all there is, as it has been in the beginning, is now and ever shall be. Thank you all for taking a glimpse into the above Content. Please don’t for get to like, comment and share.

Abbreviations:

API: Application Programming Interface

BTP: Business Technology Platform

CDC: Change Data Capture

DB: Database

DQMm: Data Quality Management Microservices

ELT: Extract, Load, Transform

ETL: Extract, Transform, Load

IDE: Integrated Development Environment

ML/AI: Machine Learning/Artificial Intelligence

ODBC: Open Database Connectivity

OData: Open Data Protocol

REST: Representational State Transfer

SaaS: Software as a Service

SDI: Smart Data Integration

SDA: Smart Data Access

SDQ: Smart Data Quality

SLT: SAP Landscape Transformation Replication

SQL: Structured Query Language

Reference:

SAP. (2023). Figure:1 Source Systems for Data Provisioning in SAP, URL: https://learning.sap.com/courses/implementing-data-provisioning-in-sap-bw-4hana/identifying-sap-bw-4hana-source-systems-and-their-use-cases