SAP Community - SAP Datasphere

Rewiring of SAP Datasphere to SAP Business Data Cloud

2025-12-19T09:58:19.950000+01:00

The rewiring of existing SAP Datasphere tenants to SAP Business Data Cloud has become available in 2025. Since then, I have been in many discussions with customers, partners, but also SAP colleagues at SAP TechEd in Berlin, Sydney and other events or meetings.

I observed that people unintentionally misinterpret the term “rewiring”. It sounds technical, hence there are associations with data, object or even full tenant migrations. This is not the case!

Let me use this blog to clarify what rewiring is, and what it is not. Moreover, give you some guidance on the steps which you need to take.

When do I need to rewire my existing SAP Datasphere tenant?

You have made a decision to sign up for SAP Business Data Cloud (BDC) and to make use of your existing tenants for SAP Datasphere and / or SAP Analytics Cloud as part of your new landscape with SAP BDC.

If you have not made any decision on SAP BDC yet, you can certainly use your existing tenant(s) until the end of your current contract without SAP BDC. However, you will miss out on new capabilities offered by SAP BDC with SAP managed data products and the enhanced open partner ecosystem options.

What is tenant rewiring?

The term “rewiring” sounds quite technical. Therefore, there are immediate associations with data, object or even full tenant migrations. This is not the case!

The term rewiring mainly refers to a change of your commercial model from a stand-alone tenant to a tenant which is managed as part of a larger SAP BDC landscape.
Rewiring helps you change the management of your tenant lifecycle to the SAP for Me portal.
Prerequisites are that you commercially move the tenant on your same customer ID and your current data center being available also for Business Data Cloud. Please check on these two points!
If these conditions are met a lot will stay the same. Meaning you stay on your existing hyperscaler and data center. Hence, your tenant and its URL will remain intact, just under a new commercial model, even without any downtime required.

What is tenant rewiring not?

Rewiring is therefore not a tenant migration as your existing tenant stays the same and is reused as part of an SAP BDC landscape.

Rewiring does not migrate your tenant between hyperscalers or data centers, as with the conditions mentioned before, your current data center – given that it is available for SAP BDC – will stay the same on your existing customer ID.

Rewiring does not migrate your data or any objects, as the full existing tenant is only commercially moved as described before.

Rewiring does not require any tenant downtime.

How can I initiate a tenant rewiring?

Before you can start to initiate the rewiring, you need to change the commercial model of your SAP Datasphere tenant to a Business Data Cloud contract – in short: a subscription to SAP Business Data Cloud must exist.

For tenants provisioned via BTP cockpit

After the commercial change, your SAP BTP administrator can initiate the tenant rewiring directly through the SAP BTP platform global account as a self-service. You control the timing of rewiring yourself, as you can trigger it via self-service.

Once the request is received, the process is initiated to update the tenant and move it to SAP Business Data Cloud. The tenant will be visible in SAP Business Data Cloud when the rewiring is completed.

For subscription tenants provisioned by SAP

After the commercial change, your subscription is moved from standalone to SAP Business Data Cloud. Be aware, that the SAP Business Data Cloud cockpit feature must be activated in your appropriate data center through SAP for Me portal prior to rewiring.

The rewiring itself is initiated as part of the subscription change request. That means, your tenant is automatically moved to SAP Business Data Cloud for the contract start date.

Where do I find more detailed information?

For more details I recommend these three resources:

SAP Community Blog by @Ammar_Naji:
🔗 UPDATE: SAP Datasphere and SAP Analytics Cloud Availability via SAP Business Data Cloud

SAP Help documentation for detailed steps:
🔗 Rewiring of Standalone SAP Datasphere and SAP Analytics Cloud Subscriptions to SAP Business Data Cloud

🔗 SAP Note 3650483

Let me know if there are still questions about rewiring and post them below 👇 in the comments section. I will try to answer them and loop in colleagues in case I do not know the answer myself.

Connecting SAP Analytics Cloud to Databricks model serving endpoint

2025-12-31T16:15:03.063000+01:00

Scenario

Imagine your data scientists have worked their magic and built a predictive (I should call it AI now) model that you wish to access from an SAP Analytics Cloud (SAC) story. The model may require parameters to analyze specific data, or you may require the latest output to be retrieved

This can be achieved using the SAC Multi Action feature. Databricks and SAC both support OAuth 2.0 for authentication. A service principal is used for authorization and authentication outside of the standard Databricks UI. The SAC Multi Action feature offers different actions(steps), including an API step which utilises a secure HTTP API connection

Prerequisites

Required Solutions

SAP Analytics Cloud
Databricks

Required users access

SAP Analytics Cloud BI Admin or SAP Analytic Cloud Planning
Databricks Workspace Admin

Required artefacts

Databricks Model Serving Endpoint

OAuth with Databricks / SAP Databricks

The steps are the same with native Databricks and SAP Databricks. The Databricks documentation describes the generic steps and required parameters for OAuth 2.0 authorization.
https://docs.databricks.com/aws/en/dev-tools/auth/oauth-u2m
The SAC HTTP API connection requires 5 parameters from Databricks.

Data Service URL
OAuth Client ID
Secret
Token URL
Scope

Databricks Service Principal

We must first create service principal in Databricks, this can be at either the account or the workspace level.
Here I have created the service principal within the workspace.

Navigate to User Settings -> Identity and Access

Identity and Access

Add Service Principal

Add Service Principal Continued

Add Service Principal Name

OAuth Secret

With the Service Principal created we can generate the secret required for authentication

Service Principal

Service Principal Secrets

Service Principal Generate OAuth Secret

Generate OAuth Secret - Lifetime

We need the secret and client details for the SAC connection, copy them somewhere safe

Generate Secret

Serving Endpoint Permissions

The service principal requires execute/query permission on the model serving end point.
We can add the permission via code or the UI as below

Model Serving Endpoint

Model Serving Endpoint Details

Model Serving Endpoint Permissions

SAP Analytics Cloud Connection

Switch to SAC, we need to define a new HTTP API connection

SAP Analytics Cloud HTTP API ConnectionSAP Analytics Cloud HTTP API Connection DialogueSAP Analytics Cloud HTTP API Connection with Placeholders

When we click "Create" it validates the connection details.

SAC Multi Action

Define a new SAC Multi Action using the connection defined.

SAP Analytics Cloud New Multi ActionMulti Action - Add API StepMulti Action API Step

Within the Multi Action properties we can select the previously defined connection.
The API Request URL is required again here, we can paste in the Model Serving Endpoint here.

SAP Analytics Cloud Story

The multi action defined above needs to be included within an SAC story

SAC Story Add Multi Action Starter ComponentSAC Multi Action Starter Properties

We can test the Multi Action call by clicking on the play button.

SAC Execute Multi ActionSAC Multi Action Complete

All being well, once completed we can see an additional notification available

SAC Multi Action Notifications

SAP Analytics Cloud Job Monitor

The Multi Action history is captured within the Job Monitor

SAP Analytics Cloud - System Job MonitorJob Monitor History

Conclusion

In this blogpost I have shown that integrating a Databricks model serving endpoint can be achieved with only a few steps using out of the box features from Databricks and SAP Analytics Cloud.

C_PURCHASEORDERITEMDEX - Add the creation date field of the purchase order item

2026-01-07T08:30:47.905000+01:00

Hello everyone,

I recently raised a case with SAP regarding the standard CDS View for purchase order items, C_PURCHASEORDERITEMDEX, which currently returns the header creation date for all items. After reviewing the case, the SAP support team confirmed that this is the expected behavior and suggested submitting an idea on their Influence platform. My ERP is S/4HANA Public Cloud.

The goal of this blog post is to gather support from the community so that this improvement can be prioritized. If you agree that having the correct item-level creation date would add value, please take a moment to vote for my idea on the SAP Influence website. Your vote can make a real difference!

https://influence.sap.com/sap/ino/#/idea/362226

Thank you in advance for your support.

Best regards,

Miquel

SAP Datasphere News in December

2026-01-09T10:11:58.169000+01:00

SAP Datasphere News in December

What a year it has been. Wow! 2025 was packed with great events, lots of new features and enhancements to extend existing features, and the launch of SAP Business Data Cloud.
Explore the latest updates in our community news blogs and enjoy watching my top feature highlights for 2025 on YouTube 📺 below.

SAP Datasphere 2025 highlights in a nutshell

What a great ride it has been in 2025 with SAP Datasphere!
Let me share a few of my personal highlights.

SAP Business Data Cloud

The announcement of SAP Business Data Cloud in February this year made quite some noise and puts data, analytics and AI at the center of our portfolio at SAP. We delivered SAP Business Data Cloud in May 2025 as a fully managed software as a service solution that unifies and governs all SAP data and seamlessly connects with third-party data.

It is an evolution of the business data fabric journey we started with SAP Datasphere and SAP Analytics Cloud some years ago and also offers a path forward for your investments in SAP BW.

Delivering out-of-the-box always-on data products is the next pivotal step in our data strategy following a harmonized data model to provide scalable access without duplication using zero-copy delta share technology.

SAP BDC serves as the foundation for your data and AI projects leveraging SAP managed and custom data products, including our growing open partner ecosystem with Databricks and the newly announced partnerships with Google Big Query, Snowflake and Microsoft Fabric.

My top features in 2025

Picking my top features for all of 2025 was not easy and here is my top feature list in no particular order.

Object store (general availability)

In 2025 we delivered the option to include an object store next to the well-known SAP HANA Cloud based persistency for a cost-efficient storage. You can use the existing tooling you already know to now create local tables on files, use replication flows, transformation flows which are then using Spark as compute engine - all orchestrated by task chains. You can load large quantities of data via replication flows and prepare the data using transformation flows and then share data to other spaces to be used as a source for additional flows, views, and of course analytic models. A really great enhancement!

Versioning of data builder objects

Object versioning has been introduced for views, analytic models, data access controls, and local tables. The new object versions are created automatically at the time of deployment which allows you now to list the past versions of objects, open a past version in read-only mode in a new tab, download a past version to a CSN/JSON file and restore a past version to replace the current version.

Be aware that the saving of an error-free restored version overwrites the current object. It is also important to understand that the versioning happens for the metadata of an object and not the data.

Efficiency features

Then there are tons of smaller enhancements that have been delivered in 2025 to increase efficiency and productivity overall. Let me just name a few here for the different components of Datasphere.

Analytic model

stacking and reusing of models
support for structures and data access controls for analytic models
we introduced a measure-dependency graph, the fiscal time dimension and unit conversion
and many enhancements with variables and parameters

Replication flows

delta only load type
support for views
reusing of source objects in multiple flows
many new sources and targets

Transformation flows

batch processing
parameter support
HANA tables as source in Spark transformation flows
other features bringing Spark transformation flows almost on par with HANA Cloud based ones

Task chains

deep retry of failed tasks in nested task chains
Rest API
Parameter support
notification tasks
amongst other improvements

Catalog

Support for SAP BW/4HANA, seamless planning, SAP BDC data products, Databricks via BDC connect
GenAI-assisted generation of catalog asset descriptions and classifications
Mass import of existing glossaries, terms, KPI’s, etc. from flat files

SAP Cloud application lifecycle management

SAP Datasphere is now integrated into SAP Cloud application lifecycle management (ALM) for health, job and automation monitoring.

Health monitoring enables you to check the health of one or more Datasphere tenants from the Health Monitoring app in Cloud ALM to check on

Memory and Disk usage
Out of Memory Events
Task Failures and
Admission Control Events

Job and automation monitoring enables you to monitor the tasks runs and executions in one or more SAP Datasphere tenants from the Job and Automation Monitoring app in Cloud ALM.

The integration overall eases the monitoring of an SAP Datasphere tenant in a complex customer landscape with a centralized, consistent way of tracking the tenants.

In addition, the early watch alert reports are now also available for SAP Datasphere. EWA is an automatic service analyzing the essential administrative areas of your SAP systems and covered by your maintenance agreement.

Data access controls based on IdP attributes

Data access controls based on identity provider (IdP) attributes are a small enhancement, but it is very powerful. You can now use values in custom attributes provided by your IdP as identifiers in your data access controls and define conditions that apply to users with these values. It is powerful because using these attributes allows you to apply a single condition to all users whose accounts contain that specific attribute value. In other words, you can group users with attributes to assign permissions to a group instead of individual users.

YouTube feature highlights of 2025

And more feature highlights of 2025

There are so many more great features which we delivered in 2025, so in addition to my 2025 top features, I pulled together the 2025 feature highlights in all different areas of SAP Datasphere for you. Enjoy the ride!

Data Integration

Catalog & Monitoring

Data Modelling

Miscellaneous

What exactly is rewiring for SAP Datasphere?

I have heard that question so many times over the last few months from customers, partners, but also SAP colleagues at events like SAP TechEd in Berlin or Sydney.
During these discussions I observed that people unintentionally misinterpret the term “rewiring”. It sounds technical, hence there are associations with data, object or even full tenant migrations. This is not the case!

Therefore, I wrote a blog to clarify what rewiring is, and what it is not. Moreover, give you some guidance on the steps which you need to take.
🔗 https://community.sap.com/t5/technology-blog-posts-by-sap/rewiring-of-sap-datasphere-to-sap-business-data-cloud/ba-p/14292149

More blogs about SAP BDC and Datasphere to check out 👇

Find more information and related blog posts on the topic page for SAP Datasphere. You will find further product information on our Community with various subpages about Business Content, the SAP BW Bridge as well as content for Best Practices & Troubleshooting. Also check out the new support content for SAP Datasphere on SAP Help for troubleshooting and analysis guides, how-to guides, technical details, and more.

Find out how to unleash the power of your business data with SAP’s free learning content on SAP Datasphere. It’s designed to help you enrich your data projects, simplify the data landscape, and make the most out of your investment. Check out even more role-based learning resources and opportunities to get certified in one place on SAP Learning site.

SAP Business Data Cloud Certification (C_BCBDC): The Ultimate Guide to Passing SAP’s New AI Scenario

2026-01-12T13:03:12.892000+01:00

Introduction

Certification is changing, and it is a good thing. Instead of cramming facts and guessing at multiple choice, you are invited to think like you do on the job. You will work through real scenarios, connect the dots, and show how you deliver outcomes. In this post we will talk about what is new, and how to prepare for SAP BDC (C_BCBDC) certification.

Understanding the New BDC Certification Format - Scenario Based

You will interact with an AI avatar to respond to real customer or project situations. For learners who prefer not to use AI there is an alternative option to upload a video for human review. Link for more details : SAP Certification is Changing What You Need to Know about the New Format, Reimagining Certification

Preparation

SAP BDC Learning Journey

For the best chance of success, complete the full SAP Business Data Cloud learning journey from start to finish. Treat every module, hands on exercise, and quiz as required to build real exam readiness. Give special attention to the core topics below

Core Topics in SAP Business Data Cloud

1. BDC

One domain model
SAP 4 Me
Formations
Intelligent Applications
Data Products
Data Security
Architecture
BDC Cockpit
Zero copy
ORD / CSN
Data Lake

2. SAP Datasphere Integration

BDC-generated spaces
Harmonization through modeling
Semantic enrichment
Extending SAP-delivered models
Installing data products to a space

3. SAP Analytics Cloud (SAC)

Role of SAC in BDC
Extending and discovering intelligent applications
Use cases for intelligent applications

4. BW Modernization

Lift, shift, and innovate strategies
Data Product Generator
Subscriptions and versions
Why BW is included in BDC

5. SAP Databricks Integration

ML Flow, delta share, notebooks, Unity Catalog
Exploratory Data Analysis
Industry use cases
Sharing data products and ML results between Databricks and BDC

Sample Scenarios to Understand the Exam Format

Scenario 1: Global Manufacturing Company – Unified Analytics & Governance

Context: A global manufacturing company operates across Europe and Asia. They run SAP S/4HANA, legacy SAP BW, and several non-SAP systems for logistics and procurement. Leadership wants a single, governed analytics platform with reusable data products and ready-made insights.

Question: How does BDC help

Scenario 2: Financial Services Firm – AI-Driven Risk & Compliance Reporting

Context: A financial services firm needs real-time risk reporting, strict compliance, and AI-driven forecasting. They must integrate SAP finance data with external market feeds and apply advanced analytics.

Question: How can Financial Services Firm integrate third-party data with its SAP data using SAP Business Data Cloud?

Exam Questions Breakdown from My Attempt

Based on my recent experience, here’s how the exam questions were distributed:

Technical Knowledge of BDC: 40%
Integration with SAP Business Warehouse: 15%
Technical Knowledge of SAP Analytics Cloud: 15%
Technical Knowledge of SAP Datasphere: 15%
Technical Knowledge of SAP Databricks: 15%

Helpful Links

Working with BW Authorizations in Datasphere on Enterprise Level

2026-01-14T22:07:10.971000+01:00

Related resources

Blog Post: Introducing Remote Authorizations from SAP BW/4HANA for SAP Datasphere
Blog Post by @Martin_Kuma: BW-Like Authorizations in Datasphere (DSP)
Documentation: Import SAP BW and SAP BW∕4HANA Analysis Authorizations
Note: 3062381

Why resolve BW authorizations in Datasphere?

In enterprise analytics SAP Datasphere often needs to federates the SAP BW analysis authorization table RSDWC_RSEC_DAC to data access controls (DAC). We build on this SAP standard (no custom BW developments are required beyond generation/extraction of the standard table) and bring the resolution step into the HANA layer for scale and performance.

Performance matters: The default runtime-only resolution evaluates long WHERE predicate strings per session user and iterates them at query time. With many users and complex rules, that becomes inperformant and unpredictable in terms of latency. By pre‑resolving per objectinto a persisted table we decouple expensive checks from BI queries, achieve consistent, auditable results, and enable scheduled refreshes.

Prerequisites

RSDWC_RSEC_DAC as a Remote Table in Datasphere. Generate/refresh it in BW with transaction RSDWC_DAC_RSEC_GEN and the BAdI RSDWC_DAC_RSEC_USER_UPDATE (and optionally schedule report RSDWC_DAC_RSEC_EXTRACT). The table can then be exposed to Datasphere.
A HANA DB user (Open SQL schema) in Datasphere. This user owns the Function, Procedure and Table and is granted to the space technical user to run in Task Chains.
Optional: an Entities/"booked values" table/view. Applying predicates to the set of actually booked combinations avoids generating all possible cross‑combinations which can be a huge performance multiplier and inflate result sizes.

Where our default approaches falls short

Runtime resolution per current user leads to high CPU/time due to long predicate strings.
SQL views in Datasphere lack loops/cursors; scaling to many users is cumbersome.
Duplicating views per user and UNION them is not maintainable and limits parallelization.

Example at a glance

In this blog’s walkthrough OBJECT01 is our authorization object that groups the key BW characteristics used for data access in the BWAUTH02 mapping. Concretely, OBJECT01 controls combinations of the following standard InfoObjects:

/BIC/0COMP_CODE (Company Code)
/BIC/0PLANT (Plant)
/BIC/0SALESORG (Sales Organization)
/BIC/0PROFIT_CTR (Profit Center)
/BIC/0CS_PLEVEL

The SAP standard table RSDWC_RSEC_DAC (federated as a remote table to Datasphere) holds per BW user and per object the dynamic WHERE‑predicate string that expresses allowed values/ranges for these characteristics.

Solution overview: Function + Procedure + Table + Task Chain + View

We implement five building blocks in an Open SQL schema and orchestrate them from Datasphere:

Table Function `TF_AUTH_RESOLVE_OBJECT01_BWAUTH02`: reads normalized WHERE predicate from RSEC view and applies it to booked entities via `APPLY_FILTER`.
Procedure `SP_AUTH_OBJECT01_REFRESH_BWAUTH02`: loops over all BW users for OBJECT01 and persists the union of results into a target table (full refresh).
Persisted table `AUTH_OBJECT01_RESULT_BWAUTH02`: mirrors function output for /BIC/0* fields.
View `GV_AUTH_OBJECT01_RESULT_BWAUTH02`: exposes the table for DACs and downstream pipelines.
Task Chain `TC_AUTH_OBJECT01_REFRESH_BWAUTH02`: schedules the refresh procedure regularly.

Source artifacts and naming

RSEC (clean) view: "DEMO_AR_001"."GV_RSDWC_RSEC_DAC_BWAUTH02" (column FILTERFIELDNM holds the normalized predicate)
Entities view (booked combinations): "DEMO_AR_001"."GV_OBJECT01_DAC_ENTITIES_BOOKED_BWAUTH02"
Target table: "DEMO_AR_001#HDB_USER01"."AUTH_OBJECT01_RESULT_BWAUTH02"
Exposing view: "DEMO_AR_001"."GV_AUTH_OBJECT01_RESULT_BWAUTH02"
Task Chain: TC_AUTH_OBJECT01_REFRESH_BWAUTH02

Target table

CREATE COLUMN TABLE "DEMO_AR_001#HDB_USER01"."AUTH_OBJECT01_RESULT_BWAUTH02" (
    "User"             NVARCHAR(256),
    "OBJECTNAME"       NVARCHAR(40),
    "/BIC/0PLANT"      NVARCHAR(20),
    "/BIC/0SALESORG"   NVARCHAR(20),
    "/BIC/0CS_PLEVEL"  NVARCHAR(20),
    "/BIC/0PROFIT_CTR" NVARCHAR(20),
    "/BIC/0COMP_CODE"  NVARCHAR(20)
);

Table Function

/* --------------------------------------------------------------------------------------------------
   Object:   FUNCTION "DEMO_AR_001#HDB_USER01"."TF_AUTH_RESOLVE_OBJECT01_BWAUTH02"
   Purpose:  For a given BW user, returns the authorized ENTITY combinations
             for OBJECT01 (fixed here; can be parameterized later).
             The WHERE predicate string is read dynamically from the RSEC view
             and applied to the Entities view via APPLY_FILTER.

   Input:    P_USER (NVARCHAR(256)) – BWUSERID for which the authorization resolution is executed.

   Output:   Result set with the following columns (aligned to the new standard InfoObjects):
             "User", "OBJECTNAME",
             "/BIC/0PLANT", "/BIC/0SALESORG", "/BIC/0CS_PLEVEL",
             "/BIC/0PROFIT_CTR", "/BIC/0COMP_CODE"

   Assumptions & behavior:
     - Exactly ONE record exists in the RSEC view for (BWUSERID = P_USER, OBJECTNAME = 'OBJECT01_BWAUTH02').
       -> SELECT ... INTO expects exactly 1 row. More than one -> "row not unique"; none -> "no data found".
     - INFOOBJ is intentionally fixed to 'OBJECT01_BWAUTH02' for this scenario.
     - APPLY_FILTER expects a syntactically valid WHERE string (without the "WHERE" keyword).

   Security/runtime:
     - SQL SECURITY INVOKER: Privilege checks are executed in the caller's context.
       -> Caller must have SELECT on the views used in the space/schema.
     - APPLY_FILTER is safer than EXEC/EXECUTE IMMEDIATE regarding SQL injection,
       however FILTERFIELDNM must be maintained reliably in the RSEC view.
---------------------------------------------------------------------------------------------------*/
CREATE FUNCTION "DEMO_AR_001#HDB_USER01"."TF_AUTH_RESOLVE_OBJECT01_BWAUTH02"(
    IN P_USER NVARCHAR(256)
)
RETURNS TABLE (
    "User"           NVARCHAR(256),
    "OBJECTNAME"     NVARCHAR(40),
    "/BIC/0PLANT"    NVARCHAR(20),
    "/BIC/0SALESORG" NVARCHAR(20),
    "/BIC/0CS_PLEVEL" NVARCHAR(20),
    "/BIC/0PROFIT_CTR" NVARCHAR(20),
    "/BIC/0COMP_CODE"  NVARCHAR(20)
)
LANGUAGE SQLSCRIPT
SQL SECURITY INVOKER
AS
BEGIN
    /* Local variables:
       - FILTER_CLAUSE: holds the dynamic WHERE predicate read from the RSEC view
       - INFOOBJ: fixed to OBJECT01 (current scenario)
    */
    DECLARE FILTER_CLAUSE NCLOB;
    DECLARE INFOOBJ NVARCHAR(40);

    /* Fixed authorization object; can be modeled as IN parameter if needed */
    INFOOBJ = 'OBJECT01';

    /* 1) Load filter string for the user.
          Expects exactly one match. If SELECT returns 0 or >1 rows, HANA raises an exception.
          This should be ensured in the RSEC view content. */
    SELECT FILTERFIELDNM
      INTO FILTER_CLAUSE
      FROM "DEMO_AR_001"."GV_RSDWC_RSEC_DAC_BWAUTH02"
     WHERE BWUSERID   = :P_USER
       AND OBJECTNAME = INFOOBJ;

    /* 2) APPLY_FILTER:
          - Applies the dynamic WHERE expression to the entities view.
          - The result is materialized in the implicit table variable lt_result. */
    lt_result = APPLY_FILTER(
        "DEMO_AR_001"."GV_OBJECT01_DAC_ENTITIES_BOOKED_BWAUTH02",
        :FILTER_CLAUSE
    );

    /* 3) Assemble result:
          - DISTINCT -> avoid duplicates.
          - P_USER and INFOOBJ are carried as constant columns. */
    RETURN
    SELECT DISTINCT
        :P_USER    AS "User",
        INFOOBJ    AS "OBJECTNAME",
        "/BIC/0PLANT",
        "/BIC/0SALESORG",
        "/BIC/0CS_PLEVEL",
        "/BIC/0PROFIT_CTR",
        "/BIC/0COMP_CODE"
    FROM :lt_result;
END;

You can check how it is working:

SELECT * FROM "DEMO_AR_001#HDB_USER01"."TF_AUTH_RESOLVE_OBJECT01_BWAUTH02"('ABC02')

Procedure

/* --------------------------------------------------------------------------------------------------
   Object:   PROCEDURE "DEMO_AR_001#HDB_USER01"."SP_AUTH_OBJECT01_REFRESH_BWAUTH02"
   Purpose:  Full-refresh build of table "AUTH_OBJECT01_RESULT_BWAUTH02":
             - Collect all BWUSERID relevant for OBJECT01 (from GV_RSDWC_RSEC_DAC_BWAUTH02)
             - For each user, call table function TF_AUTH_RESOLVE_OBJECT01_BWAUTH02
             - Persist the union of results into the target table

   Flow:
     1) Cursor over DISTINCT BWUSERID from the RSEC view (OBJECTNAME = 'OBJECT01')
     2) DELETE the target table (clear all rows for a full rebuild)
     3) FOR-loop: INSERT ... SELECT from the table function for each user

   Security:
     - SQL SECURITY INVOKER: privilege checks run in caller context
     - Caller requires:
       * SELECT on source RSEC view
       * INSERT/DELETE on target table

   Performance:
     - Loop calls the TF once per user (N calls).
       For large N, consider a set-based approach (bulk table function) later.

   Notes:
     - INSERT lists columns explicitly to guard against future schema changes
---------------------------------------------------------------------------------------------------*/
CREATE PROCEDURE "DEMO_AR_001#HDB_USER01"."SP_AUTH_OBJECT01_REFRESH_BWAUTH02" ()
LANGUAGE SQLSCRIPT
SQL SECURITY INVOKER
AS
BEGIN

  /* 1) Cursor: distinct list of relevant users for OBJECT01 */
  DECLARE CURSOR cur_users FOR
    SELECT DISTINCT BWUSERID
      FROM "DEMO_AR_001"."GV_RSDWC_RSEC_DAC_BWAUTH02"
     WHERE OBJECTNAME = 'OBJECT01'
       AND BWUSERID   IS NOT NULL;

  /* 2) Full rebuild: clear target table */
  DELETE FROM "DEMO_AR_001#HDB_USER01"."AUTH_OBJECT01_RESULT_BWAUTH02";

  /* 3) Implicit FOR-loop: HANA opens/closes the cursor automatically */
  FOR cur_row AS cur_users DO

    INSERT INTO "DEMO_AR_001#HDB_USER01"."AUTH_OBJECT01_RESULT_BWAUTH02"
      ( "User", "OBJECTNAME",
        "/BIC/0PLANT", "/BIC/0SALESORG", "/BIC/0CS_PLEVEL",
        "/BIC/0PROFIT_CTR", "/BIC/0COMP_CODE" )
    SELECT
        "User", "OBJECTNAME",
        "/BIC/0PLANT", "/BIC/0SALESORG", "/BIC/0CS_PLEVEL",
        "/BIC/0PROFIT_CTR", "/BIC/0COMP_CODE"
    FROM "DEMO_AR_001#HDB_USER01"."TF_AUTH_RESOLVE_OBJECT01_BWAUTH02"( :cur_row.BWUSERID );

  END FOR;

END;

Execute via Task Chain step or manually:

CALL "DEMO_AR_001#HDB_USER01"."SP_AUTH_OBJECT01_REFRESH_BWAUTH02"();
SELECT * FROM "DEMO_AR_001#HDB_USER01"."AUTH_OBJECT01_RESULT_BWAUTH02";

Security, view exposure, and task chain

Create Datasphere view: "DEMO_AR_001"."GV_AUTH_OBJECT01_RESULT_BWAUTH02" over the table to feed DACs and downstream processes.

Schedule a Task Chain "TC_AUTH_OBJECT01_REFRESH_BWAUTH02" that calls the procedure for regular refresh.

To add the procedure your user needs to be granted:

-- Minimal grants (adapt to your technical user / space)
GRANT EXECUTE ON PROCEDURE "DEMO_AR_001#HDB_USER01"."SP_AUTH_OBJECT01_REFRESH_BWAUTH02" TO "DEMO_AR_001";
GRANT SELECT, INSERT, DELETE ON "DEMO_AR_001#HDB_USER01"."AUTH_OBJECT01_RESULT_BWAUTH02" TO "DEMO_AR_001";
-- Optional: schema-wide
GRANT EXECUTE ON SCHEMA "DEMO_AR_001#HDB_USER01" TO "DEMO_AR_001" WITH GRANT OPTION;

If you do not know the needed Schema:

SELECT SCHEMA_NAME, SCHEMA_OWNER
FROM SYS.SCHEMAS

Performance & scalability

Pre-resolving moves string evaluation out of query-time; persisted results enable indexes and faster joins.
Keep the entities view "booked" to avoid generating all combinations (Cartesian blow-up).
For very large user counts, consider a set-based bulk table function to avoid N function calls.

Troubleshooting

Error 260 invalid column name: fix duplicated /BIC/ prefixes or mismatched InfoObject names.
No data found / row not unique: enforce exactly 1 row per (BWUSERID, OBJECTNAME) in RSEC view.
Permissions: INVOKER needs SELECT on sources and INSERT/DELETE on target; scheduler user needs EXECUTE on procedure.

What you get

A maintainable, enterprise-ready pattern to pre-resolve BW authorizations in SAP Datasphere, fully based on SAP standard RSDWC_RSEC_DAC: reproducible results, faster queries and a scheduling hook via Task Chains. From here, you can attach DACs to exposed views or plug the table into harmonization and governance pipelines.

Generating and Integrating Automated Predictive Library (APL) Forecasts in a Seamless Planning Model

2026-01-20T11:01:19.924000+01:00

Introduction

SAP Analytics Cloud has always been the one solution for BI, planning and predictive analytics. As such, it has powerful built-in capabilities for regression, classification and time-series forecasting. You know them as Predictive Scenarios and many of you have used them to support their planning processes. Our predictive scenarios are perfect for business users: they choose the best available algorithm for your data and explain the results while maintaining the semantics of the model throughout the process (e.g., hierarchies). With SAP Business Data Cloud and seamless planning, data scientists on the other hand can now leverage HANA’s predictive Predictive Analysis Library (PAL) and Automated Predictive Library (APL) directly on the HANA database of SAP Datasphere and nicely integrate the results into planning processes. This lets them tweak predictive models by picking and choosing the algorithm of their choice and use code instead of a UI. SAP BDC would also allow them to share data with SAP Databricks or another Databricks instance using data products if this was their preferred environment.

This blogpost was created with @marc_daniau, a development expert for our predictive engine. We want to demonstrate the usage of HANA APL in combination with a seamless planning model and live versions. We do this using a straight-forward prediction based on actual data.

High-level overview

This is what we are working with:

In SAP Analytics Cloud, we have a planning model deployed to an SAP Datasphere space. That makes it a seamless planning model which means that its data is not stored on the SAP Analytics Cloud database but only on the SAP Datasphere database.

In the SAP Datasphere space, we find the planning model data and a table with actuals. We are not using an SAP BDC data product, but you surely could!

We create our prediction directly on the underlying HANA Cloud database of SAP Datasphere. We created a DB user to access the database. There, we consume the actuals, create a stored procedure which we can trigger via a task chain, and surface the result in a view in the space. This fact data can then be added to the seamless planning model as a live version.

Step-by-step

1. Seamless planning model

We do not cover the full creation and set-up of the model. Let’s just check out its key characteristics:

It is a seamless planning model, deployed to the space Sales Planning Demo

Its fact table is exposed in the SAP Datasphere space (actually, this is not decisive for the use case described here but could be useful if you want to add budget data as influence for your predictive model, for instance)

We want to create a forecast version and predict the measure SALES_REVENUE along the product and region dimension

2. SAP Datasphere space

Again, we do not look at the creation of the space and all its artefacts. The following Actuals view is key as we extrapolate our forecast based on this data:

Measures and attributes nicely match the planning model structure (which is handy for our simple demo scenario J)

We excluded columns that we do not need in a projection

We filtered on date as we do not want to use the entire history

3. Setting up database access

We are now ready to learn how to create the forecast on the HANA database. First of all, we need to set up database access.

Prerequisite: Enable the SAP HANA Cloud Script Server on Your SAP Datasphere Tenant

Navigate to Space Management, find your space and Edit.

Navigate to Database Access and create a new user
Name your user and make the needed settings as highlighted. Your user’s name will be a concatenation of your space name, ‘#’ and the suffix you provide here.
Mark your user and open the database explorer. The password can be retrieved in the details of the user (information symbol).

4. Database explorer

Let’s first have an overview of what we are creating in the database explorer:

We create a view that consumes the actual data from our space schema. Note the following:

We concatenate Product and Region to one Entity column that we will use to segment our prediction. As we work directly on flat fact tables/views, we do not have the luxury of keeping all semantics as we have it in the SAP Analytics Cloud predictive scenarios.

CREATE VIEW "SALES_PLANNING_DEMO#AI_USER"."APL_SERIES_IN" ( "Entity", "Date", "SalesRevenue" ) AS (select   
  "Product" || '|' || "Regions" as "Entity", "Date", "SalesRevenue" 
 from  
  SALES_PLANNING_DEMO."V_Actual_Sales_Data" 
 order by 1, 2)

We now create the prediction task as a stored procedure.

create procedure "APL_FORECAST_TASK" 
as BEGIN
    declare header "SAP_PA_APL"."sap.pa.apl.base::BASE.T.FUNCTION_HEADER";
    declare config "SAP_PA_APL"."sap.pa.apl.base::BASE.T.OPERATION_CONFIG_DETAILED";
	declare var_desc "SAP_PA_APL"."sap.pa.apl.base::BASE.T.VARIABLE_DESC_OID";   
    declare var_role "SAP_PA_APL"."sap.pa.apl.base::BASE.T.VARIABLE_ROLES_WITH_COMPOSITES_OID";    
    declare apl_log   "SAP_PA_APL"."sap.pa.apl.base::BASE.T.OPERATION_LOG";      
    declare apl_sum   "SAP_PA_APL"."sap.pa.apl.base::BASE.T.SUMMARY";   
    declare apl_indic   "SAP_PA_APL"."sap.pa.apl.base::BASE.T.INDICATORS";   	
    declare apl_metr "SAP_PA_APL"."sap.pa.apl.base::BASE.T.DEBRIEF_METRIC_OID";
    declare apl_prop "SAP_PA_APL"."sap.pa.apl.base::BASE.T.DEBRIEF_PROPERTY_OID";      

    truncate table "SALES_PLANNING_DEMO#AI_USER"."APL_SERIES_OUT";
    truncate table "SALES_PLANNING_DEMO#AI_USER"."APL_FORECAST_ACCURACY";
    truncate table "SALES_PLANNING_DEMO#AI_USER"."APL_FORECAST_STATUS";
	
    :header.insert(('Oid', 'DSP APL'));
    :header.insert(('LogLevel', '2'));
    :header.insert(('MaxTasks', '4'));  --  PARALLEL TASKS

    :config.insert(('APL/SegmentColumnName', 'Entity',null)); 
    :config.insert(('APL/Horizon', '12',null));
    :config.insert(('APL/TimePointColumnName', 'Date',null));
	:config.insert(('APL/ForcePositiveForecast', 'true',null));
	:config.insert(('APL/DecomposeInfluencers', 'true',null));
	:config.insert(('APL/ApplyExtraMode', 'First Forecast with Stable Components and Residues and Error Bars',null));
	
    :var_role.insert(('Date', 'input', null, null, null));
    :var_role.insert(('SalesRevenue', 'target', null, null, null));
		
    "SAP_PA_APL"."sap.pa.apl.base::FORECAST_AND_DEBRIEF" (
	:header, :config, :var_desc, :var_role, 
	'SALES_PLANNING_DEMO#AI_USER', 'APL_SERIES_IN', 
	'SALES_PLANNING_DEMO#AI_USER', 'APL_SERIES_OUT', apl_log, apl_sum, apl_indic, apl_metr, apl_prop);

    insert into "SALES_PLANNING_DEMO#AI_USER"."APL_FORECAST_ACCURACY" 
    select "Oid" as "Entity", "MAE", "MAPE"
    from "SAP_PA_APL"."sap.pa.apl.debrief.report::TimeSeries_Performance" (:apl_prop, :apl_metr)
    where "Partition" = 'Validation';
	
	insert into "SALES_PLANNING_DEMO#AI_USER"."APL_FORECAST_STATUS" 
    select "OID" as "Entity", "VALUE" as "Task Status"
	from :apl_sum 
	where key = 'AplTaskStatus';
END

For our demo scenario we use the default APL forecasting method that automatically tries different hypotheses for trend, cycles and fluctuations, and eventually selects the combination that gives the best accuracy. For a faster processing on many segments, an option is to force the Exponential Smoothing method by adding to the procedure this line of code below:

:config.insert(('APL/ForecastMethod','ExponentialSmoothing',null));

This is the code to prepare the target tables of the procedure:

drop table APL_SERIES_OUT;
create table APL_SERIES_OUT (
 "Entity"  nvarchar(180),
 "Date" 	DATE,
 "SalesRevenue" DOUBLE,
 "kts_1" DOUBLE,
 "kts_1Trend" DOUBLE,
 "kts_1Cycles" DOUBLE,
 "kts_1_lowerlimit_95%" DOUBLE,
 "kts_1_upperlimit_95%" DOUBLE,
 "kts_1ExtraPreds" DOUBLE,
 "kts_1Fluctuations" DOUBLE,
 "kts_1Residues" DOUBLE
);

drop table APL_FORECAST_ACCURACY;
create table APL_FORECAST_ACCURACY (
 "Entity"  nvarchar(180),
 "MAE" DOUBLE,
 "MAPE" DOUBLE
);

drop table APL_FORECAST_STATUS;
create table APL_FORECAST_STATUS (
 "Entity"  nvarchar(180),
 "Task Status" nvarchar(180)
);

You can trigger this procedure manually using the command below:

call "APL_FORECAST_TASK" ;

However, in the next chapter, we will also create a task chain in SAP Datasphere to trigger it which can be embedded in real workflows.

You see we generate and write data into three different tables:
The prediction result goes into our results table called APL_SERIES_OUT. This is the data that we want for our seamless planning model. The table has the concatenated entity, date and the predicted revenue. It also comes with upper and lower limit predictions (95%) as well as with fluctuations, extra predictions etc.
Optionally, we create the table APL_FORECAST_ACCURACY to store the MAPE (Mean Absolute Percentage Error MAE (Mean Absolute Error) per entity. You could filter on the entities you are interested in, the best/worst entities etc. or you could get the MSE (Mean Squared Error) or RMSE (Root Mean Absolute Squared Error) as well.

Optionally, we create the table APL_FORECAST_STATUS where we log the prediction status per entity. All our entities were successful.

5. Task chain

Stored procedures can be executed via task chains. As such, you can execute them from the SAP Datasphere UI, schedule them or trigger them via external API. Check out the task chain documentation to learn more about prerequisites such as required roles. Soon, you should be able to trigger this API via multi actions in SAP Analytics Cloud as well.

We must allow the execution of the stored procedure via the SAP Datasphere UI including the creation and deletion of data in the database user schema:

CALL "DWC_GLOBAL"."GRANT_PRIVILEGE_TO_SPACE" ( 
             OPERATION => 'GRANT',  
             PRIVILEGE => 'INSERT',  
             SCHEMA_NAME => 'SALES_PLANNING_DEMO#AI_USER',  
             OBJECT_NAME => '',  
             SPACE_ID => 'SALES_PLANNING_DEMO');

CALL "DWC_GLOBAL"."GRANT_PRIVILEGE_TO_SPACE" ( 
             OPERATION => 'GRANT',  
             PRIVILEGE => 'DELETE',  
             SCHEMA_NAME => 'SALES_PLANNING_DEMO#AI_USER',  
             OBJECT_NAME => '',  
             SPACE_ID => 'SALES_PLANNING_DEMO');

CALL "DWC_GLOBAL"."GRANT_PRIVILEGE_TO_SPACE" ( 
             OPERATION => 'GRANT',  
             PRIVILEGE => 'EXECUTE',  
             SCHEMA_NAME => 'SALES_PLANNING_DEMO#AI_USER',  
             OBJECT_NAME => '',  
             SPACE_ID => 'SALES_PLANNING_DEMO');

Setting up task chains is simple. To add a stored procedures, you select Others and browse through the available procedures that are available for your space. Then you drag the procedure on to the canvas to add it as a task.

You can add replication flows, transformation flows, intelligent lookups etc. to your task chains. By that, you could for instance first refresh actuals and get them into shape so you can use them for your prediction. Or, you add an email notification task to receive updates after the execution of the task chain (I did that in the example below).

6. Consuming results in the SAP Datasphere space

Now that we have the predictive logic to calculate and we can execute it via the SAP Datasphere UI, we of course need to consume the forecast data in the planning model. To do that, we first need the results in the SAP Datasphere space. We create a view on top of the results table. We used a graphical view but depending on your preferences and skills, you may use an SQL view instead.

Pull the results table from the DB user schema in the Sources tab.

Add a calculation node to split the Entity column into regions and products again. Create two calculated columns (Product and Region) and use string functions. The functions SUBSTR_BEFORE() and SUBSTR_AFTER() can be used to split a string using the first occurence of a specified pattern (in our case '|' as the format of our Entity column is Product|Region).

Expression to derive product:

SUBSTR_BEFORE(Entity,'|')

Expression to derive region:

SUBSTR_AFTER(Entity,'|')

Join the standard time dimension with day granularity to derive the calendar month from the date. Standard time tables and dimensions can be generated in Space Management (link).

Add a projection and only keep the columns you need in the planning model.
- Calendar Month
- SalesRevenue
- Product
- Region

Make sure to expose the view for consumption and select Fact as the Semantic Usage Type.
Name the view and deploy.

7. Adding the forecast result as live version in the seamless planning model

We now move to SAP Analytics Cloud and add the forecast result to the seamless planning model as a live version.

Connect external data source.

Create a version to map the data into (or use an existing un-used version).

Select the view.
Map the columns.
Preview the data. You see that we have a live connection to the forecast results in SAP Datasphere. So every time that the forecast is updated, it will be reflected in the planning model in real time!

What (else) can/could you do with it?

In SAP Analytics Cloud:

You can display the live version data in the model data foundation and in tables, charts, etc. in stories.

You can reference the live version data in model and story calculations as well as data actions (incl. advanced formulas).

You cannot write back into the referenced views (or rather their underlying tables). But, you can copy the live version data into planning versions via copy/paste in the table, data actions and version management.

In SAP Datasphere:

You can of course leverage the forecast result in your views and analytic models and compare it to actuals, budgets, etc.

You can run further transformations and calculations and report on the results or use them in planning.

Which new features can improve the workflow in the future?

We are working on a couple of features that can make this scenario even better:

Push data from SAP Datasphere into the seamless planning model via task chains:
Live versions are awesome. But, sometimes you want to use the prediction as a proposal and then edit it. You could copy the live version data into an editable version easily (see above) but with a push of data into the planning model, you could directly bring the data into an editable forecast version. That push could be nicely added to the same task chain that triggers the APL procedure.

Trigger task chains from SAP Analytics Cloud:
Task chains shall soon offer a public API for triggering task chain runs from outside of SAP Datasphere.
We are getting ready on the SAP Analytics Cloud side to let you call this API via API steps in multi actions. With that, you could trigger the stored procedures from SAP Analytics Cloud.
Some day, we may have dedicated task chain steps in multi actions to ease this cross-orchestration. We also want to enable cross-orchestration in the opposite direction.

Conclusion

In this blogpost, Marc and I demonstrated how to integrate predictive forecast results from SAP HANA APL in a seamless planning model. We combined the power of SAP Datasphere, SAP HANA and SAP Analytics Cloud to achieve that in a quite straight-forward architecture. You could achieve more complex scenarios, use PAL instead to tweak your prediction more etc. Or you could use data products from SAP BDC to get you started even quicker.

We are looking forward to the future enhancements that shall improve such workflows and the overall integration of planning into SAP BDC!

Learn More

Seamless Planning - Product FAQ

Seamless Planning – Live Versions

SAP HANA Automated Predictive Library (APL)

SAP HANA Predictive Analysis Library (PAL)

SAP Analytics Cloud Predictive Scenarios

Connecting SAP Sales Cloud V2 to a Standalone SAP Analytics Cloud - A Practical Alternative

2026-01-21T05:54:01.435000+01:00

The Customer Challenge

A customer approached me with what seemed like a straightforward request: "We want to analyze our SAP Sales Cloud Version 2 data in SAP Analytics Cloud."

Their situation was common for mid-sized companies adopting SAP's CX portfolio:

SAP Sales Cloud Version 2 was live and running well
SAP Analytics Cloud was licensed separately (not the embedded version)
No SAP BTP subscription yet, it wasn't part of the initial scope
No SAP Integration Suite same reason
No SAP Datasphere, they wanted to avoid another data platform

The goal was simple: import Accounts, Contacts, and Opportunities into SAC for custom dashboards that combined CX data with data from other sources

What SAP Officially Recommends

Before building anything custom, I researched SAP's official guidance. The recommended architecture for Sales Cloud V2 analytics integration involves:

SAP Datasphere as a central data layer
SAP Integration Suite for event-driven data replication
SAP Advanced Event Mesh (optional but recommended) for resilience

SAP published a detailed two-part blog series on this: Event-Driven Data Integration from SAP Sales and Service Cloud V2 to SAP Datasphere

This architecture is elegant and production-grade. It's also significant additional investment for a customer who just wants to run a few reports.

Cost Implications

The official path requires more than you might expect:

Component	Status for this Customer
SAP Datasphere	Not licensed (€€€/month)
SAP Integration Suite	Not licensed (€€/month)
SAP BTP subscription	Not available
Implementation effort	Significant (weeks)

For a customer with ~24,000 accounts who wants three dashboards, this felt like using a sledgehammer to crack a nut.

The Obvious First Attempt: SAC's Built-in Connectors

SAC has connectors for various SAP sources. Surely there's something for Sales Cloud?

Attempt 1: The "SAP Cloud for Customer" Connector

SAC includes a dedicated connector called "SAP Cloud for Customer" that works beautifully with the older C4C (Cloud for Customer) system. It connects to OData services like:

https://<tenant>/sap/c4c/odata/v1/c4codataapi/AccountCollection

The problem: Sales Cloud V2 is not C4C. So the OData connector won’t work:

Attempt 2: The Generic OData Connector

SAC also has a generic "OData Services" connector. Maybe that works?

The problem: Sales Cloud V2 doesn't expose OData endpoints. The APIs on the SAP Business Accelerator Hub are REST, not OData.

GET /sap/c4c/api/v1/account-service/accounts ← REST, not OData
GET /sap/c4c/api/v1/contact-service/contacts ← REST, not OData

Attempt 3: Live Connection

Sales Cloud V2 has embedded SAC capabilities. Could we establish a live connection?

The problem: The embedded analytics are designed for "embedded" use cases. Connecting a standalone SAC tenant to Sales Cloud V2 for custom reporting seems to be... let's say "not a focus area" right now.

I opened a support ticket to clarify the expected behavior and it was put in development after a little back and forth.

The Hidden Challenge: Real-Time vs. Import

At this point, I had to step back and discuss architecture with the customer.

Even if a direct connector existed, there are implications to consider:

Approach	Pros	Cons
Live Connection	Always current data	Query performance depends on source; cannot combine with other data sources easily
Import Connection	Fast queries; can blend with other sources	Data has latency; storage costs

The customer wanted to combine Sales Cloud V2 data with SAP S/4HANA data in the same dashboard. This essentially requires import connections for both sources, plus a data model in SAC that joins them.

Real-time was never really an option as they needed data at rest in SAP Analytics Cloud

The Solution: A Lightweight REST-to-OData Proxy

Since the customer had:

No appetite for SAP Datasphere
No CPI license
A clear, limited scope (three entity types)
Tolerance for daily data refresh (not real-time)

I proposed building a thin proxy server that:

Authenticates with Sales Cloud V2
Fetches data via REST APIs
Transforms responses to OData format
Exposes endpoints that SAC can consume natively

Architecture

This sounds straightforward... until you start building it...

Challenge 1: The 999 vs. 1000 Mismatch

This one deserves its own section because it’s so unexpected.

SAP Sales Cloud V2 REST APIs have a maximum page size of 999 records. You can request $top=999, but not more.

SAP Analytics Cloud OData import has a minimum batch size of 1000 records.

Yes, you read that correctly. The source maxes out at 999. The target starts at 1000. I'll admit I stared at the logs for longer than I'd like to admit before I realized what was happening. The first 999 records imported perfectly. Then nothing. It took me a while to connect that SAC was waiting for a 1000-record batch that would never arrive.

The Solution

The proxy implements on-demand parallel fetching:

async function fetchAllRecords(entityType: string): Promise<any[]> {
  // Get first page to determine total count
  const firstPage = await fetchFromSalesCloud(entityType, {
    $top: 999,
    $skip: 0
  });
  
  if (firstPage.value.length < 999) {
    // All records fit in first page
    return firstPage.value;
  }
  
  // Calculate remaining pages and fetch in parallel
  const totalEstimate = firstPage.totalCount || 10000;
  const remainingPages = Math.ceil((totalEstimate - 999) / 999);
  
  const pagePromises = Array.from({ length: remainingPages }, (_, i) => 
    fetchFromSalesCloud(entityType, {
      $top: 999,
      $skip: 999 * (i + 1)
    })
  );
  
  const results = await Promise.all(pagePromises);
  return [firstPage.value, ...results.map(r => r.value)].flat();
}

The proxy fetches pages in parallel (not sequentially), caches the combined result, and serves it to SAC in appropriately sized batches. The cache has a configurable TTL (e.g., 1 hour) and automatically refreshes on the next request after expiry.

Challenge 2: Nested Objects and Flattening

Sales Cloud V2 returns nicely structured JSON with nested objects:

{
  "value": [{
    "accountId": "A001",
    "accountName": "ACME Corp",
    "primaryContact": {
      "contactId": "C001",
      "firstName": "John",
      "lastName": "Doe"
    },
    "addresses": [
      { "type": "BILL_TO", "city": "Berlin" },
      { "type": "SHIP_TO", "city": "Munich" }
    ]
  }]
}

SAC’s OData importer cannot handle this. The SAP Help documentation states clearly: Embedded Complex types are not supported.

The Solution

The proxy must flatten nested structures into separate entity sets:

/odata/Accounts         → Flat account records
/odata/Contacts         → Flat contact records  
/odata/AccountAddresses → One row per address, linked by accountId

Challenge 3: REST APIs Behave Differently Than OData

Coming from the OData world, I expected certain conventions:

$filter for filtering
$select for field selection
$expand for related entities
Standardized metadata endpoint

Sales Cloud V2 REST APIs have their own conventions:

Different query parameter syntax
No $metadata endpoint
Pagination via custom headers or response properties

The proxy needs to translate between these worlds, implementing just enough OData semantics to satisfy SAC’s importer.

Challenge 4: The $skip 10,000 Limit

Just when you think you’ve handled pagination, there’s another surprise waiting for datasets with more than 10,000 records.

Sales Cloud V2 returns an error if $skip is 10,000 or greater.

// This works
GET /accounts?$top=999&$skip=9000  ✅

// This fails
GET /accounts?$top=999&$skip=10000  ❌ Error!

So even if you’ve cleverly worked around the 999 page size limit, you hit a wall at record 10,001. Simple offset-based pagination ($skip) just stops working.

The Solution

Instead of offset-based pagination, the proxy uses keyset pagination with $skiptoken:

// src/constants.ts
export const SC_MAX_SKIP = 10000;

// When approaching the skip limit, switch to skiptoken
async function fetchWithKeysetPagination(entityType: string): Promise<any[]> {
  const allRecords: any[] = [];
  let nextLink: string | null = buildInitialUrl(entityType);
  
  while (nextLink) {
    const response = await fetch(nextLink);
    const data = await response.json();
    
    allRecords.push(...data.value);
    
    // Use the skiptoken from the response for next page
    nextLink = data['@odata.nextLink'] || null;
  }
  
  return allRecords;
}

Keyset pagination uses a token (typically based on the last record’s ID or timestamp) rather than a numeric offset. This approach:

Has no upper limit on the number of records
Is actually more performant than large offsets
Is the pattern SAP recommends for large datasets

If you have more than 10,000 records in any entity, you must use $skiptoken-based pagination. Simple $skip won’t cut it.

Implementation Notes

Technology Choice

I built the proxy with:

Node.js + Fastify: Lightweight, fast, excellent TypeScript support
On-demand caching: No scheduled jobs; cache fills dynamically on first request
Docker deployment: Runs on any container platform (I use Coolify on a VPS)

This is a production-ready setup. The proxy has been running reliably for the customer with no issues.

Smart Caching: On-Demand Parallel Fetching

Rather than running scheduled batch jobs, the proxy uses an on-demand approach:

First page served instantly: When SAC requests data, the proxy immediately returns the first page (up to 999 records) from Sales Cloud V2
Parallel fetching in background: While SAC processes the first page, the proxy fetches remaining pages in parallel
Cache populated for subsequent requests: By the time SAC requests page 2, the data is already cached

async function handleODataRequest(req: Request): Promise<Response> {
  // Check cache first
  if (cache.has(entityType) && !cache.isExpired(entityType)) {
    return serveFromCache(entityType, req.query);
  }
  
  // Fetch first page immediately
  const firstPage = await fetchPage(entityType, 0, 999);
  
  // Start parallel fetching of remaining pages (non-blocking)
  fetchRemainingPagesInBackground(entityType, firstPage.totalCount);
  
  // Return first page immediately
  return formatAsOData(firstPage);
}

This approach has several advantages:

No unnecessary background jobs: Data is only fetched when actually needed
Fast initial response: User doesn’t wait for full dataset
Self-healing cache: Expired cache automatically refreshes on next request

Optional: Deploying to SAP BTP Cloud Foundry

If you already have a BTP subscription with available credits, deploying there can make sense. You’re using infrastructure you’re already paying for. The migration is straightforward:

Standalone Setup Cloud Foundry Setup

Environment variables for credentials	Destination Service binding
In-memory cache	PostgreSQL Hyperscaler Option (free tier available)
Direct HTTP endpoint	XSUAA + AppRouter

Why PostgreSQL if you move to BTP?

On a single-container deployment (VPS e.g. with Coolify), in-memory caching works perfectly. But Cloud Foundry typically runs multiple app instances for high availability. If you need persistence across instances, PostgreSQL Hyperscaler Option is the lightweight choice. It has a free tier (since November 2025) and requires no additional management.

The additional effort for Cloud Foundry deployment is roughly:

Add mta.yaml with service bindings (~1 hour)
Configure Destination Service for Sales Cloud V2 credentials (~30 min)
Add XSUAA for authentication (~1 hour)
Test and deploy (~2 hours)

Total: Less than one additional day if you decide BTP is the right fit.

Don’t just default to BTP just because it’s SAP. Evaluate what you already have and what makes sense for your situation. A €5/month VPS running Docker might be exactly right for your use case.

OData Response Format

SAC expects OData V2or V4 responses. The minimum viable format:

{
  "d": {
    "results": [
      { "AccountID": "A001", "AccountName": "ACME Corp", ... },
      { "AccountID": "A002", "AccountName": "Globex", ... }
    ]
  }
}

Plus a $metadata endpoint returning EDMX that describes your entity types.

Authentication

Sales Cloud V2 supports:

Basic Authentication (simplest for server-to-server)
OAuth 2.0 (more secure, recommended for production)

The proxy stores credentials securely and handles token refresh for OAuth.

Results

After implementing the proxy:

SAC successfully imports Accounts, Contacts, and Opportunities
Data refreshes nightly via scheduled import
Customer can blend CX data with S/4HANA data in unified dashboards
No additional SAP platform licenses required
Total implementation time: ~2 days

When This Approach Makes Sense

The right solution depends on what you have and what you need. Here’s how I think about it:

A lightweight proxy makes sense when:

Limited scope (specific entities, not the entire data model)
Import/batch refresh is acceptable (not real-time requirements)
You don’t have Datasphere or CPI licensed and don’t need them for other use cases
You want to validate the integration quickly before investing in bigger infrastructure

Datasphere + CPI makes sense when:

You need event-driven, near real-time data replication
You’re integrating many entity types with complex relationships
You already have Datasphere or CPI licensed then use what you’re paying for
You have a broader data strategy where Datasphere is the central hub anyway

The key question: What do you already have, and does this use case justify adding more?

SAP’s platforms are excellent, no question! But sometimes a focused solution that solves exactly your problem is better than pulling in the big guns for a straightforward requirement.

Lessons Learned

1. Don’t Assume SAP Products Integrate Seamlessly

Sales Cloud V2 and SAC are both modern SAP cloud products. The integration gap exists because V2’s move away from OData wasn’t accompanied by a new native SAC connector (yet).

2. Check API Patterns Before Committing to Architecture

The shift from OData to REST in Sales Cloud V2 is documented, but easy to miss if you’re coming from C4C experience. Always verify on api.sap.com.

3. Test with Realistic Data Volumes

The 999/1000 mismatch only surfaces with more than 999 records. Demo systems with 50 accounts work fine. Production breaks.

4. Use What You Have, Validate What You Need

The official architecture is powerful. But if you don’t already have Datasphere and CPI licensed, ask yourself: does this specific use case justify adding them? Sometimes a focused solution that solves exactly your problem is the right call. You can always scale up later if requirements grow.

What’s Next?

I’ve published a reference implementation on GitHub that demonstrates the core patterns:

REST-to-OData transformation
Parallel fetching with caching
The 999→1000 batch bridging logic
Keyset pagination with $skiptoken for large datasets (>10k records)

👉 GitHub: SAP Sales Cloud V2 OData Proxy

A note on scope: The repository shows how the solution works, but doesn’t include everything from my production implementation. Some aspects are tied to specific client configurations that I can’t share. If you’re facing a similar challenge and want help getting to production faster, feel free to reach out.

If you’ve built something similar or found a better approach I’d love to hear about it in the comments.

Filling the gab between BW and Datasphere (a bit): Implementing a few standard variables

2026-01-22T15:38:53.605000+01:00

Introduction

Welcome to the first blog in a series where I'll try my best to close up some of the functionality gabs between the old SAP BW system and SAP Datasphere.
With the end of life for the old SAP BW systems in sight, companies and developers have been diving deep into SAP Datasphere, to build new, exciting solutions, in addition to migrating existing reports.

Personally, I've been involved with several Datasphere implementations now, and one thing I have found myself missing a lot was the standard Exit variables for things like "Current month", "Current year" and so on.

To make my life, and hopefully yours as well, I've recreated a set of the old date / calendar variables.
All the views are bundled in a nice package, which you can download from this GitHub repo.

How to install:

Note: Your Datasphere tenant does not need to be connected to the Content Network for this to work.

One the package is installed, you may, or may not, need to change the source of the view 2VR_CALENDER_WRAPPER, which is the basis of all the other views.
This was simply done to make sure that even if your space does not contain the Time Objects, you only have to remap 1 object.

Go to this GitHub Repo and download the package
Go to your Datasphere tenant and select "Import":
Click Upload and select the package downloaded in step 1.
After the import is complete click the package:
Under "Import Options", map the space:
And click "Import".
(Certain cases): If the view "SAP.TIME.VIEW_DIMENSION_DAY" is not present in the target space open the view 2VR_CALENDER_WRAPPER and replace the source.

How were they implemented:

All the views are implemented as SQL views, and only return a single column, depending on the kind of view, rather than an interval, which some SAP BW variables might give. The reason for that is explained in the example below.

All views have an input parameter, IP_OFFSET, which takes an integer, and will offset the result by one "unit" of time.
So the CALYEAR views will offset one year at a time, the month views will offset one month at the time, and so on.

Example: Analytical Models

In Datasphere, an Analytical Model is the final (kind of, that's not important now) step in the data modelling process.
In your analytical model, you can define different kind of variables, that allows you to derive a value from a view, or a dynamic default.
Note: Deriving select options and intervals are not supported.

In the above example, the user will be prompted to fill in an interval, but will only derive the first value:

If you want an interval, you'll need "Multiple single values".

In the above example, we've provided a view that returns three values, which will act as an interval.
Note: We have also provided the value -1 for the offset.

Which means all the values, 202601, 202512, 202511, and 202510 is offset by -1:

Implemented Variables

I've created variables for the following date objects:

0CALDAY: Date (YYYYMMDD)
0CALMONTH: Cal. Year / Month (YYYYMM)
0CALQUARTER: Cal. Year / Quarter (YYYYQ)
0CALWEEK: Cal. Year / Week (YYYYWW)
0CALYEAR: Calendar Year (YYYY)

Because - well, because those were the ones that had been giving me headaches.

0CALDAY: Calendar Day

View ID	SAP BW Variable ID	Description
2VR_0DATE	0DATE	Current Calendar Day
2VR_0CEPO_CDPDCY	0CEPO_CDPDCY	Last Day
2VR_0CEIO_CDL7CDCY	0CEIO_CDL7CDCY	Last 7 Days
2VR_0CEIO_CDL14CDCY	0CEIO_CDL14CDCY	Last 14 Days
2VR_0CXIO_CDCMCY	0CXIO_CDCMCY	All days in current Month
2VR_0CXIO_CDCQCY	0CXIO_CDCQCY	All days in current quarter
2VR_0CXIM_CDFYCY	0CXIM_CDFYCY	All days in current year
2VR_0CXIM_CDFYPY	0CXIM_CDFYPY	All days in last year

0CALMONTH: Calendar Year / Month

View ID	SAP BW Variable ID	Description
2VR_0CXPO_CMPMPY	0CXPO_CMPMPY	Last Cal. Year / Month previous year
2VR_0CXPO_CMCMPY	0CXPO_CMCMPY	Current Cal. Year / Month previous year
2VR_0CXIO_CMPQPY	0CXIO_CMPQPY	Last Quarter Previous Year
2VR_0CXIO_CMPQCY	0CXIO_CMPQCY	Last Quarter
2VR_0CXIO_CML12PMCY	0CXIO_CML12PMCY	Last 12 Cal. Year / Months excluding Current
2VR_0CXIO_CML12CMCY	0CXIO_CML12CMCY	Last 12 Cal. Year / Months Including Current
2VR_0CXIO_CMCQPY	0CXIO_CMCQPY	Current Quarter Previous Year
2VR_0CXIO_CMCQCY	0CXIO_CMCQCY	Current Quarter
2VR_0CMONTH	0CMONTH	Current Cal. Year / Month
2VR_0CEPO_CMPMCY	0CEPO_CMPMCY	Last Cal. Year / Month
2VR_0CEIO_CML6CMCY	0CEIO_CML6CMCY	Last 6 Cal. Year / Months Including Current
2VR_0CEIO_CML3CMCY	0CEIO_CML3CMCY	Last 3 Cal. Year / Months Including Current

0CALQUARTER:

View ID	SAP BW Variable ID	Description
2VR_0CXPO_CQQ1CY	0CXPO_CQQ1CY	First Quarter of Current Year
2VR_0CXPO_CQCQPY	0CXPO_CQCQPY	Current Quarter of Previous Year
2VR_0CQUART	0CQUART	Current Quarter

0CALWEEK:

View ID	SAP BW Variable ID	Description
2VR_0CWEEK	0CWEEK	Current calendar week

0CALYEAR:

View ID	SAP BW Variable ID	Description
2VR_0PYEAR	0PYEAR	Last calendar year
2VR_0CYEAR	0CYEAR	Current calendar year

The consequences of implementing strictly Shift-left architectures

2026-01-25T08:00:00.027000+01:00

Over the last year or so, the dialogue around Shift Left architectures and working as an enabler for Data Products has intensified across the industry. What was once maybe a niche architectural preference in the modern data stack is now increasingly a topic for enterprises aiming to scale their data operations. For decades, most data architecture followed a "Shift Right" pattern, based mostly on table-oriented replication patterns where backend applications were not held responsible for defining and implementing extraction logic, and centralized data teams were left in charge to act as "data janitors"—cleaning, modeling, and fixing data in a downstream lakehouse.

The idea of Shift Left is an architectural evolution for modern data management and platforms that treats data as a governed, high-quality data extract on the level of a curated and well defined business entity or even a Data Product at the moment of its birth. By moving responsibility for quality and semantic definition to the "left", orienting it to real business entities and closer to the place where the data is really generated, the burden of data preparation is addressed where the context is strongest. With that, it’s easy to imagine how the downstream "cleanup" cycle can become unnecessary or significantly streamlined under this model.

Several key benefits have fuelled this recent momentum:

Faster Time-to-Insight: Data is born ready for consumption, removing the weeks or months spent in logical (re-)definition and build, as well creating ingestion and cleaning data pipelines.
Semantic Fidelity: Business logic is defined by those who understand the context of the data best—the source domain experts.
Improved Data Quality: Automated validation at the source prevents structural and logical "garbage" from entering the analytical ecosystem.

SAP has made Data Products a central pillar of its analytics and data management portfolio, particularly through SAP Business Data Fabric. This approach allows enterprises, so the promise, to leverage quickly pre-defined, semantically rich products directly from the source by using the capabilities and rich semantics of the so-called CDS-views.

1. Previously in this theatre: Extractors as the Original Data Products

While the term "Data Product" feels modern, the principle has existed for decades within the SAP ecosystem. Standard Extractors (in SAP BW) were the original proto-data products. They abstracted technical table and join complexities, managed reliable deltas, and preserved business logic (semantics) during extraction. While remaining a vendor-specific approach, it paved the way for SAP BW as the central DW solution for SAP centric data environments. Can 0MATERIAL also be seen a data product, with (most of) its extraction logic encapsulated in the 0MATERIAL* extractors?

The Shift Left movement is essentially an extension of this principle. We can now imagine, also for the “modern data stack”, that these "extractors" are moving from proprietary silos to a world where every system—SAP or otherwise—is expected to publish its data via an open, governed data Contract on business entity level.

2. The System-agnostic Catalog as a consequence

As the definition of a data products hence relies in the source system, in a Shift Left world, the Data Catalog should no longer be perceived as a component exclusive to the analytics environment. It becomes a globally valid, Cross-System Registry, centring and nurturing itself where the business logic actually lives: and this is, if consequently applied, in the ERP and backend systems.

In this model, the backend systems act as the Semantic Anchor. Definitions, business rules, and schemas are registered at the source. The Enterprise Catalog then acts as a discovery layer, allowing the analytical team to "subscribe" to these definitions rather than attempting to recreate them through guesswork.

While a Corporate Knowledge Graph can technically be built using traditional "Shift Right" methods—by retroactively mapping data in a central hub—the Shift Left approach acts as a fundamental accelerator. By mapping business entities and their relationships directly at the source, the organization moves toward a "Data Network" or Corporate Ontology. Again, SAP is going this way.

3. Effects on the Medallion Architecture

A consequent Shift Left implementation matures the traditional Medallion Architecture (Bronze -> Silver -> Gold) into a Readiness Hierarchy:

The Fading of Bronze: The raw, messy landing zone becomes obsolete. Because data is validated against a contract at the source, it bypasses the "dump" phase.
Silver as the "Data Product Entry": The Silver layer is no longer a processing stage for data engineers. It is the published interface of the backend team—a collection of high-quality, documented and understandable Data Products.
Gold as the space of synergy and joint interpretation: Gold remains the space for interpretation and combination. Here, products from different domains (e.g. Finance and sales) are joined to create holistic, cross-functional metrics and analytical applications of many types are brought into live.

4. Shift Left architectures also imply a significant shift in roles and responsibility

The most significant challenge for implementing consequently a shift-left architecture is the migration of accountability. This requires a fundamental change in team dynamics:

Specifically for SAP environments, it is important to note that SAP has effectively taken on this responsibility. As the owner and architect of the underlying backend application logic, SAP is increasingly expected to deliver corresponding data products and semantic definitions directly. This shifts the burden of initial structural definition from the customer's internal data team back to the software vendor, who must deliver accordingly to support the modern data fabric.

However, for customer-owned data producers (custom-built applications and microservices), the consequences are more demanding. These teams could no longer view data as a byproduct of their application. They must take on the role of a data provider, which implies:

Increased risk of Technical Debt: Development teams must build and maintain the infrastructure for data contracts and delta determination.
Skill Gap Challenges: Backend developers must acquire data modeling and semantic layering skills that were previously the domain of data engineers.
Operational Overhead: Data quality incidents are no longer "downstream problems" but production bugs that must be resolved by the source team.

A major strategic advantage of this shift is that Data Quality (DQ) and DQ fulfillment metrics are now located exactly where the data is born. By addressing DQ problems at the source, organizations can identify and resolve the root causes of errors in real-time, rather than applying reactive patches downstream. This proximity inherently leads to higher data quality and more accurate business reporting.

How other software vendors will act, and how quickly internal development teams can pivot to this new standard of accountability, remains to be seen.

5. A critical technical thing: Trustworthy Deltas

For Shift Left to sustain larger datasets, it relies on reliable delta determination with often cross-table dependencies in rather complex relations. able-oriented delta mechanisms introduce complexities on the receiver side when it comes to rebuilding the correct delta image of the businesss entity. Hence, the backend must provide a stream of changes on business entity level, not just a dump of data. Whether through transaction-like Log-Based CDC or the Transactional Outbox Pattern, the backend must guarantee that updates to complex, multi-table business objects are captured correctly. SAP’s multi-table CDC mechanism is right on track, even if for very large scenarios, more basic techniques could become necessary.

6. Dealing with Fragmented Sources

A common real-world problem occurs when multiple systems describe the same global entity. For example, three different regional SAP instances may each manage their own "Material" data. In a strict Shift-Left model, you cannot force a single backend to own the "Global Material" if they don't actually manage the other regions' data. Instead of defaulting to a centralized "Gold Layer" cleanup, consider these alternatives:

Regional Responsibility: Each regional system stays responsible for its own data product (the "Silver" layer). North America and Europe each publish their own high-quality Material products according to a shared contract.
Federated Identity Resolution: The challenge of unification is addressed by establishing a shared Semantic Standard across contracts. By mandating that each regional source provides a common global identifier (e.g., a GTIN or an MDM-supplied Global ID) as a mandatory field in its data contract, the technical "merger" is downgraded from a complex transformation to a simple join or union. This virtualization allows for a global view where source-specific attributes are treated as extensions of a core, harmonized entity, preserving local ownership while enabling global interpretability.
MDM Products: A dedicated Master Data Management (MDM) team publishes a "Linkage Product." Consumers then "subscribe" to both the regional Data Product and the Linkage Product to build their own global view, keeping ownership decentralized. This is a solution approach for many customers with spread out, but logical equivalent master data objects.
Analytical Extensions (ML-driven Attributes): Not all master data attributes are generated in the operational source. Advanced attributes like "Customer Lifetime Value" or "Churn Risk Clustering" are often the result of analytical processes or ML models. These should be treated as Data Product Extensions. they exist as specialized "Sidecar Products" that share the same global identifier. Consumers combine the operational "Silver" product with the analytical "Extension" product via a join. This is not contradictory to so-called “Closed loop scenarios”; rather, it provides a clean architectural separation. While the analytical insight is consumed as a product, the actual write-back of results into the ERP remains an operational task, ensuring the semantic definition is enriched without burdening the operational backend with analytical overhead.

7. Strategic Alignment: Adapting to Pre-Defined Semantics

SAP has taken the lead by pre-defining data products in its SAP Business Data Fabric. While this provides instant maturity (starting at "Silver"), it implies that SAP provides the core business semantics and definitions. For the customer, the choice is no longer about building the definition from scratch, but about accepting and adapting to these standardized SAP semantics. This shifts the organizational responsibility from "designing the truth" to "adhering to the standard" provided by the software vendor.

Conclusion

Consequential "Shift Left" is a clear organizational direction, but it is rarely a binary switch. It demands that backend teams stop viewing data as a byproduct and start treating it as a first-order deliverable. For many organizations, the trade-off remains a strategic choice: you either invest in upstream discipline and source-side engineering to achieve high-velocity scalability, or you rely on the proven, defensive strengths of a traditional Shift Right model to manage fragmented legacy landscapes.

The path forward will be usually a hybrid one, also given by constraints of each environment. While Shift Left provides the semantic foundation and quality required for AI-ready platforms in the place where data is generated, it succeeds only when the burden of stewardship is either automated by vendors like SAP or embraced as a core technical competency by internal development teams. Ultimately, the "chaos" of modern data is not solved by shifting it from one team to another, but by establishing a shared culture of accountability across the entire data value chain.

#ShiftLeft #DataArchitecture #DataProducts #DataStrategy #DataQuality #DataEngineering #Semantics #DataTeams

Building a Graph Native AI Chatbot on SAP BTP

2026-01-25T11:36:50.119000+01:00

The true intelligence and accuracy of an enterprise AI chatbot depends on one thing, Data Access. While most chatbots rely on standard API calls to fetch information, it hits a wall when dealing with complex, interconnected business logic.

The Problem: Chatbot's rely on API calls to extract data from source system. APIs are designed for transactions and not data exploration. For example, if a user asks, 'Show me the total revenue impact of C4C service tickets related to S/4HANA orders delayed by supplier issues,' .

A traditional chatbot is stuck. It would need to orchestrate a number of cross-system OData calls, join the data in memory, and calculate the results often resulting in high latency or timeouts.

The Solution: SAP HANA Graph

The better approach is by combining the LLM with the Graph Schema in SAP HANA, we can bypass these limitations entirely. In this blog, I will show you how to build a Graph-Native Agentic AI that understands your data relationships intuitively.

Instead of viewing data as API endpoints, we model it as a Property Graph. Here, the business entities (like Customers, Sales Orders, Service Tickets, Products) are represented as Vertex(Nodes) and their relations are the Edges.

Deep-Node Traversals:

The graph allows the chatbot to traverse relationships instantly. In a single query it can hop from a Customer node -->Sales Order node -->Line Item node --> Product node, collect the necessary data and send it to the LLM which is then sent to the user.

The graph engine natively supports aggregations(SUM,COUNT) directly in the database layer during node traversal

We can physically link a C4C Service Ticket node to a S/4HANA Sales Order node. The graph schema identifies only the data relation and not the system of origin.

Representation of interconnected Business Objects that form a graph.

Let's See it in Action - Demo Video

Conceptualizing a graph traversal can be difficult, so let’s look at how the chatbot handles a query involving Sales Cloud scenario:

Why I chose Property Graph?

vs. Knowledge Graphs: While Knowledge Graphs focus on semantics and inference (great for unstructured data), Property Graphs focus on structure and attributes, which fits enterprise data better.
Rich Context: Relationship between Business Objects carry properties. For example, a relationship between a Customer and a Product isn't just a relationship link, it contains transaction details like: CUSTOMER --> BOUGHT {ProductID: 10005824, Date: 20/01/2026, Quantity: 5, Price: $100} --> PRODUCT
Mathematical Aggregations: Property Graph engines are optimized to perform aggregations (Sum Revenue, Count Tickets, Avg) on the fly during data traversal.
Accuracy: We rely on strict graph schema definitions to ensure the LLM generates accurate, hallucination free queries.

Architecture:

To enable this real-time graph agent, I implemented a robust data fabric on SAP BTP.

Real-time data fabric

We utilize SAP Datasphere to act as the pipeline for data source & replication. By replicating the raw tables into SAP HANA via Datasphere, we ensure the data is real-time. Once the data is replicated to HANA, the Graph Workspace is created on top of these tables, defining the vertexes (nodes) and edges (relationships) virtually.

Handling Federated Data Scenarios

In cases where data cannot be not replicated to SAP HANA(volume constraints), the architecture adapts seamlessly. The Graph LLM retains its logic and identifies the necessary nodes(vertices) and query parameters based on the user prompt. Instead of generating a openCypher query these parameters are mapped to optimized OData calls to the source system. This ensures a streamlined, real-time federation strategy that shows the precision of a graph traversal.

Schema Injection

The graph schema(the definition of Nodes and Edges) is injected to LLM at run time. Based on the user query the LLM will first identify the nodes, data traversal path and the logic.

For the query: “Who are my top 5 customers by sales volume who also have open critical tickets?”

The LLM determines the execution plan:

Nodes Identified: BusinessPartner (Customer), SalesOrder, ServiceRequest
Traversal Paths:
- BusinessPartner --> SalesOrder (To calculate volume)
- BusinessPartner <-- ServiceRequest (To find the open tickets)
Filters Applied:
- ServiceRequest.LifeCycleStatus = 'Open'
- ServiceRequest.Priority = 'Critical'
Aggregation: Sum of SalesOrder.NetAmount grouped by Customer.
Ranking: Order by Sum Descending, Limit 5.

openCypher Query Generation

Once the LLM identifies the graph query plan, the actual openCypher query is generated programmatically using python. I did not use LLM here as the openCypher wrapped in HANA is tricky and requires precise syntax.

The openCypher query is executed directly against the SAP HANA Graph engine and the result is passed to the LLM to generate a natural language summary.

Transactional Updates (Write Scenarios)

For maintaining transaction data, the Agentic AI brings in a mandatory human-in-the-loop confirmation step before any data is written via a POST request. You can see the technical implementation steps for this human intervention logic in my earlier blog post.

User Interface & Deployment

The core Chatbot agent is deployed on SAP BTP. For the front end, I utilized SAP Build Apps to create a clean, low-code user interface. The app connects to the BTP agent and provides a seamless chat experience for the business user.

Conclusion

By moving from API loops to Graph Traversals, we unlock the true potential of Enterprise AI. We get the speed of database-level aggregation with the flexibility of natural language processing.

Reference:

Refer this blog to understand about the implementation of Agentic AI Chatbot using LangGraph - The power of AI Agents: My Journey with LangGraph

How to call AS ABAP from SAP Datasphere

2026-01-27T08:11:27.946000+01:00

Introduction

For communication between SAP systems based on AS ABAP, RFC connections are the standard method. For example, when extracting from SAP ERP, the SAP BW system first launches the extractor in the source system via a BW->ERP RFC connection. The source system then returns the data collected by the extractor via an ERP->BW RFC connection.

For SAP Datasphere (hereinafter referred to as DSP), the natural successor to this approach is the BW Bridge use. However, not all projects justify using BW Bridge in terms of system landscape, architecture, project scope, and implementation costs.

This blog proposes an approach that allows DSP to execute code in a system running AS ABAP (without using BW Bridge). We will call this code as target code. The idea is to leverage CDS functionality known such as Virtual Elements. A Virtual Element is a field in a CDS View Entity, specifically defined using annotations. This field is not retrieved from the database table level and is not calculated by formulas within the CDS View itself. Instead, accessing the CDS View triggers execution of specialized ABAP code that implements the calculation logic for this field. For BW specialists this may remember well-known virtual characteristics and key figures. On the other hand, the CDS View is an object accessible to DSPs, for example, as a Source for Data Flow. Thus, by running Data Flow Run on the DSP side, we automatically initiate execution of the code that calculates the Virtual Element values in AS ABAP. We can, in turn, include calls of any desired programs, functional modules, or class methods to this code—that is, achieve the execution of the target code.

Unfortunately, this approach has one significant drawback that complicates its implementation: Virtual elements are only function within OData services generated from CDS views. This means that we cannot use connections of type SAP S/4HANA On-Premise, SAP ABAP, or SAP BW on the DSP side. You will need to create a Generic OData connection.

Implementation plan is following:

Create a CDS View Entity in S/4HANA using a Virtual Element.
Create an ABAP class that implements the calculation of this Virtual Element. The class code will include the target code call.
Register and test the OData Service.
Review the permissions required for the technical user on whose behalf the connection will be made. These permissions should be broader than the standard permissions for connections of type SAP S/4HANA On-Premise or SAP ABAP.
Notes regarding creating a Generic OData connection in DSP.
Create a simple Data Flow in DSP using the OData connection to the created CDS View. Running this Data Flow (manually, scheduled, or as part of a Task Chain) will trigger our target code in S/4HANA.

CDS View Entity with Virtual Element

We will create a dummy CDS View. It doesn't need to provide any meaningful data. We only need the ability to add a Virtual Element field to it. Therefore, as an example, we'll use a selection from the TKA01 (Controlling Areas) table. Typically, a single Controlling Area with a value of 1000 is used. Based on this assumption, we'll create the CDS View. Of course, you can use other selections that are guaranteed to return a single record in your system. The following code is proposed:

@EndUserText.label: 'CDS Virtual Element Test'
@analytics.dataCategory: #DIMENSION
@AccessControl.authorizationCheck: #NOT_REQUIRED
@vdm.viewType: #BASIC
@Metadata.ignorePropagatedAnnotations:true
@ObjectModel.usageType.serviceQuality: #A
@ObjectModel.usageType.sizeCategory: #S
@ObjectModel.usageType.dataClass: #MASTER
@analytics.dataExtraction.enabled: true
@analytics.dataExtraction.delta.changeDataCapture.automatic: true
@ObjectModel.supportedCapabilities:[#EXTRACTION_DATA_SOURCE]
@AbapCatalog.viewEnhancementCategory: [#NONE]
@odata.publish: true

define view entity Z_CDS_VIRT_ELEM as select from tka01
{
key kokrs,
substring( cast( tstmp_current_utctimestamp() as abap.char(17) ), 1, 8 ) as dat,
@ObjectModel.readOnly: true
@ObjectModel.virtualElement: true
@ObjectModel.virtualElementCalculatedBy: 'ABAP:ZCL_CDS_VIRT_ELEM_CALC'
cast('' as abap.sstring( 1 )) as flag
}
where kokrs = '1000'

The annotation at line 13 enables access to the CDS View via the OData Service.
The ‘dat’ field is added as a simple log: to see the date of the last calculation run.
The ‘flag’ field of the Virtual Element type is defined at line 22. It has the simplest form of a flag field. By default, as can be seen from the code, this field is empty. However, upon completion of calculations on the AS ABAP side, it will be filled with the value 'X'. This flag confirms at DSP side the successful completion of ABAP calculations.
The annotations at lines 19-21 define the "flag" field as a Virtual Element. In particular, the @ObjectModel.virtualElementCalculatedBy annotation specifies the name of the ABAP class that should calculate the ‘flag’ field.

When you initially activate this code, you'll receive, in particular, two warnings. For line 13 the system reports that the corresponding OData service is not active. For line 21 the system reports that the corresponding class doesn't exist. In the next steps, we'll resolve these issues.

If we use the context menu command 'Open With -> Data Preview' on our CDS View at this point, the result will be as follows:Here the 'kokrs' and 'dat' fields are populated using standard CDS syntax. The virtual 'flag' field is left blank.

Creation of ABAP Class implementing our calculations

A Virtual Element calculation implementing class must use two standard interfaces: IF_SADL_EXIT и IF_SADL_EXIT_CALC_ELEMENT_READ:

class ZCL_CDS_VIRT_ELEM_CALC definition
public
final
create public .

public section.

interfaces IF_SADL_EXIT .
interfaces IF_SADL_EXIT_CALC_ELEMENT_READ .
protected section.
private section.
ENDCLASS.

Second of these interfaces has two standard methods. A detailed description of these methods and their parameters can be found in the official documentation: https://help.sap.com/docs/ABAP_PLATFORM_BW4HANA/cc0c305d2fab47bd808adcad3ca7ee9d/4b5e56ec6f28453f81a...

In our example, the following code is proposed for the GET_CALCULATION_INFO method:

METHOD if_sadl_exit_calc_element_read~get_calculation_info.
IF iv_entity = 'Z_CDS_VIRT_ELEM'.
"Here we'll list which fields of CDS View will be used in virtual element calculation
IF line_exists( it_requested_calc_elements[ table_line = 'FLAG' ] ).
APPEND 'KOKRS' TO et_requested_orig_elements.
ENDIF.
ELSE.
" Potential errors should be processed by public constants,
" described in exceptional class, inherited from CX_SADL_EXIT.
" For example, if CDS View is unknown:

RAISE EXCEPTION TYPE zcx_ds_virt_elem_exceptions EXPORTING textid = zcx_ds_virt_elem_exceptions=>c_entity_not_known.
ENDIF.
ENDMETHOD.

Errors occurring during execution of the class described above must be handled using system messages. For this purpose, public constants will be created in a separate exception class. The class must be a child of the standard CX_SADL_EXIT class. An example of such an exception class is provided in the official documentation: https://help.sap.com/docs/SAP_NETWEAVER_AS_ABAP_751_IP/cc0c305d2fab47bd808adcad3ca7ee9d/8cf451799476...

As we can see, after specifying the parent class, a small constructor override is required. Each error is then declared as a public constant attribute. Naturally, such attribute contains a system message, previously defined in the Message Class relevant to the task.

Since our CDS View is designed to be as simple as possible, the IT_REQUESTED_CALC_ELEMENTS table will contain a single row with the ‘FLAG’ field. The ET_REQUESTED_ORIG_ELEMENTS table of regular source fields also contains only one record with the ‘KOKRS’ field.

Let's move on to the CALCULATE method:

METHOD if_sadl_exit_calc_element_read~calculate.
CHECK NOT it_original_data IS INITIAL.
DATA lit_calculated_data TYPE STANDARD TABLE OF z_cds_virt_elem WITH DEFAULT KEY.
MOVE-CORRESPONDING it_original_data TO lit_calculated_data.

LOOP AT lit_calculated_data ASSIGNING FIELD-SYMBOL(<lfs_calculated_data>).
"Here we need to add a call to the code that we wanted to initiate from the DSP side.
"For example, we can submit some program:
" SUBMIT z_some_program AND RETURN.
<lfs_calculated_data>-flag = 'X'. "Formal flag of the calculation readiness
EXIT. "Additional guarantee that calculation will be performed only once
ENDLOOP.
MOVE-CORRESPONDING lit_calculated_data TO ct_calculated_data.
ENDMETHOD.

Our CDS View is intentionally designed to return exactly one record. This means that in the CALCULATE method, the IT_ORIGINAL_DATA table also contains exactly one record. Consequently, the LOOP/ENDLOOP loop will only be executed once, and the target code will also be called only once.

After the ZCL_CDS_VIRT_ELEM_CALC class is activated, the warning on line 21 of our CDS View code will disappear.

OData Service activation

Our CDS View annotation specifies that it can be accessed via an OData Service. When we activate the CDS code, the corresponding OData Service will be created but not activated. Activation must be performed manually in the /IWFND/MAINT_SERVICE transaction. The OData Service name consists of the name of our CDS View followed by the _CDS suffix. In our example, the resulting name is Z_CDS_VIRT_ELEM_CDS.

Click the 'Add Service' button:

An ‘Add Selected Services’ window will appear. Enter Z_CDS_VIRT_ELEM_CDS as the technical name, and select LOCAL from the list in the 'System Alias' field. Then click 'Get Services'. A line with our service should appear:

Select the row with service description and press the ‘Add Selected Services’ button:In the 'Package Assignment' field, specify the appropriate package for subsequent import of developed objects to the production system. In the 'ICF Node' section, select the 'SAP Gateway OData V2' setting. An important setting is the 'Enable OAuth for Service' checkbox. Check with your BASIS Team to determine whether OAuth will be used when connecting the DSP to the ABAP system. If so, enable this checkbox. After confirming the settings, service activation will begin. Upon completion, the following message will appear:

Now you need to return to the main transaction window and filter the newly activated service in the list:

Check the traffic light in the 'Status' field (the leftmost field in the ICF Nodes section). If it's yellow, select the drop-down list from the 'ICF Node' button directly above it and execute the 'Activate' command:

The window for selecting a package for importing activated objects into production will appear again:

After this, the traffic light will change to green. You can now test the service by clicking the 'SAP Gateway Client' button:

By default, the test transaction specifies a URI that reads the service description:

But we're interested in what entities this service can provide. This can be found by clicking the 'Entity Set' button:

Here we see that the service can read data for CDS View Z_CDS_VIRT_ELEM. By double-clicking this name, we can change the URI:

If you now click the 'Execute' button, data will be read from the CDS View Z_CDS_VIRT_ELEM via the OData Service. This is precisely the operation that should trigger the calculation of the 'flag' field via the ABAP class described above. If you need to study or debug this class, you can set breakpoints there before clicking 'Execute'. The calculation results will be presented here in XML format:Here you can see that the 'flag' field is filled with the value 'X'. This means that the CALCULATE method was executed along with the target code call.

System user authorizations

When creating a DSP connection to an ABAP system, a special user of the System type is used. Typically, the BASIS Team assigns this user a standard set of permissions sufficient for connections of SAP S/4HANA On-Premise, SAP ABAP, or SAP BW types. Experience has shown that a Generic OData connection requires a number of additional permissions.

First, you need to add authorizations to launch the created OData Service. This is the S_SERVICE authorization object. Editing it differs from most standard authorization objects. After adding the object to the role profile, click the edit button for the SRV_NAME field. In the window that appears, select 'TADIR Service' from the Type drop-down list, which will display a list of services allowed to launch:

We need to add two types of objects for our service: IWSG and IWSV. First, select the appropriate type from the list in the 'Object Type' column. Then, use the search (for example, by pressing F4) in the 'Object Name' column. In both cases, you can use the name of our service (i.e. - Z_CDS_VIRT_ELEM_CDS) to search for the desired value:The name of the object being searched consists of the service name and the ordinal number of the active version of this service (four digits). For the first version of service in our example, the IWSG type value is 'Z_CDS_VIRT_ELEM_CDS_0001', and the IWSV type value is 'Z_CDS_VIRT_ELEM_CDS 0001'.

After saving the selected values, you will see that the value 'HT' is automatically added to the SRV_TYPE field. Thus, the S_SERVICE authorizations have been added and the values specified.

It is also important to note that specific authorizations may be required, which depend on the specific target code you want to run in the CALCULATE method described above. For example, if you plan to use a command like 'SUBMIT z_some_program AND RETURN' to launch a program, you will need the corresponding authorizations in the S_PROGRAM object. If the target code performs, for example, calculations related to the FI module, the system user may require authorizations specific to that module. If the target code accesses the contents of database tables, authorizations in the S_TABU_DIS and S_TABU_NAM objects may be required. And so on. Most likely, during the OData service testing phase (as described in the previous section), you will need to collaborate with the BASIS Team to monitor missing authorizations and add them to the system user's roles. The simplest solution is to temporarily (and only in the development system!) change the user type from system to dialog. In this case, you can log in to the SAP GUI with this user's login and manually run OData service testing while simultaneously monitoring the authorizations being checked.

The security issue of assigning such non-typical authorizations to a system user is mitigated by the fact that this user is created only to connect DSP to the ABAP system and cannot be used for manual data access, like a dialog user.

DSP OData Connection

Creating a Generic OData connection should be performed by BASIS Team specialists, as the required settings depend on the system landscape used in your specific project, as well as the security settings applied (for example, whether OAuth is enabled). Here's an example of the settings used in our project:The address entered in the ‘URL’ field ends with the technical name of our OData service. Therefore, a separate connection will need to be created for each OData service.

In the ‘Version’ field, V2 is selected. This is because we specified OData V2 when activating the service above and didn't have the option to select V4:

The example also shows that OAuth is used. That's why the corresponding checkbox was enabled earlier when activating the service:

At the bottom of the connection settings window, click the ‘Show Advanced Properties’ button:

Here you need to specify for the OData service the client number of the ABAP system we are connecting to:

DSP Data Flow to use OData connection and trigger the target code

We need to create a Data Flow with our OData service as a source. To do this, open Data Builder (in the same Space where you created OData service connection). In the left pane, select the 'Sources' tab and open the 'Connections' folder:

If all connection settings and system user authorizations were configured correctly, you will see the connection, and when you expand it, you will see the name of our CDS View:

The icon next to the CDS View is the same as for a regular table. You can drag and drop this View into the Data Flow being created. It will be automatically recognized as a Source. At this step, you can select the 'Preview Data' command for this table:

The result should look like this:

The presence of an 'X' in the 'flag' field means that the target code has been executed on the ABAP system side. If the target code execution takes a long time, you will also notice the delay when viewing it in the DSP.

As a final step, you need to complete the Data Flow by adding the target table to it, which in this case only serves as a dummy. Save the Data Flow and deploy it.

Result

Now you can initiate execution of the target code in the ABAP system in any way convenient for you: by manually starting the Data Flow, scheduling it, or including the Data Flow in a Task Chain.

Note: Data Preview is not working for Virtual Element

As a result of our implementation, OData Service is activated and implementing ABAP class exist for our Virtual Element. And we can repeat attempt of CDS data preview in ABAP Development Tools (CDS editor context menu command 'Open With -> Data Preview'). Unfortunately, result will be the same: field ‘flag’ is empty:

This is explained by the fact that the Virtual Element calculation is performed only when accessing the CDS through OData Service.

BW-Like Authorizations Multiple-Dim DAC vs Single-Dim DACs

2026-02-02T22:45:26.816000+01:00

One Multiple-Dimensions DAC or Multiple Single-Dimension DACs
Basically
If the Restriction is equal, then the DAC considers the Criterion entries as an "AND"
If the Restriction is not-equal, then the DAC considers the Criterion entries as "OR"

Use Case:
We have multiple RSIOBJNM-like f.e.: 0PLANT, 0COMP_CODE and we would like to implement RSINFOCUBE-like access as well.

One Multiple Dimensions DAC:
In DSP it is not mandatory to have all Criterions used in DAC populated. Enough is one Restriction/Criterion row.

Example:
User has maintained access to 0PLANT, but 0COMP_CODE is missing.
DSP: User will see maintained PLANTS and ALL COMP_CODEs
BW: User will get "No authorizations" message

DSP does not check all Restriction columns used in DAC, like BW does. If a User has access to one Dimension, user has access to * for all other Dimensions used in the DAC.

If the source for the authorization view/table is BW (loaded regularly from BW system into a secure DSP space), then we always should have both dimensions correctly populated. But in case not, we bypass security.

Multiple Single-Dimension DACs:
Better approach is to use multiple DACs --> implicit "AND", with the mandatory requirement to have at least one Restriction/Criterion row, all DACs must have the user populated, else nothing will be shown.
Example:
User has maintained access to 0PLANT, but 0COMP_CODE is missing.
DSP: User will get "No authorizations" message
BW: User will get "No authorizations" message

We implicitly ensure, that the user has to be authorizes for all in DAC used Dimension. BW-Like behavior.

How-to simulate Multiple RSIOBJNM (compounded IOs as well):
Create Multiple (Single Dimension) DACs for each RSIOBJNM
Implicit "AND" ensures, that security will always fail in case the user has no values assigned to DAC relevant Dimension.

How-to simulate RSINFOCUBE with RSIOBJNM:
Create Multiple (Single Dimension) DACs for each RSIOBJNM and RSINFOCUBE
For the RSINFOCUBE: create a new dimension in each DAC relevant View, simulating 0INFOPROV. Assign PROVIDER_DAC to the new dimension.
How-to simulate RSINFOAREA:
Visibility cannot be adjusted withing a Space, but if RSINFOCUBE-like access is implemented, the users will not be able to access the data anyway.

Possible use-cases for Multiple-Dimensions DAC :
MasterData based DACs: Based on single Master Data objects, for example: all of it's attributes can be moved into single DAC. Easier maintenance/support. User is not expected to be for each MD-Attribute populated.

Touples Dimensions based DACs: two different Dimensions (same master data/different view of the data) from the same source field

BTW: Nice new feature in DSP(RSECADMIN Execute-As): Now possible, however only in Views (not in Models / AMs yet).

Initial BW-Sec to DSP-Sec Blog: https://community.sap.com/t5/technology-blog-posts-by-members/bw-like-authorizations-in-datasphere-dsp/ba-p/14153918

Main Blog: https://community.sap.com/t5/technology-blog-posts-by-members/bw-vs-datasphere-dsp-amp-sac/ba-p/1428...

A.I Assisted Transition at your own Pace: BW7.5 to BDC Pilot Experiences

2026-02-05T12:03:31.021000+01:00

This blog is part of a series related to a complex BW7.5 customer case where we explored together the opportunity to transition to BDC. You can find the intro blog below, plus the links to the subsequent blogs will be listed here -

Link 1
Link 2
Link 3

The Time is Now

At the time of typing this it is still early 2026. The year’s priorities and goals are being set. For many of our BW customers, the end of maintenance deadline set at the end of 2027 for NW 7.5 is looming large. Questions such as the below are being asked –

How can we maintain what we have built over many years without rebuilding from the ground up?
How can we make use of the latest and greatest from A.I?
Can we continue to use our existing skills?
Can we remain compliant to our regulatory requirements?

The goal of this series of blogs is to outline what customers on BW 7.5 can do to modernise their mission critical system of engagement at their own pace whilst embracing A.I, ensuring they avoid expensive regret cost migrations to third party platforms that can often spiral, seldom delivering on the initial promise. These blogs are based on first hand experiences in a pilot project for BW7.5 to BDC.

Interested to learn more? Read on

The Problem Statement

BW 7.5 is the last version of the ABAP stack SAP Business Warehouse that was initially released in the late 1990’s, it delivers out of the box capabilities (business content, extractor logic, authorisation concepts etc) to manage complex third normal form data structures from SAP ECC (legacy SAP application around finance, HR, supply chain etc) across a layered scalable architecture (LSA). In 2016, the ‘new’ BW was released called BW/4HANA with a roadmap to 2040+, triggering a timeline that will culminate in 2027 with the end of mainstream maintenace of the ABAP version BW7.5.

However, many customers for a variety of reasons chose not to move to BW4. Either they have too much logic built in objects not easily ported (for example APDs, BEx) or they were enticed by promises made by best of breed data vendors in cloud native data lake architectures to migrate their BW workload. Often these project spiral in costs and effort without delivering sufficient value.

The Solution

SAP Business Data Cloud (BDC), released in February 2025 is SAP’s fully managed SaaS solution built for modern workloads and to natively handle data-as-a-service from across the SAP Business Suite applications (S/4HANA Public and Private, SuccessFactors, Concur etc). This is achieved at the click of a button whereby ‘Data Products’ are loaded and kept in sync with backend transactional systems, removing the leg work for customers looking to get rapid value from their data. Business Content and Intelligent applications are installed via the BDC Cockpit, surfacing KPIs for decision making across a host of business areas, with a vast amount more to come in 2026. With BDC, SAP Databricks, SAP Snowflake and other best of breed solutions work in tandem with SAP Datasphere on the semantically rich data products providing customers a unified data architecture to deliver next generation A.I use cases.

What adds to the excitement of BW customers setting their new year goals? BDC is the go-to solution for SAP Business Warehouse. BDC provides a strong roadmap for customers to modernise their BW7.5 environment at their own pace, retaining their investments, their skills, keeping their data available and opening their rich sets of historic business data for A.I. This is achieved through the ‘Lift, Shift & Innovate’ approach that allows for large complex BWs to move to a managed cloud service (Private Cloud Edition) of BW within the formation of SAP Business Data Cloud.

The Lift phase moves existing SAP BW systems as-is to SAP-managed cloud infrastructure, extending maintenance timelines (2030 instead of 2027) and reducing operational overhead. The Shift phase exposes SAP BW data as Data Products using the Data Product Generator tool, enabling integration with SAP Datasphere and external data sources while preserving existing use cases. The Innovate phase explores SAP-managed Data Products and Intelligent Applications to replace legacy SAP BW scenarios with modern analytics, AI capabilities, and access to an open partner ecosystem. This phased approach mitigates risk by maintaining existing investments while providing a clear path to cloud-native analytics and reduced Total Cost of Ownership.

There are some prerequisites for your BW system to fulfil prior to embarking on the journey with BDC, the minimum version supporting the data product generator is SPS 24 and it needs to be running on HANA.

Many large global organisations reliant on BW for their strategic reporting, planning/business steering have already begun the transition to BDC to realise the benefits of modernisation to cloud native at their own pace.

‘Vibe Migration’

In Autumn/Winter 2025 the SAP EMEA and Global Solution Advisory teams worked cohesively on a pilot to demonstrate the BW to BDC move for a specific customer case.

This case assessed all aspects of a complex BW7.5 system after the Lift to PCE, demonstrating how the Shift and Innovate phases can look using the modern tools available in SAP BDC with a focus on SAP Datasphere, SAP Analytics Cloud and Artificial Intelligence. Throughout the subsequent blog posts we will share the findings with specifics on how end of life (EoL) objects such as APDs, BEx and other scenarios can be covered either through automated migration to SAP BDC Datasphere, A.I assisted conversion or rebuild.

Please stay tuned for the next in this series of blog posts related to the Data Product Generator.

SAP Business Data Cloud and Datasphere News in January

2026-02-05T23:27:35.276000+01:00

SAP Business Data Cloud and Datasphere News in January

This year we really started with a bang!
You will see many great features and enhancements being delivered with our three releases already in the first month of the year.

Learn more in my short top features video in January 2026 on YouTube 📺

In addition, explore the latest updates in our community news blogs and more.

My top features in January

2026 really starts with a firework of excellent new features. Therefore, this list is a lot longer than usual.

Task Chains & Scheduling

Task ports of tasks in Task Chains

You can now use task ports to provide enhanced flexibility and error handling in task chains. Task ports allow you to control the flow of task based on success or failure outcomes. Additionally, the new Ignore Error feature allows you to disregard the status of a specific task from the overall task chain status evaluation. This feature is beneficial for maintaining workflow continuity, managing errors effectively, and customizing task execution paths.
🔗 More information

Enable Auto-Retry for a Task in a Task Chain

The auto-retry feature allows up to three automatic retries for a task in a task chain. This feature will automatically retry failed tasks to help complete a task in a task chain and improve resilience.
🔗 More information

Technical user for scheduling

You can now assign a technical user to schedule data integration tasks. This allows you to minimize the number of users that manage schedules. In addition, this avoids failed schedules and tasks related to expired consent and authorization.
🔗 More information: Modify the Owner of a Schedule and Scheduling Data Integration Tasks.

Manage Tasks with REST APIs

You can use REST APIs to run task chains and retrieve task logs and existing task log history without using the Command Line Interface (CLI). In addition, you can now also use a new CLI command to get flexible and controlled log information.
🔗 More information Manage Tasks Using REST APIs and Manage Tasks and Task Chains via the Command Line.

Analytic Model: New functions in the Expression Editor (YTD, QTD, MTD)

For calculated and restricted measures, as well as for calculated and restricted structures, the new standard functions YTD, QTD, MTD and %GrandTotal are now supported in the expression editor for these measures in the Analytic Model.

Replication Flow

Reuse a target table of one replication as a source of another replication flow

You can reuse a target local table (with or without delta capture enabled) of an existing replication flow as a source table in another replication flow. It will automatically create a relationship between the two replication flows and allows now multi-step replication scenarios. This simplifies the distribution of data from SAP Datasphere to multiple downstream systems.
🔗 More information.

Support parquet format for cloud storage sources

You can now replicate data stored in parquet files when using the following cloud storage providers as the source of your replication flow:

Amazon Simple Storage Service
Google Cloud Storage
Microsoft Azure Data Lake Gen2

🔗 More information.

Partitioning of local tables with data

You can create partitions on existing local tables that contains existing data stored in a standard space (HANA Cloud storage and compute). That way you can break down your data into smaller tables and better manage tables with a large volume of data.

At deployment you can decide if the partitioning task happens

Automatically After Deployment: Once the deployment is complete, the partition creation task will start automatically.
Automatically at a Scheduled Time: You define a timestamp when the partitions must be created. Once deployment is complete, the partition creation task will start automatically at the scheduled timestamp.
Manually at Later Time: Once the deployment is complete, you will be able to create the partitions when you want.

🔗 More information.

Longer technical names of data builder objects

We increased maximum length for technical names for data builder objects and elements that now allow for more descriptive and detailed naming conventions.
The maximum length has changed for the following objects and elements:

Views, tables, E/R models: 100
Views with semantic usage Analytical Dataset (Deprecated): 60
Analytic Models: 60
Hierarchies created inside a dimension view: 20
Columns: 100
Input parameters: 100

🔗 More information.

BW Modernization

We also delivered major milestones for the BW modernization scenarios. Metadata extraction of SAP BW 7.5 is now enabled for the catalog, plus also a data lineage from SAC story down to the BW 7.5 objects.

Extension of the data product generator with

Introduction of Scenario Generator: semantical import of BW 7.5 and BW/4HANA InfoProvider and its dependencies including attributes and texts
Multiple target support: select any Datasphere tenant and space
Transport of BW subscriptions
Mass handling of BW InfoProviders

Requires BWMT 1.27 and SAP Note 3692414

🔗 More information Connecting to BW Systems and Catalog Asset Details.

📺 Check out the Top Features Playlist: https://www.youtube.com/playlist?list=PL3ZRUb1AKkpS8bwmAREXo78O2xccXzoiy

Webinar Series: The Next Era of Business Data

SAP Business Data Cloud is part of a comprehensive strategy for enterprise data designed to address complex enterprise data management challenges. We provide an overview of SAP Business Data Cloud and guide you through its integral pillars and strategic vision.
👉 Sign up now: SAP BDC: The Next Era of Business Data

🔹Jan 29 | Discover SAP Business Data Cloud in 2026 - recording

🔹Feb 05 | The Smart Path to SAP BW Modernization - recording

🔹Feb 19 | Empowering SAP Datasphere Users with SAP Business Data Cloud Innovations

🔹Feb 26 | From Data to Decisions: Analytics with SAP Analytics Cloud and SAP BDC

🔹Mar 12 | From Data Silos to Business Insight: The Power of BDC Connect

🔹Mar 19 | An Architect's Blueprint for SAP Business Data Cloud as the Foundation for Enterprise AI

🔹Mar 26 | Design. Govern. Deliver. Data Products - Data Product Studio

🔹Apr 16 | Breaking Down the SAP BDC Pricing Model

My favorite blogs in January

What a great blog about seamless planning by @Max_Gander and @marc_daniau.
The blog shows how to run SAP HANA APL time‑series forecasts in SAP Datasphere and feed the results into a seamless SAP Analytics Cloud (SAC) planning model as a live forecast version:
🔗 Generating and Integrating Automated Predictive Library (APL) Forecasts in a Seamless Planning Model

@Phil_from_Madrid wrote a nice piece about implementing a "Shift-Left" architecture within the SAP business data fabric shifts responsibility for data quality to the source, aiming for data to originate as a governed, high-quality asset. While this approach accelerates time-to-insight through the creation of reusable data products, it necessitates development teams taking on new data-modeling tasks and managing increased technical debt upstream.
🔗 The consequences of implementing strictly Shift-left architectures

This blog from @vitvas details a method to execute ABAP code directly from an SAP Datasphere Data Flow without relying on the BW Bridge. This approach uses virtual elements in CDS views and a generic OData connection to trigger complex logic or external processes in the source system.
🔗 How to call AS ABAP from SAP Datasphere

More blogs about SAP BDC and Datasphere to check out 👇

Find more information and related blog posts on the topic pages for SAP Business Data Cloud and SAP Datasphere. Also check out the on SAP Help for SAP Business Data Cloud and SAP Datasphere, including for troubleshooting and analysis guides, onboarding guides, and more.

Currency Conversion in SAP Datasphere Using Graphical Views

2026-02-09T07:20:44.359000+01:00

Introduction: Currency Conversion in SAP

In SAP landscapes such as S/4HANA, SAP Datasphere, and SAP Analytics Cloud (SAC), handling multiple currencies is a fundamental requirement due to global operations and diverse organizational structures. Business transactions are usually posted in the local currency of the country where the transaction takes place, but reporting and analysis often require additional currency perspectives

SAP systems can support multiple organizational and reporting currencies, depending on system configuration and business requirements, such as:

• Company Code Currency - The legal reporting currency used for statutory and financial reporting.

• Group (Global) Currency - Used for consolidation and group-level reporting across multiple company codes.

• Profit Center Currency - Enables profitability analysis when a dedicated profit center currency is configured.

• Cost Center Currency - Supports internal cost tracking and management accounting scenarios.

• Transaction Currency - The currency in which the original business transaction is posted.

• Controlling Area Currency - Used for management accounting and controlling across company codes.

• Document Currency - The currency stored at the accounting document level for audit and traceability.

• Functional Currency - Represents the primary currency of business operations, commonly used in management reporting.

To ensure accurate and comparable reporting across entities and regions, these currencies must be converted using well-defined exchange rates.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Currency conversion in SAP Datasphere plays a key role by:

Centralizing currency conversion logic
Enabling consistent currency handling across models
Providing harmonized financial data for analytics and planning in SAC

This makes SAP Datasphere the foundation for reliable multi-currency reporting in modern SAP analytics scenarios.

In SAP Datasphere, currency conversion can be implemented in Graphical Views, Analytical Models, and SQL Views, and can also be applied dynamically at runtime in SAP Analytics Cloud.

In the following post, I will be covering the currency conversion in graphical view with BW as a source system.

Prerequisite:

Based on the source system (BW, S4HANA etc) the ABAP connection should be configured from source to Datasphere system.

Step 1: Import TCUR* tables, views, data flows and remote tables into Datasphere Tenant

Go to Data Builder

Select '+' symbol

Select 'Currency Conversion Views'

Note: In the Source drop down list, based on the type of source different artifacts will be created in Datasphere.

Manual - Local tables and views

SAP S/4 HANA Cloud / On Premise - Local tables, views and data flows will be imported
SAP ABAP connection - Local tables, views, data flows and remote tables will be imported.

Select Connection as ABAP

By selecting ABAP as a source - Local tables, views, data flows and remote tables will be imported.

Click on 'Create' - All the objects will be imported in to Datasphere tenant, localize the objects in a folder to use globally for different spaces.

Note: Naming convention as follows by default, option available to change Business Name

Remote tables - SAP.CURRENCY.RTABLE.TCUR*

Data Flows - SAP.CURRENCY.DATAFLOW.TCUR*

Tables - SAP.CURRENCY.TABLE.TCUR*

Views - SAP.CURRENCY.VIEW.TCUR*

Once you’ve created the currency conversion views and their supporting objects, you can share the TCUR* tables with other spaces. The spaces to which you share these tables will then see your space as an alternative source in the Source field above.

Step 2: Make TCUR* data available from BW to SAP Datasphere

Since the connection type is ABAP with BW, the existing source tables in the BW system must be mapped to their corresponding remote tables in SAP Datasphere.

Select the Remote table which are imported automatically in the system.

Example: For TCURR configuration, the same steps must be repeated for all imported TCUR* tables.

Go to 'Data Builder'

Search 'SAP.CURRENCY.RTABLE.TCURR' table

Open the table and set up the 'Remote' connection details (BW_ABAP : configured ABAP connection from BW to Datasphere)

Note: There are different methods to replicate data into Remote table:

Start Data Replication: Fetches data from the source BW ABAP table (TCURR) into the corresponding remote table in SAP Datasphere and stages it. This step can be included in a task chain. (STAGED)

Enable Real-Time Replication: Reads data virtually from BW into SAP Datasphere on demand, without staging. (VIRTUAL)

Once this step is completed, go to 'Data Builder' -> search for data flow 'SAP.CURRENCY.DATAFLOW.TCURR' -> Select 'Run'

Note: The data flow can be scheduled to run at regular intervals or included in a task chain for daily execution.

Once the data flow is completed (which can be monitored using the icon below), the data can be viewed in the currency view SAP.CURRENCY.VIEW.TCURR, a default SQL-based view provided by SAP

SAP.CURRENCY.VIEW.TCURR (provided by SAP)

Step3: Currency conversion in Datasphere views.

Go to 'Data Builder' (select space if applicable)

Select 'New Graphical View' or select if any existing view available which requires currency conversion.

There are three columns:

'Sale_Amount_CompCode' the measure contains amount to be converted

'Company_Code_Currency' holds the source currency

'Profit_Center_Currency' holds the target currency

Go to View -> Select the node where currency conversion need to apply -> Add 'fx' step

Select 'fx' step -> Details -> select '+' -> select 'Currency Conversion Column'

Set the Element Properties

Provide Business Name, Technical Name, Data type and Decimal changes

Currency Properties

Source Amount Column - Represents the column which requires conversion in this case 'Sale_Amount_CompCode''

Source Currency - Currency works as a source value in this case 'Company_Code_Currency'

Target currency - Currency works as a target value in this case 'Profit_Center_Currency'

Reference Date - This is Year/Date to be used as a reference during conversion.

Steps: Different options available and one of the key element

Shift	The amount is multiplied to remove decimals. Example 123.45 USD → 12345
Convert	The shifted amount is converted using the exchange rate. Example 12345 × exchange rate
Round	The converted value is rounded based on the target currency’s decimal rules.
Shift Back	The value is divided to restore the correct decimal places.

Note: For any simple currency conversion, recommended to select only 'Convert' and 'Round'

Conversion properties

Client - Client 000 is typically used for currency tables in SAP systems

Conversion Type - Exchange rate type maintained in TCURR (for example, M – Standard rate, P – Budget rate)

Error Handling - Defines the value to be generated if no valid exchange rate is found.

Advanced Properties

Lookup: Regular and Reverse (to be selected if reverse exchange rate is maintained in the system)

Accuracy: Compatibility (Matches classic SAP (ECC/BW) currency results) and Highest (Provides maximum precision for accurate financial reporting)

Date Format: Auto Detect (Automatically picks the correct date format from source data and applies the right exchange rate)

All remaining views relevant to TCUR* will get auto populated by default, as its configured at beginning.

Activate and Deploy the view

Result Output:

Exchange rate from AUD to GBP is maintained as 1.75 in BW system

Data From BW System:

Client	Exchange Type	From	To	Valid From	Exchange Rate
600	P	AUD	GBP	01.01.2025	/1.75

Values are converted using the applicable exchange rate and published in a new column called “Profit Center Amount.”

Conclusion: By centralizing currency conversion logic in SAP Datasphere, organizations can ensure consistent, auditable, and scalable multi-currency reporting across analytical and planning scenarios. This approach simplifies downstream consumption in SAP Analytics Cloud while maintaining alignment with SAP’s standard currency configuration.

Thank you for taking the time to read this article. I hope this overview helps you understand and implement currency conversion in SAP Datasphere more effectively. Your feedback, questions, and suggestions are welcome in the comments section.

Regards

Lokesh Kumar Pothapola

Triggering SAP Datasphere/BDC Task Chains from SAP BTP ABAP Cloud using the Task Chain REST API

2026-02-10T16:44:24.061000+01:00

Our modern data landscapes thrive on automation. In this post, we’ll walk end-to-end through a real-life integration scenario that tackles one of the building blocks in this context:

Starting a SAP Datasphere Task Chain remotely from SAP BTP ABAP Cloud using OAuth 2.0 Client Credentials and the Datasphere Task Chain REST API

The goal is to orchestrate Datasphere data processing flows directly from ABAP Cloud — securely, programmatically, and fully automated.

We will cover the below points:

Creating a Task Chain in Datasphere
Configuring OAuth clients in Datasphere
Setting up Communication System & Arrangement in ABAP Cloud
Executing the Task Chain remotely: Consume REST API from ABAP Cloud

1 Architecture Overview

Solution Architecture

ABAP Cloud Task Chain Execution: ABAP Cloud acts as the caller. Datasphere exposes the Task Chain API protected by OAuth.

2 Prerequisites

You must have:

SAP Datasphere tenant with developer + admin access
A Task Chain to trigger
SAP BTP ABAP Environment (ABAP Cloud) with developer access
BTP ABAP: Authorization to configure OAuth clients and Communication Arrangements

3 Creating a Task Chain in SAP Datasphere

The first step is defining a Task Chain that encapsulates the processing logic you want to execute remotely.

In the given example:

The Task Chain starts with a Begin node
Executes a Replication Flow for loading dimension data
(Can be extended with transformations, views, or further flows)

SAP Datasphere Task Chain with Replication Flow inside

Main takeaways:

Task Chain must be Deployed
Note the Technical Name (used later by API)
Verify it runs manually before you automate via the API

4 Configuring OAuth Client in SAP Datasphere

To allow ABAP Cloud to call Datasphere APIs securely, we create an OAuth Client in Datasphere Administration.

Navigation Path

Datasphere → Administration → App Integration → OAuth Clients

OAuth Endpoints Provided

Amongst others, SAP Datasphere automatically exposes:

Authorization URL
Token URL

These are required for OAuth2 token retrieval. Moreover you must create a technical user that has access to the Datasphere space and is able to run the task chains that are in scope. Typically this is done by assigning a scoped role.

Configured OAuth Client (based on Technical User)

Menu Path Datasphere

This is where you see the OAuth Client specific URLs - take note of them for later configuration in ABAP BTP:

OAuth Client Generation

Your specific OAuth client for consumption from ABAP BTP. The secret key is only shown once upon creation - keep this in mind!

OAuth Client Details

Important settings:

Parameter	Value
Grant Type	Client Credentials
Purpose	Technical User
Roles	space access must be given (e.g. by scope role in DSP)
Token Lifetime	e.g. 60 minutes

Save the Client ID and Client Secret — this is used in ABAP Cloud later on.

5 Creating Communication Artifacts in ABAP Cloud

Now we configure the outbound technical connection from ABAP Cloud to Datasphere.

First create an Outbound Service of type HTTP:

Create Outbound Service

Create a Communication Scenario with Authentication Methods = OAuth 2.0 only, bind it to your outbound service and publish it:

Communication Scenario Creation

Navigation: ABAP Environment Web Access → Communication Management → Communication Systems

ABAP Cloud Web Access

ABAP Cloud Communication Systems

The hostname is your Datasphere tenant URL. Outbound OAuth URLs (like Token/Authentication) can be found on the Datasphere side under Administration -> App Integration (previous step).

ABAP Cloud Communication Systems Details

You have to add a user "OAuth 2.0 (Basic)":

ABAP Cloud Communication Systems Users

Key fields:

Field	Example
System ID	DSP_REST
Host Name	<your-datasphere-host> (tenant URL)
Port	443
OAuth 2.0 Endpoints	From Datasphere for Token & Authorization URL (Audience URL not required)
Client ID	Datasphere OAuth Client ID
Client Secret	Datasphere Secret (you must take note of this when creating the OAuth Client)

6 Creating a Communication Arrangement in SAP ABAP Cloud

Now we bind the Communication System to a Communication Scenario that allows REST calls.

Navigation: Communication Management → Communication Arrangements

Arrangement Example: ZDSP_CS_REST

ABAP Cloud Communication Arrangements

Note the service path: The dynamic parts of the path are set from ABAP (this is to keep the space + task chain flexible and controllable from the actual caller: ABAP).

ABAP Cloud Communication Arrangements Detailed Configuration

What this provides:

Preconfigured service URL
OAuth token handling by ABAP runtime
Secure outbound API consumption

7 Calling the Datasphere Task Chain API from ABAP Cloud

Now comes the fun part: consuming the REST API. Datasphere exposes endpoints such as:

POST /api/v1/datasphere/tasks/chains/<space_id>/run/<task_chain_technical_name>
GET /api/v1/datasphere/tasks/logs/<space_id>/<log_id>

Which triggers a new execution or reads the status of a task chain run.

ABAP Cloud Implementation Pattern

In ABAP Cloud when working with REST API to integrate external services, you typically:

Create HTTP destination automatically from the created Communication Arrangement
Use CL_WEB_HTTP_CLIENT_MANAGER
Perform POST/GET requests
Parse response (run ID, status)

The code snippet performs the following parts in more detail:

Run as a class-run program (IF_OO_ADT_CLASSRUN) -> you can execute directly from ADT and print to the console.
Locate the configured Communication Arrangement (bound to our Communication System DSP_REST) using CL_COM_ARRANGEMENT_FACTORY.
Create an HTTP destination from the Communication Arrangement using CL_HTTP_DESTINATION_PROVIDER=>CREATE_BY_COMM_ARRANGEMENT.
Create an HTTP client from the destination (CL_WEB_HTTP_CLIENT_MANAGER).
POST request to the Task Chain “run” endpoint:

triggers a new run
receives a JSON response containing a LogId

WAIT a few seconds.
GET the log endpoint using the returned LogId:

retrieves the execution status (RUNNING, later SUCCESS / FAILED, etc.)

Print everything in the ABAP Console.

ABAP-Code Snippet:

CLASS zcl_dsp_rest_api DEFINITION PUBLIC FINAL CREATE PUBLIC.
   PUBLIC SECTION.
    INTERFACES if_oo_adt_classrun.

    CLASS-DATA:
      out TYPE REF TO if_oo_adt_classrun_out.
    CLASS-METHODS:
      call_dsp
        RAISING
          cx_http_dest_provider_error
          cx_web_http_client_error.
ENDCLASS.

CLASS zcl_dsp_rest_api IMPLEMENTATION.
  METHOD if_oo_adt_classrun~main.
    zcl_dsp_rest_api=>out = out.

    TRY.
        call_dsp( ).
      CATCH cx_web_http_client_error
            cx_http_dest_provider_error INTO DATA(exception).
        out->write( exception->get_text( ) ).
    ENDTRY.

  ENDMETHOD.

  METHOD call_dsp.

    TYPES: BEGIN OF ty_log,
             logId  TYPE i,
           END OF ty_log.

    DATA: ls_log TYPE ty_log.
    DATA(communication_system) = 'DSP_REST'.
    DATA(arrangement_factory) = cl_com_arrangement_factory=>create_instance( ).

    DATA(comm_arrangement_range) = VALUE if_com_arrangement_factory=>ty_query-cs_id_range(
      ( sign = 'I' option = 'EQ' low = communication_system ) ).

    arrangement_factory->query_ca(
      EXPORTING
        is_query = VALUE #( cs_id_range = comm_arrangement_range )
      IMPORTING
        et_com_arrangement = DATA(arrangements) ).

    DATA(arrangement) = arrangements[ 1 ].

    DATA(destination) = cl_http_destination_provider=>create_by_comm_arrangement(
      comm_scenario  = 'ZDSP_CS_REST'
      service_id     = 'ZDSP_REST_SRV_REST'
      comm_system_id = arrangement->get_comm_system_id( ) ).

    DATA(http_client) = cl_web_http_client_manager=>create_by_http_destination( destination ).
    DATA(request) = http_client->get_http_request( ).
    request->set_uri_path( '/chains/BDCPOC/run/TC_DIM2_A001' ).

    DATA(response) = http_client->execute( if_web_http_client=>post ).

    CALL METHOD /ui2/cl_json=>deserialize
      EXPORTING
        json = response->get_text( )
      CHANGING
        data = ls_log.

    data(log_str) = CONV string( ls_log-logId ).
    out->write( 'Task Chain started. LogId created:' && log_str ).

    WAIT UP TO 10 SECONDS.

    CONCATENATE '/logs/BDCPOC/' log_str INTO DATA(uri).

    clear: request, response, http_client.
    http_client = cl_web_http_client_manager=>create_by_http_destination( destination ).
    request = http_client->get_http_request( ).
    request->set_uri_path( uri ).
    response = http_client->execute( if_web_http_client=>get ).
    out->write( 'Task Chain Log REST API - via GET - Task Chain Status:' && response->get_text( ) ).

  ENDMETHOD.
ENDCLASS.

Once triggered:

Datasphere creates a new Task Chain run
Execution can be monitored in Datasphere UI
API returns run ID and status

Follow-up calls can fetch:

Execution status
Logs
Completion result

8 Execution Results

ABAP Cloud Console Log: Results

The Task Chain in Datasphere is in Running status:

SAP Datasphere Task Chain Running Status

9 Conclusion/Outlook & Further aspects:

Overview of the major building blocks:

Capability	Result
Secure Auth	OAuth 2.0 Client Credentials
No hardcoded secrets	via Communication Arrangements/System
Remote orchestration	Creating an end-to-end orchestrated chain of tasks
Cloud-native ABAP	Using appropriate ABAP Class APIs
Datasphere automation	Using Task Chain API

Use Cases that the implementation enables:

Nightly batch orchestration
Event-driven data loads
Cross-system workflows (this was our ultimate focus)
CI/CD data pipelines
API-driven analytical data refresh

References

SAP Datasphere Task Chain API Documentation: https://api.sap.com/api/DatasphereTasks/overview
SAP Datasphere Task Chain Help Page: https://help.sap.com/docs/SAP_DATASPHERE/c8a54ee704e94e15926551293243fd1d/274f2736465c4c48a091c675880502a2.html?locale=en-US
ABAP Cloud - how to call external APIs: https://jacekw.dev/blog/2022/oauth-client-credentials-from-abap-cloud/

These are your main takeaways to further sharpen the solution:

Use short token lifetimes
Separate technical OAuth users
Monitor task runs via API from ABAP
Handle retries & failures in ABAP
Log run IDs for observability

If you have any aspects, comments and/or concerns, please raise them and let's discuss. I'm happy and proud that this article was written by at least 90% human and only 10% AI 🙂

Automating Invoice Predictions from SAP Databricks with SAP Datasphere Task Chain

2026-02-11T13:01:18.220000+01:00

In our previous blog, we demonstrated how to process semi-structured PDF invoices, parse them and store the results in a Delta table before applying machine learning algorithms to predict payment delays. These predictions are hosted in SAP Databricks as part of a customer managed data product, with results written to a target Delta table in Unity Catalog and shared via Delta Sharing with SAP Business Data Cloud (BDC).

This is how our customer managed data product holding the first processed 1.000 invoices looks like before any updates:

The next challenge on our journey was automation. Once new invoices land in the S3 bucket, we need a fully automated process that parses them, updates predictions, and refreshes the Delta share, without any manual intervention. Since SAP consumes the data federatively, business users expect up‑to‑date predictions in near real time.

To meet this requirement, my colleague @Mostafa_Shaarawy1 and Idesigned and implemented an end‑to‑end orchestration flow using SAP Datasphere (DSP) Task Chains and Databricks API to:

trigger the Databricks notebook on schedule or on demand,
monitor its execution,
and ensure updated predictions flow instantly to SAP BDC.

To achieve this, we are going to utilize the run-now and the get run APIs described in Databricks API reference, with authentication handled via a Databricks Service Principle.

The result: a fully automated pipeline that delivers fresh, reliable insights to business users, improving decision‑making, operational efficiency, and payment‑risk visibility.

Business Use Case: Finance Manager’s Perspective

Imagine you’re a finance manager responsible for optimizing cash flow. You rely on accurate predictions of payment delays to decide which customers to follow up with and when. With Datasphere orchestration, the moment new invoices arrive, the system automatically processes them, updates predictions, and makes the latest insights available in SAP BDC, without any manual steps. This means you can act faster, reduce overdue payments, and improve working capital, all while saving time and effort.

We’ll achieve this using Datasphere Task Chains, leveraging their API Task (REST) capability and a Generic HTTP connection to seamlessly integrate with SAP Databricks.

Disclaimer: The scenario we are covering in this blog is technically working fine but is not officially supported in the SAP Databricks documentation.

Solution Overview

The architecture looks like this:

What makes this work?

The architecture consists of three components:
- DSP Task Chains orchestrate Databricks jobs using REST API calls.
- Databricks Jobs API triggers and monitors the notebook execution.
- Delta Sharing + Unity Catalog exposes the updated prediction table live to SAP BDC/DSP.

This ensures new data flows from invoice ingestion → prediction → Delta share → SAP consumption end‑to‑end, without manual steps.

Prerequisites

Before building the orchestration, we need to first complete these steps:

Create a Service Principal in SAP Databricks Workspace at the workspace level to generate:
- Client ID
- Client Secret
- OAuth Token Endpoint.
Assign the service principal permissions to run the job and access the target Delta table.
- This avoids using personal tokens and ensures secure automated API calls.
Prepare the Databricks Job
- Identify the Job ID of the notebook that:
  - Processes new invoices,
  - Runs ML predictions,
  - Writes results into the target Delta table in Unity Catalog.
Update the Delta Share and the Share CSN document

Step-by-Step Implementation

1) Set-up Databricks and create service Principal

A service principal in Databricks is a dedicated, non-human identity built for automation. It enables secure, API‑only access to Databricks resources for scripts, automated workflows, and CI/CD pipelines, all without depending on personal user credentials.

a- Go to Settings > Ìdentity and Access > Service Principals > Add service principal:

b- Generate a secret for this Service Principal and copy it as well as the Client ID (you'll need them for the definition of the Httpconnection in

SAP Datasphere😞

c- Go to the notebook under Scheduleand create a new one:

d- Switch to the Jobs & Pipelines section and select your scheduled job and copy the job ID (you'll need it in the API Invocationrequest body of the task chain in SAP Datasphere to call the right job):

2) Set-up your Http Connection in your Datasphere space

In your space, navigate to Connections in Datasphere and choose Http as connection type.

Enter the connection details.

Host   = <your_tenant_url without 'https://' >.
Port    = 443
Protocol     = HTTPS

Use Cloud Connector
    = False

Authentication Type
     = OAuth2.0

OAuth Grant Type
 = Client Credentials

OAuth Token Endpoint = <Your_token_Url_endpoint>

OAuth Scope = all-apis

OAuth Response Type: Token

OAuth Token Request Content Type: URL Encoded

Client ID: <Client_Id_of_DBX_Service_Principle>

Secret: <Secret_of_DBX_Service_Principle>

Save the connection.

3) Create Task Chain with API Task

Navigate to Data builder and create a new Task Chain.
From the top of the canvas, drag and drop API Task to the editor.

From the properties of the Task Chain, select under Connection the Http connection created in step 2.

In the API Invocation section, choose POST method and the mode to be Asynchronous.

The base url will be automatically generated from the connection details
In the API Path, write run-now (more information can be found in the Databricks API documentation https://docs.databricks.com/api/workspace/jobs/runnow).
The body should be a json object with the job-idof the Databricks job.

Choose Get result from HTTP status code and response body and set the path to run_id. This will map the property run_id in the API response to be used for status retrieval to be able to monitor the Databricks job from the task chain.

In the Status API section, we will use the get single run API for status retrieval as documented in Databricks API documentation https://docs.databricks.com/api/workspace/jobs/getrun.

Set the API path to runs/get?run_id={id}. Since we have, in the previous step, configured the path to run-id, the API task will dynamically use its value in the query parameter during run time.

In the response section choose Get result from HTTP status code and response body to determine whether the databricks job was successful or not, based on the result from the response body.
As success indicator use: status.termination_details.code equals to SUCCESS
As error indicator use: status.stateequals to TERMINATED

4) Run the API task to trigger the Databricks job

Now the API task is ready to be triggered. Once triggered the Invocation API will be called with the job-id in the request body and the status API will be called periodically to evaluate the completion of the Databricks job.

For testing purpose, you can conveniently just run the API task without having to run the task chain. Click on Test Run in the API task to run it.

You can now click on the Data Integration Monitor icon to see the run details.

When you switch to the monitoring, you can then see the run details of the job:

This is also reflected in SAP Databricks:

Once the job has completed successfully, you can see the 5 invoices newly added to the S3 bucket now updated in the custom managed data product shared from SAP Databricks to SAP Business Data Cloud:

Key Benefits for Business Users

Real-time insights: Data is refreshed immediately as new invoices are processed.
Zero manual effort: A fully automated end‑to‑end pipeline.
Improved cash flow management: Faster decisions supported by fresh insights.
Operational efficiency: No delays between data processing and business consumption.
Federated access: Data is consumed directly in SAP BDC—no physical copies or complex integrations required.

A SAP BTP Kyma Encryption/Decryption Microservice for ALL Contexts (e.g. SAP Datasphere/BDC/ABAP)

2026-02-11T16:12:01.099000+01:00

G’day!

Warning: This is only for readers who are really interested in this subject 😍 The article is quite comprehensive and has got some cross-article links inside, that also need to be considered. Therefore: A lot of reading and a lot of hands-on + thinking if you want to rebuild the thing.

Below is the structure of this article:

1 Status Quo on En-/Decryption in BTP
2 Motivation
3 Fundamentals
4 Architecture
5 HDLFS Configuration
6 Insights into Python
7 Containerization & Deployment in Kyma
8 Achievements
9 More Scope
10 References

Let's begin:

1 Status Quo on En-/Decryption in BTP

When it comes to En-/Decryption on SAP BTP, there are a couple of pitfalls to be considered. The following roughly outlines the situation, platform and application-wise:

Individual applications (e.g. SAP Successfactors) got proprietary encryption/decryption mechanisms available. They are typically not reusable.
SAP Integration Suite has got PGP En-/Decryptor nodes available. However, it is not advisable to run large volume scenarios using this approach. It has been built for data sizes on the levels represented within message handling scenarios (I‘m not an expert therein, but I guess thats typically not in the area of GBs).
A policy/rule for data to remain encrypted until its landing into BTP: You face challenges to securely onboard large volume files containing sensitive data into BTP, because there is no out-of-the-box solution to keep them encrypted in motion and decrypt them at rest once available in BTP (and vice versa for an outbound flow).

The Challenge

2 Motivation

Due to the above limitations, we were eager to find an appropriate solution approach that first of all satifies our specific scenario, but can also be augmented to other contexts. This is why the following motivation arose:

Especially En-/Decryption for larger files on BTP side seems to be a missing piece in the puzzle.
Our ambition is to encapsulate such kind of mechanism centrally as a reusable service. Multiple kinds of contexts can be served, exposing this service: Applications, data flows & transformations etc. – in our context the consumer will be SAP Datasphere.
There is another consequence of finally being able to load decrypted large data sets into BTP. To be more precise: To onboard this data into SAP Datasphere to further process and transform, the fact that Base64 encoded data must be handled in our context too, why not also expose or incorporate columnar Base64 En-/Decoding into the microservice?
The intended solution is custom-built and must be managed individually. However, the baseline that will be set can be reused and adapted to various customer needs and application contexts.

3 Fundamentals

The following represent some fundamentals and success criteria we‘ve established for our solution:

Use HANA Data Lake Files (HDLFS) as a file store for inbound/outbound persistency
Implement the service as containerized Python code logic
Service capabilities must be controllable and schedulable within Kyma
Service can handle PGP encrypted files, but can also decrypt for outbound delivery
Service can decode Base64 columns for inbound source files
Service manages and cleans up directories and file in-/output on HDLFS
Service can handle files with sizes in GB range
Deploy service into a Kyma Cluster (Cloud Foundry? -> no, Memory/Sizing limitations)
Security: Facilitate simple and secure handling of all required secrets and certificates
Service not exposed via HTTP endpoints (this is kind of a contradiction to a true microservice, however this requirement applies in our context. Under "More Scope" you will find guidelines/directions how to enable HTTP based microservices)

4 Architecture

The following architectural components represent what we‘ve picked out for our solution:

Usage of HANA Data Lake Files (HDLFS) as a file store for inbound/outbound persistency
Use BTP Kyma for service runtime (Kyma deployed in the 2nd smallest T-Shirt size)
SAP Datasphere represents the persistency layer (DSP as a consumer is exchangeable but in our case relevant for the business scenario)
The other more general blue box "other BTP Solutions" indicates, that the service implementation on Kyma isn't necessary limited to SAP Datasphere only, but should be rather considered as something agnostic. This of course heavily depends how it is built and finally implemented.

BTP Solution Architecture Kyma Microservice

5 HDLFS Configuration

I strongly recommend to read the tutorial by Jason Hinsperger to familiarize yourself with the HDLFS REST API dependencies and the essential steps to spin up the instance. You must follow all the steps described therein.

Its important to note that the generated client key and client certificate is what you are about to use from python level. Moreover you should take note of the REST API endpoint. This is the endpoint you run all your requests against (you find it in HANA Cloud Central -> Instances -> HDLFS Instance -> "Files REST API Endpoint".).

To familiarize yourself in general with the HDLFS REST API documentation, you can use this link.

HDLFS REST API Endpoint

One essential detail I don't want to hide, is the IP whitelisting of the Kyma environment. In order to do that, navigate to the start page of your cluster dashboard. There is a section called "Cluster Overview --> Metadata". Copy the (typically three) IP listed there: "NAT Gateway IP Addresses".

Go to your HANA Cloud Central and navigate to your HDLFS instance. Click "Manage Configuration" and go to the section "Connections": If you have enabled the setting to only allow specific IP addresses, likewise maintain the ones of the Kyma Cluster that you previously copied:

HDLFS Configuration Connections

6 Insights into Python

The python coding part is split into several modules, which are all described in a detailed way further below:

Inbound processing
Outbound processing
Common functions
Crypto function for encryption/decryption

Python Modules Sketch

Inbound Module (1)

The inbound part scans a given datalake path for .gpg files, then decrypts each file locally in parallel using a configurable worker count. After decryption, it samples the CSV (semicolon-delimited), detects which columns contain base64-encoded data, and runs a transformation step to produce a cleaned output file. The transformed result is uploaded to a decrypted output path, and the original encrypted input is moved to an archive folder. It logs success or errors with timing details on file level and exits with a failure code if any file fails.

This code snippet describes the parallel processing of files:

    with ThreadPoolExecutor(max_workers=MAX_PARALLEL_JOBS) as exe:
        futs = {exe.submit(processFile, f): f for f in files}
        for fut in as_completed(futs):
            f = futs[fut]
            try:
                res = fut.result()
                ok += 1
                print(f"[OK] {res['input']} -> {res['output']} sec={res['sec']:.2f} base64_cols={res['base64_cols']}")
            except Exception as ex:
                err.append((f, str(ex)))
                print(f"[ERR] {f}: {ex}")

Sampel code of processing an individual file:

def processFile(file_name: str):
    t0 = timer()
    out_name = normalize_output_name(file_name)

    with tempfile.TemporaryDirectory() as td:
        enc_path = os.path.join(td, file_name)
        dec_path = os.path.join(td, out_name + ".decrypted.csv")
        out_path = os.path.join(td, out_name)

        download_to_file(f"{PATH_IN_DATALAKE}/{file_name}", enc_path)
        decrypt_symmetric(enc_path, dec_path)

        header, sample = sample_csv(dec_path, delimiter=";")
        base64_cols = detect_base64_columns(out_name, header, sample)

        transform_csv(dec_path, out_path, base64_cols, delimiter=";")

        mkdirs(PATH_OUT_DATALAKE)
        upload_file_chunked(f"{PATH_OUT_DATALAKE}/{out_name}", out_path)

    mkdirs(PATH_ARCHIVE)
    rename(f"{PATH_IN_DATALAKE}/{file_name}", f"{PATH_ARCHIVE}/{file_name}")

    return {"input": file_name, "output": out_name, "base64_cols": base64_cols, "sec": timer() - t0}

Outbound module (2)

It scans the HDLFS outbound folder for .csv files and subdirectories containing Parquet parts, then processes them in parallel using a configurable worker count. CSV files are downloaded locally, then encrypted and uploaded back again to an encrypted outbound path. The originals are moved to an archive folder. Parquet file based directories on HDLFS (when enabled) are downloaded, consolidated into a single CSV using PyArrow, encrypted and uploaded similarly. As a last step, they are marked with an _SUCCESS file to prevent reprocessing and moved to the archive.

Crypto Module (3)

It provides helper functions for symmetric file encryption and decryption using the gpg command-line tool with an AES-256 cipher. It requires a passphrase supplied via the PASSPHRASE environment variable (from a Kyma Secret). Both encrypt_symmetric and decrypt_symmetric execute GPG in batch mode through subprocess, capturing stdout/stderr and validating the return code.

The decrypt_symmetric function is implemented as follows:

def decrypt_symmetric(enc_path: str, dec_path: str):
    _require_passphrase()
    cmd = [
        "gpg", "--batch", "--yes",
        "--pinentry-mode", "loopback",
        "--passphrase", PASSPHRASE,
        "--output", dec_path,
        "--decrypt", enc_path
    ]
    r = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    if r.returncode != 0:
        raise RuntimeError(f"GPG decrypt failed: {r.stderr.decode('utf-8', 'replace')}")

Common Functions Module (4)

All major control variables are built using environment variables (defined as secrets in Kyma) for endpoint/container, TLS certificates, verification behavior, timeouts, retries, and chunk size.

In one function it builds a reusable SSL context. API requests are wrapped with exponential-backoff retries for transient HTTP statuses (e.g., 429/5xx) and network errors. It provides HDLFS related helpers to list files/directories, check existence, create directories, rename paths, download remote content to disk in chunks, and upload files.

The below part builds the SSL context to connect to HDLFS via its REST API:

def buildSSLContext() -> ssl.SSLContext:
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
    if os.path.exists(CRT_PATH) and os.path.exists(KEY_PATH):
        ctx.load_cert_chain(certfile=CRT_PATH, keyfile=KEY_PATH)

    if TLS_VERIFY:
        if CA_CERT_PATH:
            ctx.load_verify_locations(CA_CERT_PATH)
        else:
            ctx.load_default_certs()
        ctx.verify_mode = ssl.CERT_REQUIRED
        ctx.check_hostname = True
    else:
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE

    return ctx

This part establishes a connection to the HDLFS REST API endpoint as defined in the environment variable FILES_REST_API:

def _conn():
    if not FILES_REST_API:
        raise RuntimeError("FILES_REST_API is empty (set env FILES_REST_API).")
    return http.client.HTTPSConnection(FILES_REST_API, timeout=HTTP_TIMEOUT, context=_SSL_CTX)

The below function represents the upload (HDLFS REST API PUT) of a file in streaming mode to an HDLFS directory:

def _put_to_location(location: str):
        url = urlparse(location)
        if u.scheme and u.netloc:
            host = url.netloc
            path = (url.path or "/") + (("?" + url.query) if url.query else "")
            conn = http.client.HTTPSConnection(host, timeout=HTTP_TIMEOUT, context=_SSL_CTX)
        else:
            path = location
            conn = _conn()

        try:
            conn.putrequest("PUT", path)
            conn.putheader("x-sap-filecontainer", CONTAINER)
            conn.putheader("Content-Type", "application/octet-stream")
            conn.putheader("Content-Length", str(size))
            conn.putheader("Connection", "close")
            conn.endheaders()

            with open(file_path, "rb") as f:
                while True:
                    chunk = f.read(CHUNK_SIZE)
                    if not chunk:
                        break
                    conn.send(chunk)

            r = conn.getresponse()
            data = r.read()
            status = r.status
            r.close()

            if status in _RETRYABLE_STATUSES:
                raise RetryableHttpError(f"PUT retryable: {status} {data[:200]!r}")
            if status not in (200, 201):
                raise RuntimeError(f"PUT failed {remote_path_no_leading_slash}: {status} {data[:200]!r}")
        finally:
            conn.close()

7 Containerization & Deployment in Kyma

I won't describe in detail, how the creation of a docker container is made step by step. I just provide some essentials + hints to guarantee that you succeed and don't stumble upon the same issues I had faced. The following prerequisites must be met in order to create a runnable docker container based on your Dockerfile + python code:

Local installation of docker (Docker Desktop)
Docker registry (e.g. GitHub Docker Registry) – this is from where the Pod pulls the docker image
kubectl cli
IDE such as Visual Studio code

For an overall procedure, including the part in Kyma, you can refer to this blog entry on SCN. I strongly recommend that you try to rebuild the example the author walks through first, BEFORE you tackle your actual project.

Recommendations out of my personal learnings:

Try to not create any Kyma artifacts via the UI, but rather define those within descriptor yaml files of the corresponding kinds (e.g. kind: Deployment or Service or APIRule).
!!! A minor but very important thing to consider when running the docker build command, run it as follows:

docker build . --tag ghcr.io/<github_user>/<git_repo_name>:latest --platform linux/amd64

The deployment.yaml in the blog specified above is based on three Kyma/K8S artifacts that are created: Deployment, Service + APIRule. Which kind of artifacts you actually require strongly depends on your python implementation and how your service/logic runs. In the event of a purely job schedule based execution of your processing logic, you don't need a service or APIRule artifact, of course. Instead, you'd only need a deployment of kind: CronJob. This will spin up all dependent artifacts implicitely. In a CronJob based deployment you will have to specify the following details in the yaml file: scheduling details (time, frequency, time zone), required volumes, container spec including secrets to pull image, container level: mounts & env variables, resource allocation.
A template for a CronJob based deployment can look like this:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: <job_name>
  namespace: <kyma_namespace>
spec:
  suspend: false
  schedule: "* * * * *" 
  timeZone: "Europe/Berlin"

  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        spec:
          restartPolicy: Never
          imagePullSecrets:
            - name: ghcr-pull-secret

          volumes:
            - name: gnupg-home
              emptyDir: {}

          initContainers:
            - name: init-gnupg
              image: busybox:1.36
              command: ["sh","-c","mkdir -p /tmp/gnupg && chmod 700 /tmp/gnupg"]
              volumeMounts:
                - name: gnupg-home
                  mountPath: /tmp/gnupg

              resources:
                requests:
                  cpu: "10m"
                  memory: "32Mi"
                limits:
                  cpu: "50m"
                  memory: "64Mi"

          containers:
            - name: inbound
              image: <container_registry_path>
              imagePullPolicy: Always
              command: [<custom_os_level_command_if_required>]
              volumeMounts:
                - name: gnupg-home
                  mountPath: /tmp/gnupg

              env:
                - name: KEY_PATH
                  value: "/keys/keyfile.key"
                - name: PASSPHRASE
                  valueFrom:
                    secretKeyRef:
                      name: kyma-secret
                      key: PASSPHRASE

              resources:
                requests:
                  cpu: "100m"
                  memory: "1Gi"
                limits:
                  cpu: "200m"
                  memory: "2Gi"

8 Achievements

The following functionality and service details were delivered:

The Python code was improved multiple times: Handling approx. 30 files in one go with a total of 10 GB in encrypted state could be accelerated to < 30 minutes of processing time
Cron Jobs in the Kyma cluster can be used to schedule individual services, e.g. inbound processing of files that reside in a specific folder on HDLFS
SAP Datasphere writes local tables via Replication Flows back into HDLFS, on which the outbound service performs encryption and marks the files to „collectible“

9 More Scope

...up to come soon. I will cover aspects how to further mature the microservice/python logic and moreover exemplify on HTTP based encryption/decryption service endpoints, exposed on the Kyma Cluster.

10 References

K8S/Kyma + Python SCN Blog

Getting Started with Data Lake Files HDLFSCLI

HDLFS REST API Guide

Thx...

...to my fellow colleagues at SAP who provides meaningful input and discussed about solution options. Furthermore I am especially grateful to our implementation partner who also supported, implemented and showed strong endurance in building up this scenario.

I hope you enjoyed reading this article and could have gained some deeper insights into what we did. If you have comments or suggestions of any kind, don't hesitate to comment and start-off a discussions with me and hopefully other SMEs.

Deep Dive into SAP Datasphere Object Store of BDC: Benefits, Architecture and implementation

2026-02-15T17:50:03.644000+01:00

In this document, I will go through the architecture and benefits of HANA DataLake Files service in the BDC Datasphere application. In the SAP Business business Data cloud there are 2 instances of HDLF object stores,

Foundation services HDLF object store - managed by SAP
Datasphere HDLF object store - managed by Customers

Object store is a file storage from Hyperscalers like Amazon s3 to store large amounts of data files(csv, parquet and delta tables)

FOS HDLF object store is used for SAP standard data products, when an intelligent application is activated in BDC Cockpit, required standard data products are installed and data is loaded to FOS HDLF object store. these Data products are available as Delta tables in the FOS HDLF, can be shared with Datasphere HANA Space or Databricks for data analytics and data science models. FOS HDLF is managed by SAP, Customers don't have access to it.

Datasphere HDLF object store is used for storing large data files from SAP and non SAP systems. these files are available as Delta tables. Customers can configure the ETL to replicate and transform data into Datasphere HDLF space. Also, custom data products can be generated in the Datasphere HDLF from BW PCE in BDC.

In this article, we focus on the Datasphere HDLF object store.

HDLF object store was made available in Standalone Datasphere(BTP) as part of data fabric solution from SAP, later it is introduced as part of BDC.

Benefits:

HDLF object store is cost effective solution, 1TB is 0.1 CU per hour.
Spark compute for delta table creation, scalable runtime for transformation flows in the HDLF space.
"SQL on files" feature, read data from HDLF space into HANA cloud models for analytics.

You can check the price quote for the object store and compute in the BDC price estimator https://bdc-pricing-estimator-sac-sacus10.cfapps.us10.hana.ondemand.com/

Architecture:

Embedded HDLF object store in BDC Datasphere is optional.
It requires additional capacity units, BDC pricing estimator can provide the number.
Data is replicated from Source systems and stored as Delta tables using Replication flows.
Delta tables are parquet files with transaction logs and maintained in the HDLF object store.
SQL on Files engine can read data from Delta tables of HDLF object store in HANA cloud without loading them to HANA Database. It is a Native HANA feature and explained in detail in this SAP blog SQL on Files
Apache SPARK compute provides scalable runtime for the transformation flows to process delta table(Parquet files) from Bronze to Silver to Gold layer Delta tables(Medallion architecture).

Configuration of Datasphere object store

The minimum capacity units required to activate Object Store in BDC is 4150.05 CU (128 GB of SAP HANA memory). You can Simulate price the BDC estimator, steps are provided in the SAP blog Objectstore CUs

After that you can configure the Tenant in Datasphere --> System --> Configuration --> Tenant configuration and add the Object store units. You can find the detail steps in the SAP blog Configuring the Tenant

after that, when we create a new space in Datasphere, option to choose HANA cloud or HDLF space will be enabled.

HANA Space Vs HDLF Space in Datasphere:

After configuring HDLF object store, we have 2 spaces in Datasphere.

HANA space
HANA DataLake file space.

HANA cloud space is the traditional space available in the Standalone Datasphere tenant(BTP), it requires HANA memory and HANA compute engine to store and process the data. On the other side, HDLF space is available in the BDC Datasphere and uses SPARK compute and object storage.

we must choose the space that we intend to work and build the objects accordingly. monitoring of jobs are specific to the space. Overall the look and feel experience for object development and monitoring jobs is same in both spaces.

In the HDLF space, replication flows and transformation flows are used to replicate and transform the large data files. Modeling objects like Graphical or SQL views and analytic models can't be developed in this space.

We can share the HDLF local delta tables(files) in HANA space and use them in transformation flows and Vice versa. that is Share the HANA space Local tables in a HDLF space and use them transformation flows.

End to end process flows Replication flow + Transformation flows

Replication flows are used to integrate/replicate data from Source systems into Inbound buffer first, then a merge task needs to be executed to write data to Delta table file. Transformation flows are used to transform the data to silver and gold layers of Delta table files in HDLF object store. These Delta tables can be shared in HANA space of Datasphere in the view builder.

Importance of merge task:

Since the Local tables (files) contains large data volumes, merge task updates the data by allocating required amount of SPARK compute resources. You can enable the merge task directly in the Replication flow configuration or manually run it from the monitor. You can schedule it as a task in the Task chains. Now you can cancel a running merge task in the monitor tab. Merge task is to be required for BW Data product generator delta table files also.

Note: In case of merge task is not run, then data is not available for reporting.

Python script can be written in the transformation flows and run with SPARK compute to process large data files.

Data product generator for Object store in Datasphere:

Data products are created as Local Delta tables(files) in the object store of Datasphere from BW InfoProviders. For the Subscriptions that are created in BW PCE, even delta data can be pushed from BW ADSO into Object store. these delta local tables can be shared in other applications.

Source from SAP

More detail explanation on the BW data products can be found this blog Data Product Generator

Limitations of Object store in Datasphere:

Replication Flows currently do not support local table (files) as source. So only Inbound.
Only 5 HDLF spaces can be created in Datasphere tenant.
Objects without Primary key cannot be loaded into embedded object store
Support of local type “initial load” only in Replication Flows is not supported.
Some datatypes don't support in Replication flows of HDLF space.
You will need to check the connection documentation to make sure if it is supported for Data lake files or not.

Summary: SAP Datasphere object store is a cost effective solution to store large data files as delta tables and transform the data with SPARK runtime and share them as data products in SAP Databricks.