SAP Community - SAP Data Services

Join vs LookUp - Which function is efficient in SAP CPI-DS?

2023-11-08T23:03:13+01:00

Hi SAP,

In this blog I am going to put forth a detailed discussion between the two functions in SAP Cloud Platform Intergration Data Services (CPI-DS) which can be said are two sides of the same coin, namely -"JOIN" (Inner Join) and "LOOKUP".

In my working with CPI-DS these are two are most common functions that I have used to build various dataflows to load MasterData and KeyFigure data across various business lines.

A little bit of background on the functionality of both functions.

"JOIN" function:

The "JOIN" function in CPI-DS is a complete tab on its own when creating a dataflow as visible under the Transform Details. "JOIN" helps to bring common results by matching same columns from Table A and Table B. There are two types of Joins which are available in CPI-DS:

Left Outer Join - In this Join, let's say we Table A on the Left Side input and Table B on the Right-Side input. A Left Outer Join will bring all the records from the Table A and only the matching records from Table B which is a left join will always bring all the records from Left Side Input and only matching records from Right Side Input.

Inner Join - An inner join will only bring records which are matching in both tables - Table A & Table B regardless of whether the tables are inserted into the Left or Right-Side Input.

When defining the Join function, we have to fulfill 3 things:

Select Left and Right-Side Input tables you want to use.

Select the type of join whether you want to use Left Outer join or Inner join.

Give proper matching condition in the Join Condition box and save it.

We can define multiple Join Pairs under the Join Tab. Each Join should be relevant and should make logical sense.

Additionally, there is a feature under the Join Tab called Join Rank. This Join Rank comes in handy when a single dataflow has multiple joins defined and it's required that there is order of priority set to the joins as to which join will be executed from first to last. Join Rank also as an option called "Cache" which allows data to be Cached according to the option set against it. (The Cache options include - Automatic, Yes, and No)

"LOOKUP" function:

LookUp fucntion allows result to be retrieved from a second table based on the comparison of columns between the original table and the lookup table. Additionally, apart from defining comparison between matching columns between the tables, CPI-DS also allows matching of specific Strings/ Characters present in the columns of the lookup table (For example: If a user only wants to LookUp records where in the column "PRODUCT DESCRIPTION" of the lookup table there exists String 'CHEMCALS'. This will only allow records from the column "PRODUCT DESCRIPTION" which has 'CHEMICALS' record in it. (Please note that the any given String is Case sensitive)).

LookUp too has three different types of cache functions:

NO_CACHE - This reads value from lookup table without caching records.

PRE_LOAD_CACHE - This loads the result column and the compare column into memory after applying filters but before executing the function. (By default, when applying LookUp function you can see that PRE_LOAD_CACHE is selected).

DEMAND_LOAD_CACHE - This loads the result column, and the compare column values into memory as the function identifies them.

As mentioned previously, Lookup and Join (Inner Join) function are two sides of the same coin. What I mean here is that both functions are used to validate data sets of first table (most likely a Source Table from SAP or NON-SAP Source) with a secondary table ( most likely in a reference or index table) so that only required data can be loaded into the system, garbage data can be eliminated, hence decreasing un-necessary processing and increasing efficiency of each dataflow in-terms of runtime and system resources utilized.

Scenario - I will be using a very simple scenario to analyse the use case of both functions. Let's say XYZ Company has multiple business lines - A,B,C,D,&E for which they want to create separate planning areas to perform demand and supply planning individually without bearing of business lines on each other. IBP team is able to define various Master Datas like Product, Location, Customer etc by identifying filters when loading data from source tables for each planning area. The catch is that the transactional source tables consist of data from across all the 5 business lines and IBP team needs to find a way to bring only relevant data for every business line. Now to tackle this IBP team decides to create extraction tables for each Master Data in MySQL and use that as a reference table when loading data to KeyFigure.

(Note: Join and LookUp functions used is enabling the "Cache" feature in all sub-scenarios mentioned further. For sake of simplicity PRODUCT MD load tasks are used as below to discuss the sub-scenarios)

TSK_SQL_PRODUCT_TASKWITHJOIN - Cache set to "YES" for the both source tables

TSK_SQL_PRODUCT_TASKWITHLOOKUP -Cache set to "PRE_LOAD_CACHE"

Sub-Scenario A: In this scenario we are loading data in both tasks without adding any other function or logic before Join or Lookup function.

"Image A"

"Image B"

We can see below in the Monitor Log for TSK_SQL_PRODUCT_TASKWITHJOIN when there is absolutely no other logic or function to be performed before the Join as in "Image A" the Absolute Time taken for it to complete the run is 7.421 secs.

Similarly, we can see below for TSK_SQL_PRODUCT_TASKWITHLOOKUP when there is absolutely no other logic or function to be performed before the LookUp as in "Image B" (Note: After the LookUp the "Default Value" in this case 'N/A' has to be filtered out) the Absolute Time taken for it to complete the run is 8.359 secs.

Findings - We can see in this case the difference between the two is negligible and Lookup took a second extra as it has to filter out Default Value 'N/A' as we don't want to load such records.

Sub-Scenario B: But what happens when we employ simple function just before the Lookup and Join function Here, we are taking a scenario where the data has to be cleansed using ltrim and rtrim functions as below.

Image C

Image D

We can see below in the Monitor Log for TSK_SQL_PRODUCT_TASKWITHJOIN when there is simple logic to be performed before the Join as in "Image C" the Absolute Time taken for it to complete the run is 10.468 secs.

Similarly, we can see below in the Monitor Log for TSK_SQL_PRODUCT_TASKWITHLOOKUP when there is simple logic to be performed before the Join as in "Image D" the Absolute Time taken for it to complete the run is 8.843 secs.

Findings - Again in this case we can observe the difference between the Join and Lookup is hardly 2 seconds which can be called negligible, but this time Lookup function was the faster task to completion.

Let's explore sub-scenarios where there are more than 2 tables involved in the Join and Lookup functions as below. (Here for sake of simplicity, I will be loading data to Location Product MD)

Image E

Image F

We can see below in the Monitor Log for TSK_SQL_LOCATIONPRODUCT_TASKWITHJOIN when there are more than 2 tables involved in an Inner Join as in "Image E" the Join function starts building a cartesian product between the 3 tables used here. This results in the creation of more than 337 million lines of record and counting as seen in the Monitor Log below. At some point of time this task will fail if it keeps performing the cartesian product process as it will timeout and/or lose connection to the server. Hence, making the Join Function unreliable in such scenarios.

We can see below in the Monitor Log for TSK_SQL_LOCATIONPRODUCT_TASKWITHLOOKUP when there are more than 2 tables involved in a LookUp as in "Image F" the Absolute Time taken for it to complete the run is 9.281 secs and unlike the Join function which most likely will fail even if left to run for a full day.

Findings - So we observed that Joins in cases where more than 2 tables are involved is going to build a cartesian product between the tables creating millions of lines of records. This results in a waste of system resources, time and money. While the Join function performs the same task within seconds and succeeds to load the data.

In conclusion, although Join and LookUp can be used interchangeable, in some complex scenarios it's best practice to pro-actively use LookUp function over Inner Join when building dataflows involving 2 or more tables. At the end of the day a Data Integration specialist should look to use the most efficient logics and functions to build dataflows which help build an efficient system and also save resources and time in the process.

I hope, I have added some value via this blog. Thank You for your time and patience for going through this write up.

Success Factor Adapter Issue in Data Services.

2023-11-14T22:24:03+01:00

Issue:

We have Success Factor as a source and BW as target, sometimes we used to get the below error and the jobs will be in hanged state or will be very slow. We got an update that reboot of system will help, but we have to leave the job to run slow as other DB jobs will be running as it will be affected.

Success Factor Adapter in DSMC during issue

Even reboot was not helping, and adapter was going into stopped state.

Adapter Status in DSMC after server reboot.

Solution:

Adapters in BODS will write their trace and error in server path like all the other jobs.

Example:

Z:\ProgramData\SAP BusinessObjects\Data Services\adapters\log

File name : SuccessFactorAdapter_trace.txt

When the path of adapter file was investigated it occupied the whole drive space and the file size was bigger. We renamed the existing file as backup and started the Success Factor Adapter, it started working and created new file in the adapter path of server. The old file was kept in a shared path for any further issue after retention policy so it can be removed.

Conclusion:

We have to figure what are the necessary files needed and always clear up logs after checking their importance. Cleaning it up in a retention period will be a best practice. Instead of reboot this work around help us to fix the issue and completes our job within SLA. After this there was no hang or slowness to the jobs.

New free Learning Journey: Exploring SAP Data Services

2023-11-20T10:26:16+01:00

SAP Data Services is a comprehensive data integration solution enabling businesses to extract, transform, and load (ETL) data from various sources. With features like data quality management, text data processing, and real-time data integration, SAP Data Services addresses the challenges of a heterogenous data environment.

The SAP Data Services Learning Journey provides a structured and comprehensive learning experience for both beginners and professionals. Embark on a focused Learning Journey designed for Developers and Consultants seeking in-depth expertise in SAP Data Services. Dive into key aspects like installation, scheduling, security, and configuration.

Upon completion, participants will gain the skills to create, configure, and execute jobs seamlessly using the SAP Data Services Designer.

Get started with the new learning journey on Learning.sap.com Exploring SAP Data Services to take the first steps in creating your own jobs in SAPs Data Services Designer.

Fill the gap through upskilling and enjoy SAP’s learning offerings on the SAP Learning site. This article is created and brought to you by SAP Product Learning CoE experts!

Partner Function update in PS Project WBS (Table IHPA)

2023-12-11T10:58:26+01:00

Introduction:

In Projects there are various people involved. This may include the project managers, project owners, Division Head, Finance etc. Normally most consultants use “Pers. Resp.No.” (VERNR) field to maintain this information. However, the requirements is to add multiple ids with different roles. This is where the concept of Partner Function in Project Systems comes in the picture.

Partner functions can be added both at Project Definition and at WBS level.

Issue

During PS Project system data migration there are no standard template to do so. The partner function data gets reflected in the IHPA table. There is no direct way to update the table.

Solution

We can use FM “PM_PARTNER_UPDATE” to do so. The same FM can be called through data services as well.

Sample dataset looks like below :-

Please keep in mind to populate the field UPDKZ. Otherwise it won’t reflect in the table.

'D' Delete

'L' Physically delete

'E' Inherited

'I' New

''' No alteration

'U' Update

UPDKZ = 'D' ---> Updates the field KZLOESCH as 'X' in the table IHPA.

UPDKZ = 'U' -> Updates entry in the table.

UPDKZ = 'I' --> Inserts new record into the table.

UPDKZ = 'L' ---> Record is deleted from the table.

Transformation Summary in SAP data service ( BODS) - Part1 - Validation and Case Transformation

2023-12-28T03:10:00+01:00

Today I am going to discuss different Transformation scenarios in SAP data service ( BODS) which is required for transformation in Dataflows.

1. Validation Transformation :

This Transformation is used to validate data. It filters the data based on the validation rule. If passed the rule, go to one target, If it fails go to another target set to deliver the failed items. If the data fails and violates the rule the the rule or failed condition goes to another target based on the scenario.

We need to create a validation rule on the validation transformation based on that it filters the data and sends it to the destination you mentioned in the target.

There is a standard project structure that we will use for all of our transformation and given below -

For example, if the Datastore of the source table consists of the below rows -

Sample DF Diagram for Validation Transformation :

In the PS1 table (set to get Passed) -

in fl1 table ( set to get Failed) -

Validation transformation filters the data based on certain conditions and sends the data to different targets based on rule validation.

2. Case Transformation: Similar to like case statement in JAVA, HANA, and ABAP. You can direct your source data row into multiple(N numbers of direction) targets based on different conditions.

For example, if the Datastore of the source table consists of the below rows -

Sample DF Diagram for Case Transformation :

Here we will apply 3 conditions on the case statement and if Cond 1-> Passed ->CUST1, Cond 2-> Passed ->CUST2, and Cond 3-> Passed ->CUST3

CASE 1: CUST.CREDIT_AMOUNT > 400000

Case 2: CUST.CREDIT_AMOUNT > 300000

Case 3: CUST.CREDIT_AMOUNT > 1000000

Put all the conditions similar to be below one by adding cases :

As a result :

1. CUST1 table generated with the below 3 entities :

2. CUST-2 table generated with the below 1 entities :

2. CUST3 - table generated with the below 2 entities :

# What is the difference between Validation and Case transform and which scenario we will use each transformation? which we will used to develop a better, cost effective and faster dataflow design ?

The execution time difference between the Case and Validation transforms is probably very very small. But the target limits two main tables/files based on fail and pass. Case can use for route the data to multiple target based on different case.

In some business scenario , the load failed reason required by business as a log. The Validation transform can gather statistics about the validation process and very useful in that scenario.

Transformation Summary in SAP data service - BODS - Part 2 - Merge and Query Transformation

2023-12-28T21:17:44+01:00

In this blog I am going to discuss the Merge Transformation and Query Transformation in SAP Data Services :

Merge Transformation :

Merge Transformation is used to merge multiple data sources i.e. tables or files into a single data source. But the field all the sources need to have a similar number of fields, the same data type, and sequences.

Basically, merge transformation acts as a UNION.

For example, if we want to merge below source tables of Indian and Pak employees for an organization that exists in two countries:

Table 1: IND_EMP :

Table 2: PAK_EMP :

Design of a Merge Scenario :

The target table EMPLOYEE should contain all the data due to the union.

Query Transformation :

Query transform is the most used transformation in BODS dataflow. It projects the data source fields to a subset or the same set. it is used in single or with other transformations based on the business scenario.

It is used to -

Narrow down the column subset by projection.

Narrow down the data using the filter.

Merge data from multiple sources using join or union.

Perform other operations also.

For example, if there are two data sources like the below -

Data Source Table 1 :

Data Source Table 2 :

The business wants to know the highest salary of an employee outside India region. We can use query transformations for this -

Now double click on the Query transform and it will let you go to the query editor.

Now in the FROM tab put the joining condition.-

In the Where Tab editor add the below filter.

PAK_EMP.SAL >=10000.

Save the Job and execute it. The Data preview of the target table should look like the below -

# Which scenarios Merge and Query Transformation is preferable?

Merge transform only can merge data from data sources having the same number of fields, the same data type, and the same sequences. It acts like an UNION operation.

But Query transformation can combine any set of datasets, project a subset of data, filter data, and add parameter to data. The usage of query transformation is a very large set of utilities. It is commonly used with other transformation also.

Transformation Summary in SAP data service - BODS – Part 3 – SQL and MAP Transformation

2024-01-03T04:10:54+01:00

SQL and MAP Transformation are two transformation is important transformation used in different business scenarios.

SQL Transform: Similar to Query Transform It projects the data source fields to a subset of the same set. it is used in single or with other transformations based on the business scenario. But Query transform generates optimized SQL, while SQL transform allows directly entering SQL queries

But in SQL transformation the Query which we use to project or filter the data that is executed and mapped into output. Complex logic needs to extract data from procedures or Table functions based on certain input parameters. In those scenarios SQL transformation is beneficial.

In our case, we will implement the same business scenario where we populate the employees outside India having a salary of more than 10000.

Source Table :

The design :

The Editor :

The result should be the same as what we got using the query editor -

MAP Transformation :

Map transformation is a very important transformation. Map Operation modifies Opcodes (I/U/D) operations. Allows conversion between opcodes provided as a input from previous transformation. After map operation use can only use transformation that understand opcode or a permanent table. Query transformation is optional to use. The ‘Map_Operation’ transform changes the operation codes on the input data sets.

In Simple word If any row updated on the source table , using map transformation you can update or insert the data in the target data set.

Important Topic : When we will use MAP without Table comparison and when with Table comparison?

Map Operation doesn't identify the new or updated rows from the source automatically.

Table Comparison return Opp. Code for Insert, Update and Delete for each source row after comparing with target table data as an output.

So we Can implement Insert, Update, and Delete operations using a single flow for a data source if we use Map with table comparison. But if we want to used a Single MAP operation We need to use Parallel flows for each operation (insert, update, and delete).

The transformation Table Comparison able to identify the records that need to be updated or inserted in the target table by comparing the source. If there is a requirement of mix of insert and updates use TC along with Map operation.

But if you know all the records need updates then use Map only. Also, say you want to delete a set of records based on some condition then the map will work better.

Scenario 1 :

A. Map Operation with Table comparison :

In this scenario if a row is updated in the target system. The source will be updated. If a row is inserted or deleted in the source it will insert to and delete from the target.

It is similar result we can obtain from History preservation and Row generation along with table comparison -

A. Map Operation with Table comparison :

For a simple update, Insert and delete we can use the below flow with the default MAP Opp code setting -

But Without using Table Compare and Generation we can do the same update, insert and Delete. using below flow and opcode :

For Normal Opcode Setting -

For Insert Opcode Setting -

For Update Opcode Setting -

For Delete Opcode Setting -

In the next blog I will discuss about the Data Integration Transform like History Preservation, Table comparison Row generation etc.

Transformation Summary in SAP data service – BODS – Part 4 – Table Comparison and History preservation and Row Generation Transformation

2024-01-05T00:36:44+01:00

Today I will discuss on three useful Data Integration Transformation. Together if they used they can perform a number of transformation output required from business. These transformations are Table Comparison and History preservation and Row Generation Transformation

These major transformations are given below –

1. Table Comparison : Used to compare data between Source and Target Table/Flat files. Others and return opcode for Insert or update based on the expected changed column you defined in the transformation. It is not mandatory but good practice to use Query transform before table comparison to project/select the specific data. Key Generation is not mandatory or map function is not mandatory with Table comparison but in different scenario where insert required for update operation i.e. historical preservation needed they Key Generation and map Operation is useful.

2. History Preservation : This transformation is used to keep historical data which is updated as a separate row and set the flag to ‘N’ and for new data row, set the flag to ‘Y’. With Key_Generation transform history preservation transform gives better result. When source table row has operation code of Insert/Update then it insert a new record in the target table.

The below 4 field is preferable in Target Table structure to for History Preservation –

S-key (surrogate key) will act as the primary key as you will get duplicate records

STRT_DATE – have the valid from date

END_DATE – have the change date

Flag Indicating the current record or the old record.

Also preferable to have start and end date in the source column as well. Using History preservation transform with Table comparison and Key generation is preferable.

3.Key Generation : This transformation generate and extra key in incremental order to identify the old and new record inserted.

Using History preservation transform with Table comparison and Key generation is preferable.

Now we will discuss different delta mechanism implementation in BODS using those above transformation.

Most common and popular types of delta mechanism are SCD Types and CDC types (source and Target Based) which we are going to discuss.

The three transformation used in different scenario together to achieve different goal.

In the next session I will discuss about some of the Data Quality Transformation that will required day to day basis as a BOS+DS developer.

Important Quality Transformation Summary in SAP data service – BODS – Part 5 – Match, Data Cleanse and Global Address Cleanse

2024-01-12T05:30:38+01:00

Before push the data to a data flow it is better to profile and cleanse or maintain quality of the data. For data profiling, Information Steward is the best tool which we will discuss later in a different blog.

My intention in each and every blog is to a point explanation of any object which is only required for any developer to start development. For more details, there are many places where it is described.

Today I will discuss three important Quality Transformation in SAP data service. These are given below -

1. Match:

By seeing the name you can predict that this transformation match the records and Identify the records. But Match transformation do more than that.

For an example in one of the cases It will identify that potential identical data may be duplicate from data those do not have primary key to identify the duplicated. Based on matching method an criteria it can identify those duplicated, then -

Classify into Master and Subordinate data

Fill the master data from the Subordinate data.

Remove the subordinate and keep only master data row.

Source Table :

There are many kind of MATCH transformation is there like Match Base, Address Match , Different type of consumer match.

By changing the configuration need to set rules for matching records then establish the thresholds and scores that BODS uses to determine near matches and matching records. it is based on the below rules-

Rule Based method

Weighted Scoring method

Combination method

We will use simple Base_Match for that.

Configuration in Match Wizard:

Expected Result Table :

There are multiple scenarios where we can use Match Transformation different types.

2. Data Cleanse :

Used to clean the data to prepare for other transformation . The below option you can use to clean the data , replace a pattern of string using the below option -

Remove diacritical characters.

Information and status codes

Phone parser

Parse discrete input

New Data Cleanse output fields

New Data Cleanse input fields

For more detail you can visit the below SAP URL in which it is described in very clear and details .

https://help.sap.com/docs/SAP_DATA_SERVICES/9f1b4472ec98409682d91953b9e68c92/578e652a6d6d1014b3fc9283b0e91070.html

3. Global_Address_Cleanse :

Global Address Cleanse transformation are used to parse the input data ,cleanse the data , correct and standardize address data for different countries.

The address proposed field are given below -

Street Number

Street Number

Apartment Numbers

Cardinal Directions

Locality

Region

Zip Codes.

In the panel we need to configure the scenario and customized accordingly.

In each scenario we need to customized as per retirement.

Repository , Job Server and Job server Group Creation in SAP Data service

2024-01-16T02:01:05+01:00

Today I am going to discuss about Repository , Job Server Creation, Job server Group in SAP Data service. Some part is part of the administration job but it is better to let know all the stubs before starting design and development in the BODS Designer.

SAP BODS System Preparation :

1. Database Creation :

This is purely a DB work and DB admin is used to create the DB. This is the first part of the preparation process. There are many places where the steps are given and I will not discuss in detail this steps.

2. Repository Creation(Local, Central, Profiler) :

What is Repository?

Repository is a place used to store metadata of objects used in BODS. Each and every Repository should be -

Registered in Central Management Console, CMC.

Linked with single or many job servers, which are responsible to execute the jobs that are created.

Different type of repository :

A. Local Repository : The local repository stores metadata of all objects like , workflow, data flow and projects which you developed.

B. Central Repository − This repository like a main branch of Git repo. There are local branch(Local repository here) where developer used to develop their artifacts and push the version to the main branch .It is used to control the version management of the objects. Central Repository stores all the versions of an application object.

C. Profiler Repository −

Data Profiling means cleansing, prepare and standardize the data for use in enterprise landscape.

This repo used to manage all the metadata related to profiler tasks performed in SAP BODS designer. Information Steward used for data profiling and this repository stores metadata of profiling tasks and objects created in information steward .CMS Repository stores metadata of all the tasks performed in CMC on BI platform.

Steps to create Repository :

A. Go to SAP Business Objects Data Services 4.2 SP3-> Data Services Repository Manager.

B. Create the repository by providing the below details :

Repository type , Database Type, DB name, user name and password.

Click on the Create button.

If a REPO was already created with the same name then it will give you a warning and not allow you to create another one.

3. Attach the local repository to the Job Server or Job Server Group :

The created local repo needs to be attached to a Job server. The object like Data flow created in the Local repo needs to be executed on a server. The Job server is the server where we will execute the developed objects.

Note : In the popup wizard you can add and existing Job Server or you can create an new Job Server and associate it. Our case we will create new jobs server.

A. SAP BusinessObjects Data Services 4.2 SP3->Data Services Server Manager.

B. Fill the credentials : Job Server name, Job Server port, Database type, Database Server name, Database name, Username and Password and check the default repository.

C. Test the created Job server :

3. Repository Registration in CMC :

A. After logging into the BO CMC URL go to the Dara Service -> Repository :

B. Then right-click on the link Repositories -> Configure repository or right-click on the link Repositories -> Manage -> Configure repository and fill in the credential.

4. Job Server Creation:

That portion was already discussed while attaching the repo with the Job Server in section 3B.

5. Job server group creation :

Before creating it let's know what the Job Server Group is and why we need Job Server Group?

Server Group is a logical SAP Data Services component that is used to group Job Servers on different computers or the same computers.

A Job Server group automatically measures resource availability on each Job Server in the group and distributes scheduled batch jobs to the Job Server with the lightest load at runtime.

All the Job Servers in an individual server group must be associated with the same repository, which must be defined as a default repository. The Job Servers in the server group must also have:

- Identical SAP Data Services versions
- Identical database server versions
- Identical locale

Each computer can only contribute one Job Server to a server group.

Note : This diagram taken from SAP.

** Steps to create Job Server Group :

Job server Group required at least two or more Job servers.

From the SAP Data Service Management Console -

Select Server Groups- > All Server Groups

Click the Server Group Configuration tab.

Add a server group by using the "Add" button

select a repository, all Job Servers registered with that repository display. You can create one
server group per repository. Keep the default server group name. It is the name of your repository with the prefix SG_ (for server group). You can change the default name, however, labeling a server group with the repository name is recommended.

One Job Server on a computer can be added to a server group. Use the Host and Port column to verify that the Job Servers you select are each installed on a different host.

After you select the Job Servers for a server group. Job server Group required at least two or more Job servers.

For more details, you can visit the reference URL -

https://help.sap.com/docs/SAP_DATA_SERVICES/4d2e69d5c5c2492681f0c64062ba66f1/574f3d336d6d1014b3fc9283b0e91070.html

Transport in SAP Data Services

2024-01-31T04:56:40.675000+01:00

Today I will discuss about the different transport techniques in SAP data services. Based on the project scenario and configuration we will select appropriate techniques.

1. Normal Import ATL file and export:

Taking Backups of ATL Files.
Select the object that needs to move in the local object repository.

Right-click ->export->export to ATL File. Select the Target REPO of QA or Prod Environment. Then - Right click->Import->Import from the backup ATL File. change the Datastore connections according to the environment.

2. Using Central Repository:

By using Check in/check out concept.

1. Create a central repository and connect it to the local repository and activate the central repository.

2. From Local repository: Select Local Object -> Right Click. Add to central repository.

In another way -

1. You need to go to tools and select Central Repository:

2. From the dialog box we need to select the object which we want to promote.

3. From Target Repository in different environment - Check out the PROJP from central repo "with objects and dependents without replacement" option and check in again "with objects and dependents" OR

4. Right click on the Object à Get Latest Version à With Filtering.

3. Transport or promote the object with CTS + :

Here I will not discuss about the CTS+ configuration. You can go through the below link –https://help.sap.com/doc/f6e6dd2617e44e1fa06e8a6e023b63fa/4.2.14/en-US/ds_42_cts_en.pdf

I will discuss about how to transport or promote object with CTS++. Below are the steps –Basic Architecture of CTS + configuration with BODS is given below -

Steps to Transport:

1. Navigate to SAP Data Services Development system Administrator screen and click on Object Promotion

2. Select repository type and Type -

3. Select the objects you want to add and press export button:

4. Check when the Job/Object export completes :

5. When export complete release transport for next system :

6. Import transport will be in your target queue :

In the next blog I will discuss about the job scheduling in SAP Data Services.

Load data to SFTP folder Using Authorization Type as Public Key in SAP Data Service

2024-01-31T13:18:53.136000+01:00

Requirement:

Load data to SFTP folder Using Authorization Type as Public Key in SAP Data Service

Description:

The blog will provide information about the use of file location object to upload file in SFTP server using Authorization Type as Public Key

We already have a blog that give you a good idea of how to use the File location object to access files in SFTP and FTP after Document Version: 4.2 Support Package 6 (14.2.6.0). The blog talks of the file config using the authorization type as password.

Export File to SFTP/FTP

I will be focusing only on the SFTP file upload using authorization type as Public Key:

Follow the steps to create the file location for SFTP as mentioned in the above blog.

Select the authorization Type as “Public Key” and you would get the below options:

You will be provided by the Username and RSA Private key by the SFTP team. With the provided RSA private Key, you won’t be able to create the connection and would get the below error message:

To resolve the error, use a software that would be able to generate the Public and Private key using the provided RSA Private key. We have used “PuTTY Key Generator” (You can use any other available software).

Open the PuTTY Key Generator and Go to File and select “Load Private Key”.

Once loaded you can select “Save public Key” followed by “Save private key”. The Public key should be saved by file extension “.pub” and the Private key with file extension “.pem”.

Note – You can also paste the generated public/private key in a note pad and save it with the extension as mentioned above.

File Location Configuration:

Name: Name of the file location
Protocol: Type of protocol (SFTP in our example)
Port: you can leave it default. Usually for SFTP it is 22 and for FTP it is 21
HostKey fingerprint: It is required for SFTP not for FTP. But don't worry system will propose this key once user and other provided details are passed.
SSH Authorization Private Key file path: The file path where the Putty Generated Private key is saved.
SSH Authorization Private Key Passphrase: Generated
SSH Authorization Public Key file path: The file path where the Putty Generated Public key is saved.
Remote Directory: SFTA Path where the generated file is to be pushed.
Local directory: Required only if you are copying it to local system folder.
Connection retry count and interval can leave as it is.

Once the file location is configured go to the file in data flow chose the file location from drop down.

Provide the file name and you are good to go 🙂.

Job Scheduling in SAP Data services

2024-02-04T18:35:05.210000+01:00

In This blog, I will discuss the steps to schedule a job using the Data Services Management Console.

A. Steps are given below for batch Job scheduling -

1. Log In to the Data Services Management Console.

2. Now we need to 'Administrator' and expand the 'Batch' menu –

3. Select the repository where the job you want to schedule has been developed.

4. Please Go to the 'Batch job Configuration' Tab. Select the project of your job. Click on the right hand side 'Schedules' which is under the other information column of the job for which we want to create a schedule –

5. Then click on the 'Add' button. Give a name for your schedule. We need to select Date-Day for our schedule. There is an option of 'recurring' if want to schedule the job in recurring. In our case, we will schedule it once a day at 11.23 PM. Need to click on the 'apply' button and a schedule for your job is created.

7. If we want to check, activate, and deactivate the created job then go to the 'Repository Schedules' tab and there we can see the scheduled job.

B. Real-Time job scheduling:

Real-time jobs do not need to wait for an internal trigger or a scheduling like a batch job. It is waiting for a message from the access server, once it is it starts automatically.

Realtime Jobs are suitable for XML Messages. The basic architecture is like below –

Before we need to start development, we need to configure the access server.

Real-time jobs start with a message and end with a message. The input message must be the first step in the job and the end message the last one. After the real-time job is published, DS checks the job inside for tables, syntax, and all objects and if everything is OK, the job starts and can accept requests.

Hope it will help. In the next blog I will discuss about the different fine tuning techniques/performance tuning techniques in SAP data services.

Moving a Task from One CI DS Organization to Another

2024-02-13T09:39:40.899000+01:00

The process of importing a task from one SAP Cloud Integration for Data Services (CI DS) client to another is like copying a job or task. You can transfer a single task or all tasks within a project by exporting them and subsequently importing them into a different organization or a new datacenter.

Export Task:

Choose the specific task or project that includes the tasks you want to export.

Click More Actions > Export.

A file is saved to your local Downloads directory. Single tasks are exported to a flat file in XMI format and saved with a .xml file extension. All tasks in a project are exported in a zip file.

Import Task:

After exporting a single task or all tasks in a project, finalize the transfer by importing them into a new organization or datacenter. To import tasks, you must have the Administrator role. Tasks are imported into a project, so identify the project where you intend to import the tasks.

Choose the project where you intend to import the individually exported task or group of tasks from an exported project, then click on "More Actions" > "Import."

Once you click "Import," a new window will pop up. Follow the instructions on the screen to choose the file containing the task or project you want to import.

Browse to the location where you saved the exported task or project.

If you exported a single task, the file has a .xml extension.
If you exported a project, the file has a .zip extension.

Additionally, provide the task name, source datastore name, and target datastore name. If the Datastore belongs to a File Format Group, check the box located on the right side of the Datastore details.

Click OK.

The task will be imported successfully. Now, you can change the target or source by editing the dataflow. You can modify the transformations, filters, and mappings as per the required.

SAP BI 4.x Installation - "Error" An install is already running at this location on Linux

2024-02-14T09:34:56.200000+01:00

Recently I was tasked to install SAP Data Services on Linux and while doing so I faced an issue where the installer failed with the error - "Install already in progress"

Environment:

Application: Business Object Data Services

OS: Linux

Steps to reproduce the error:

Log on as a non-root user
Set the environment variables
Run ./setup.sh of the BI installer (IPS in this case)
Enter the install directory
The installer will fail with the error message - "Install already in progress"

I tried the below steps to resolve it but failed:

1. Delete and recreate the installation folder

2. Perform a reboot of the server

I found a few notes and blogs related to this error that pointed towards deleting all files ".mutex" in the /tmp folder. While trying to follow one of the SAP notes 2790905, I found no ".mutex" file in the /tmp folder as no installer was running in parallel or the root user was used to perform a failed installation.

The logs were not helpful at this point as they were directing to an installer running in parallel or failing causing the issue.

To gather more helpful logs I tried running the trace as follows:

strace -f -o install.out ./setup.sh (followed by your normal patch install command arguments)

(note, install.out needs to be written to a location this user has permission to write to. So you might need to run it like so: strace -f -o /tmp/install.out for example)

Once the installer fails, check the install.out file for "mutex" by running the below command

grep mutex install.out

In my case, the output showed that the user was unable to set the "mutex" file due to lack of permission and was assuming that the presence of another such file was preventing this hence the error message read - "Install already in progress"

After setting the correct permission I ran the installer again and this time it moved past the error resulting in a successful installation.

Conclusion:

In case anyone comes across this issue and is unable to find any ".mutex" file in the /tmp folder to delete then perform a check on the write permissions of the non-root user being used to perform the installation. The strace command can also be run to ascertain the root cause.

Refer to the note: 2790905 - An install is already running at this location on Linux in BI 4.x while performing update

Performance tuning in SAP Data Service

2024-02-19T20:07:03.690000+01:00

Today I will discuss about performance tuning in SAP Data Service.

Performance tuning in SAP Data Service can be divided into two parts :

Configurations/ Setting Changes :

1. Use Degree of parallelism (DOP) option in the data flow to a value greater than one and 3-4. Use the data integrator features like Table partitioning, File multithreading , Degree of parallelism for data flows with DOP to get better performance.

2. In Source database If we increase the size of database I/O, increase the size of the shared buffer to cache more data if will help to perform the Select statements quickly.

3. In Target Database disable the Archive logging, disable the Redo logging for all tables. It will help to perform INSERT and UPDATE quickly.

4. We need to increase the monitor sample rate to 40K or more as per standard to get more performance.

5. To avoid performance degradation we need to exclude the Data Services logs from the virus scan if the virus scan is configured on the job server.

6. For the first time execution Select the option COLLECT STATISTICS FOR OPTIMIZATION . For the second time onwards. Use collected stats which is selected by default.

7. Design a job such a way that runs one ‘Al engine’ process per CPU at a time.

8. Based on the scenario ( for Large scale of data) increase the Array Fetch size value to avoid repetitively go to the database every time to fetch the data. Set it more than 800.

Design/Development Changes :

1. Push Down all the transformation logic implemented in SQL to Database layer to leverage the power of the database engine. To check the Optimized code pushed to the database engine please check the below -

But there are few restriction and clause for SQL push down operation.

When pushdown is not possible in a DF then enable Bulk Loader on the target table. Bulk loader is much faster than using direct load.

In case there is a "select Distinct" then the code will not push down fully. If we can not avoid to use it please use select distinct then use it in the last query transform before the target table.
Use Single Query Transform you want to use Group by and Order by.
Try to avoid data type conversions which prevents full push down.
Always try to avoid parallel execution of Query Transforms which prevents full push down.

2. Join Rank : Please define the rank properly to make execution speed faster. Open the Query Editor assign the rank for larger tables. The highest ranked table will act as a driving table and execute . Monitor log file allows you to see the order in which the Data Services Optimizer performs the joins which will help you to identify the performance improvement. To add the trace go to Optimized Data Flow --> Trace tab --> Execution Properties dialog.

3. Set the "Rows per Commit" value between 500 and 2000 where default is 1000.

4. Try to split complex logics of a single dataflow into multiple dataflows which is easy to maintenance and help to SQL pushdown.

5. Create index for columns which used in where clause as joining condition. This is a major point to improve performance drastically.

6. Always full pushdown is not possible. In that scenario if the dataset is larger enable the Bulk Loader on the target table.

7. Use join instead of Lookup function if possible.

Data Quality Audit

2024-02-21T12:46:34.652000+01:00

One of the indispensables, essential and dominant parts of our lives is our smart phone. You can almost say there is nothing a smart phone cannot do nowadays. However, technological advancements do not stop there, we have continuous innovations been introduced into our surroundings.

Over 20 years, we saw the first mobile phones with antennas that evolved into faster, lighter, graphical, and more intuitive devices. This is a splendid example of technological transformation.

We started off with just voice communication over analog signals which transformed into so much more. We can now access the world’s news and information at our fingertips, communicate and network within our circle - even with strangers based on our geographic location - shop, trade, manage finances and organize our lives globally from wherever we may be.

As mobiles get smarter and smarter, think about the data that has been transferred from one device to another. And these transfers came with several implications and adjustments to the data over the years. For instance,

Compliance wise – with the onset of international calling, phone numbers were suddenly required to be stored in international format.
Competition wise – amongst varied brands, more features kept pouring in terms of speed, additional fields, and memory to store more contacts and addresses.
Looking at technology trends – devices now have the power of internet telephony, connecting themselves to self-driving cars and comprehensive mobile applications to become a pocket computer.

Data quality is the least of our concern as we are focused on the outcome to move to the new device and get started, by just ensuring that we retain the data in the new device in some form. The advanced intelligent machines assume implicitly that the data transferred to them is suitable for their intended purposes. We all know that transfers across platforms and vendors are limited by data models and technical limitations.

Consider this scenario: during the transfer of data between different device platforms, envision a situation where contact numbers or addresses are jumbled. You may have been consuming this data over years across devices manually, knowing the limitations and issues in the data, and deciding accordingly e.g., Mobile phone and Office phone fields have data interchanged during transfer, and postcode missed in the addresses during transfer from Apple to Google platform. Now, imagine relying on this data on your mobile device with a self-driving car to navigate you to your destination e.g., AI considering only street name to navigate and deciding between multiple options!

How can you trust the data that has been moved from your old devices into the newest device via multiple devices and platforms?

Would you be able to use the full capability of the new AI self-driving cars?

Is the data that we have been maintaining for decades fit to be used in these innovations without human intervention?

Or more simply, is the data fit for the new intended purpose?

Now coming back to the corporate world and considering the business point of view for customers who have moved their enterprise data assets across multiple platforms over last few decades to SAP ECC or any other ERP system. Most of these customers are now in the process of transforming their technical landscape and business processes to get ready for the future, i.e., Cloud! Whether these organizations are moving to private or public cloud, RISE with SAP or GROW with SAP, going greenfield or brownfield; the transition mirrors the evolution we discussed earlier with mobile phones but, in this case, businesses are shifting from their familiar ERP ECC systems to the futuristic landscape of SAP S/4HANA. It involves migrating from mixed landscapes, including SAP, non-SAP, and legacy systems, into the innovative SAP S/4HANA platform. The AI and Generative AI capabilities embedded in SAP S/4HANA mimics driverless cars we mentioned before. So, we find ourselves asking the same questions once again:

Does the data make sense for the S/4HANA system? Can we take advantage of the GenAI capabilities?

Can we rely on and trust the data for the system to predict your business outcomes or take business decisions?

In the end, is the data fit for intended purpose?

Data quality (DQ) is always the key focus and concern from the business at the beginning of the transformation, however as program evolves, DQ topic fades as the business priority moves to ensure processes are designed and tested. The challenge is that no one could really quantify and establish the problem to define a business case to invest in data cleansing to save costs for the organization. By the time business users realize the scope of issues through migration, it is too late to fix this problem and they are already testing the processes. If at the end they are successful in migration, they still end up with technically correct and not “business correct” data set.

This helps us see those businesses, like individuals with mobile phones, need to make sure their data is ready for the changes and innovations in their systems.

How do we ensure data is fit for purpose?

Let us address this topic again with the same mobile phone analogy. Consider this scenario: you are moving from an old Nokia phone to an iPhone. The former mobile phone saved contacts separately and did not have any tagging features to link landline or mobiles numbers of the same person together. However, the new iPhone has the feature to add multiple numbers with tags to a single contact. Now when you are moving to the new phone quite literally, the contacts can become redundant in your new device and cause confusion. How would you be able to tell apart which number is landline and which is mobile?

Similarly, what if some mobile numbers were transferred incorrectly and lead to missing information in your contacts? Or during transfers and conversion of formats to comply with international codes, incorrect patterns were transferred, or the numbers truncated?

This would affect your new device’s ability to function as expected and for you to utilise innovative features on your contacts.

Many such examples can come up in various parts of the transfer journey between the devices especially when you are moving between technically diverse landscapes. How do we ensure we do not move junk or bad data into our systems? Or if we already have moved without auditing the data first, how can we identify inferior quality data in our current landscapes?

The first step to check if data is fit for purpose is to understand the depth and extent of your data issues. To understand your data issues, you need to audit the data. The data may appear clean, but the question is if it’s fit for the intended purpose.

In the business world, the information spread across your systems is undeniably the most valuable asset. Achieving 100% data quality is a challenging task for any organization, but depending on how crucial each aspect is to the business, you can prioritize and make it feasible over time. For example, if there are 100 fields or attributes that require attention, and only 70% of them are essential for your business operations, you can identify key areas to concentrate on and begin planning for improvement. So, where do we begin?

What is Data Quality Audit?

Data Quality Audit, a service offered by SAP, assesses, and checks the actual data stored in systems to uncover the existing issues and determine fitness for target systems like S/4HANA, C4C, ARIBA, MDG etc. Irrespective of the number of systems you have, or at which stage you are in your transformation journey, Data Quality Audit checks the ‘quality’ of the data in terms of ideal principles set by your business and generates a ‘scorecard’ along with a business impact ‘cost analysis’. This allows organizations to assess the status of their data and then help formulate their cleansing, resource and migration plans accordingly.

Let us briefly look at the steps that this service follows to achieve the results:

Once scoping is established, data quality ‘rules’ are defined on the data with business or IT. The ‘rules’ originate from ideal principles of data quality: Accuracy, Completeness, Conformity, Consistency, Integrity, Timeliness, Uniqueness. SAP offers best practices content of data quality rules as a QuickStart to any quality audit.

These ‘rules’ are then executed on the data, and ‘scorecards’ are established in a dashboard-like view. The scorecards also depict the associated cost impact analysis for the poor data quality. This then helps enterprises to assess their data, constantly review their data during the cleansing activities, supporting control and governance.

To conclude,

Data Quality Audit is a service that is pivotal in the initial 'Get Clean' phase of your transformation. Unfortunately, these aspects are often underestimated, leading to the failure or delay of numerous transformations. Taking these steps seriously is crucial, as neglecting them can jeopardize the success of your large-scale investment, impede growth, and hinder the identification of business issues.

Consider these three key takeaways:

Recognize the vital role of Reference Data Quality Audit and Data Quality Audit in achieving a clean transformation. Look out for a similar blog on Reference Data Quality Audit - stay tuned!
Be mindful of the commonly underestimated nature of these steps, as it can significantly impact the success of your initiative.
Formulate robust 'Stay Clean' and ‘Get Clean’ plans based on the insights gained, avoiding pitfalls that have derailed many transformations."

Farewell to an Era: The End of an Iconic Certification of SAP BODS - C_DS_42

2024-03-31T19:16:30.842000+02:00

Hello All,

continuing to the previous blogs on SAP DS 4.2 and 4.3

Today, we're diving into a topic near and dear to the hearts of many in the data services realm: the legendary certification, C_DS_42, for BODS 4.2. For over a decade, this certification has been the holy grail for data services consultants, a badge of honor symbolizing expertise and proficiency in SAP's Data Services platform.

However, as with all good things, its time has come to an end. On the 28th of March 2024, the C_DS_42 certification officially expired, marking the end of an era for many professionals who have cherished and pursued it.

What makes this transition even more surprising is the fact that its successor, C_DS_43, the certification for the latest version of Data Services, has also met the same fate, entering the realm of expired certifications.

For those who have dedicated countless hours to mastering BODS 4.2 and preparing for the C_DS_42 exam, this news may come as a bittersweet moment. It's a reminder of the ever-evolving nature of technology and the necessity of staying current in a field where innovation waits for no one.

But fear not, for every ending marks a new beginning. While bidding farewell to C_DS_42 may evoke a sense of nostalgia, it also presents an opportunity for renewal and growth. As the industry moves forward, so must we.

So what's next? For those who held C_DS_42 with pride, it's time to embark on a new journey. Whether it's upgrading to the latest version of Data Services or exploring other avenues within the data management landscape, there are plenty of opportunities to expand our horizons and continue making a meaningful impact.

As we reflect on the legacy of C_DS_42, let's remember the skills and knowledge it has imparted upon us. Let's honor the hard work and dedication that went into achieving this certification, and let's carry that same passion into the next chapter of our professional lives.

Farewell, C_DS_42. You may be expired, but your legacy lives on in the countless professionals you've inspired along the way.

Here's to new beginnings and the endless possibilities that lie ahead.

Thanks,

Venkatesh Golla

Futuristic Aerospace or Defense BTP Data Mesh Layer using Collibra, Next Labs ABAC/DAM, IAG and GRC

2024-04-11T19:00:57.747000+02:00

Background

In this blog, we will explore few ideas for creating a domain-centric data mesh using SAP components, without engaging in the debate of whether data fabric or data mesh is the right approach.

We will focus on a futuristic use case within the Aerospace or Defense industry, which demands stringent data governance due to compliance requirements such as ITAR, EAR, BAFA, DOE 810, NERC/CIP, and SEC. Additionally, safeguarding intellectual property is a critical concern as business growth often relies on increased collaboration, both internally and externally, spanning product and engineering, supply chain, cross industry- partnerships, and joint ventures. I am happy to hear your ideas too!

Business Problem

Aerospace and defense customers often face challenges in keeping up with rapid technological advancements due to the burden of data debts and the need to comply with stringent legal and regulatory data security requirements.

This challenge is further amplified when dealing with legacy ERP systems, as identifying and restricting sensitive data becomes complex, hindering the end-to-end data lifecycle management. Moreover, the Aerospace and defense industry grapples with the formidable challenge of reducing IT costs, as the presence of significant data risks renders offshore operations an impractical option.

Let's consider the example of a product manager based in the United States, working for a US corporation. Their product is subject to ITAR regulations but has both government and commercial applications.

In order to comply with the business rule, the access to ITAR data in SAP should only be granted to US persons while they are in US locations. However, when this product manager is on a business trip to Singapore, meeting with suppliers at their APAC regional headquarters, exposing material data, CAD drawings, or BOMs stored in SAP would violate ITAR regulations.

In the context of a UK energy company establishing a joint venture with a local company in China to cater to the emerging market, an added layer of data security is required based on location. This ensures that access to BOM items and intellectual property not related to the joint venture is restricted, safeguarding sensitive information and preserving the integrity of the collaboration.

In the context of the aerospace and defense , "Data is not only the ammunition that fuels engines but it is also an the armor that protects international and national peace"

BTP SAP S/4 HANA Data Mesh Pattern

To solve the problem described above, we will hypothetically integrate Collibra, Next Labs ABAC/DAM, IAG and GRC and SAP S/4 HANA to provide data insights to users without compromising data security requirements. Please note that is is hypothetical pattern and we have to review and apply it according to client specific data security requirements.

Solution Components	Usage
Collibra	This component can be used to document and define data governance policies, meta data catalogue, end to end data lineage, data quality KPI's, data protection and data privacy rules for bill of materials.
SAP IAG	SAP IAG can handle the user access request and provisioning workflows to authenticate and authorize the user identity with the client's identity provider (ex: Active Directory)
SAP GRC	SAP GRC can provide the necessary controls and policies for access management and perform risk analysis and segregation of duties (SoD) checks, ensuring compliance with regulatory requirements across all SAP applications in the landscape.
Next Labs ABAC/DAM	Next Labs ABAC/DAM provides robust protection against unauthorized access to sensitive SAP data by implementing fine-grained access controls. These controls can be applied at the level of individual data attributes or data ranges, enabling customers to safeguard their data while meeting compliance requirements. By examining the attributes of the data being accessed, the context of the request, and the user's identity, Next Labs ABAC/DAM allows organizations to control access to data, business transactions, and batch processes based on defined policies. With SAP DAM, any changes in the attributes of the data or the user are dynamically considered, and the relevant policies are applied in real-time to enforce fine-grained access controls across various business functions. For example, a rule may specify that only US-based employees can access ITAR-classified materials from US locations. When a user attempts to access such materials, this rule is validated in real-time, ensuring that access is granted only to authorized individuals who meet the specified criteria. Through the integration of Next Labs ABAC/DAM with SAP systems, organizations can effectively protect their sensitive data, maintain compliance, and enforce granular access controls across a wide range of business operations.
SAP Data sphere	This is optional but we can use this if you want to provide flexible predictive analytics to the users.
SAP BTP AI Launch Pad	This is optional and can be used to identify repeat breach patterns, time and detect security data anomalies in advance and add further access controls.

MVP- Bill of Material Data Mesh

In the world of aerospace and defense, organizations face the challenge of managing bill of material (BOM) data across multiple systems. R By integrating diverse systems such as SAP S/4HANA, Team Center, and Siemens, they created a unified network of interconnected data mesh. This will enable seamless collaboration among internal and external engineering, supply chain, and product sales teams. With real-time visibility into BOMs, teams made informed decisions, optimized designs, synchronized manufacturing, and tailored offerings. The BOM data mesh can empower the organizations to achieve faster product development cycles, reduced costs, and improved customer satisfaction.

Step 1: Define Role Requirements, Meta-data catalo Data Governance and access policies for BOM in Collibra

Identify the specific role requirements based on your organization's needs and compliance regulations.
Determine the attributes that will be used for access control, such as user roles, data sensitivity, and contextual factors.

Step 2: Understand BOM Creation and Editing Requirements

The organization requires that only users with specific engineering roles can create and edit BOMs. Additionally, certain fields in the BOM may be restricted for external supply chain users based on data sensitivity, such as pricing information.

Determine the functional access (actions user can perform) and data access (data records or fields users can see) and governance (rules for access)

Step 3: Define SAP S/4HANA Role

Create a custom role named "BOM Specialist" or copy standard role in SAP S/4HANA.
Assign the authorization object M_BOM_GRP to the role, which controls access to BOM groups.
Assign transaction codes CS01 (Create BOM) and CS02 (Change BOM) to allow users with this role to perform BOM creation and editing tasks.

Step 4: Configure Next Lab ABAC DAM

SAP Next Lab ABAC DAM works natively with SAP and manages authorization logic through an externalized, standards-based policy framework. For instance, a rule may state, “Allow only US-based employees to access ITAR-classified materials from US locations.” When a user attempts to access materials, this rule is validated in real-time before access is granted.

Configure Next Lab ABAC with attributes such as "User Role," "Data Sensitivity Level," and "Contextual Factors."
Define "User Role" as an attribute to determine the user's role in the organization.
Define "Data Sensitivity Level" as an attribute to classify BOMs based on their sensitivity, such as "Confidential" or "Public."
Define "Contextual Factors" as attributes to consider additional factors, such as project or department.

Step 5: Integrate SAP GRC and Next Lab ABAC

This MVP use case leverages SAP GRC Access Control and SAP authorization for Governance and Functional Authorization and leverages ABAC for Data Authorization. It combines the features and fully integrated capabilities of SAP GRC Access Control and SAP authorization, such as ease of user assignment and role management, to efficiently supporting data attributes and avoiding the “role explosion” and custom development that would otherwise be necessary and costly.

Integrate SAP GRC and Next Lab ABAC to synchronize roles and access control policies.
Map the "BOM Specialist" role in SAP GRC to the corresponding role in Next Lab ABAC, ensuring consistency in access control.

Step 5: Define ABAC Policies

Define ABAC policies in Next Lab ABAC to enforce attribute-based access control for BOM creation and editing.
Create a policy that allows users with the "BOM Specialist" role (User Role attribute) to create and edit BOMs.
Create a policy that restricts access to certain fields in the BOM based on the "Data Sensitivity Level" attribute and user profile policy (Ex: Manufacturing, Engineering, Supply Chain).

Step 6: Test and Validate the Role using AI

Train the model on different user profiles to validate data access using SAP BTP AI Launch Pad
Train the model further to auto-correct access issues and detect patterns to suggest changes to role access profiles (don’t let AI implement dynamic role access changes as it can be dangerous 😊)
Perform supervised automated test to test the “BOM Specialist" role by assigning it to a user and verifying that they can successfully create and edit BOMs.
Validate that the ABAC policies.

Step 7: Integration SAP BTP Identity Access Governance to Active Directory

By integrating SAP BTP Identity Access Governance directly, users can seamlessly access data from multiple systems, including SAP S/4HANA, Team Center, Siemens, and other engineering, manufacturing, and supply chain systems. This integration enables a cohesive data mesh approach, allowing users to view and manage bill of materials across various systems.

Step 8: Integrate Collibra and Datasphere to Monetize and Publish bill of material insights to engineering, supply chain and product sales team.

You have the ability to define and design self-service analytic insight reports, which can be monetized and shared with both your internal and external engineering, supply chain, and product sales teams.

Why Don't We Use Data and Analytics Group on SAP Community?

2024-04-18T18:28:14.375000+02:00

Hey Data and Analytics Folks! Why Don't We Use Data and Analytics Group on SAP Community instead of Technology?

Of course, it does not matter at the end. As long as if you tag your posts with SAP Managed tags everybody can find your posts. But there is a really one huge difference for usability that here in Interests Groups we can reply-the-reply but under Products and Technology we cannot! Check out this blog post SAP Datasphere is ready to take over the role of SAP BW by @kenneth_dalvik and all responses that are very all boring sequential --there is no reply-the-reply option! 🙃

So let's start using this group Here is already available Data and Analytics. The group will be definitely a perfect fit for SAP Analytics Cloud SAP Datasphere SAP Analytics Cloud, hybrid analytics SAP HANA Cloud, SAP HANA database SAP Analytics Cloud, connectivity SAP Analytics Cloud, analytics designer SAP S/4HANA Cloud, extended edition SAP S/4HANA Embedded Analytics SAP Business Planning and Consolidation, version for SAP NetWeaver SAP Business Warehouse Accelerator BW (SAP Business Warehouse) SAP BW/4HANA BW SAP BEx Analyzer BW SAP BEx Web SAP ASE - BW Enablement BW SAP HANA Data Warehousing SAP Data Services SAP Data Intelligence SAP Data Custodian SAP Big Data Services SAP BusinessObjects - Platform Administration SAP BusinessObjects - Semantic Layer SAP BusinessObjects Mobile SAP BusinessObjects - Semantic Layer - SDK SAP BusinessObjects - Web Intelligence (WebI)

These are welcome too SAP Master Data Governance SAP NetWeaver Master Data Management SAP Data Quality Management for SAP NetWeaver MDM

One thing I've noticed we cannot submit the blog posts directly. We need to "Submit for Review" only. You can still post discussions aka questions directly.

Thanks owners @ColinC, @meganhoy, @moshenaveh, @caroleighdeneen, @craigcmehil for their support!

Regards,
Tuncay Karaca