SAP Community - SAP HANA

Shaping the Future with Data and AI: SAP HANA Cloud Knowledge Graph Engine and Generative AI Toolkit

2025-04-04T19:35:42.714000+02:00

We’re pleased to announce the general availability of two groundbreaking technologies: the SAP HANA Cloud Knowledge Graph Engine and the Generative AI Toolkit. These solutions are designed to elevate your business intelligence and AI capabilities, offering enhanced data connectivity, smarter insights, and faster decision-making.

This blog post highlights the features of these new offerings and showcases how they align with SAP’s long-standing commitment to deliver innovations that drive your business forward. More importantly, it’s our way of saying: We’ve kept our promise.

SAP HANA Cloud Knowledge Graph Engine: A New Era of Data Connectivity

At SAP TechEd 2024, we introduced a major leap forward in how businesses can manage and navigate their data. The SAP HANA Cloud Knowledge Graph Engine empowers organizations to integrate, query, and leverage their data in entirely new ways by making connections and relationships between data points transparent. The new engine in SAP HANA Cloud enables you to fully harness your data's potential, driving smarter analytics, streamlined data management, and enhanced business results.

The Knowledge Graph Engine unlocks a deeper understanding of your data by capturing and analyzing complex relationships—something traditional databases struggle to do. This feature introduces a semantic layer on top of your data, enabling you to define, connect, and visualize entities across multiple sources. It’s built to handle and optimize data in the triple store format, an efficient method for representing complex, interconnected data, which enables better decision-making and more flexible data models.

Key Features:

Informed Decision-Making: Utilizing the Knowledge Graph Engine enables you to extract richer insights from your data, facilitating more accurate and well-informed decisions.
SQL and SPARQL Interoperability: The Knowledge Graph Engine will support SPARQL, the specialized query language for knowledge graph data, while tightly integrating it with SQL.
Enhanced Data Structure: With RDF and SPARQL's structured framework, your data models gain enhanced logical precision, leading to more dependable and accurate results.
Enhanced AI & Analytics: Provides structured context to AI-driven applications, improving decision-making, recommendations, and search accuracy.
Automated Reasoning & Inference: Uses built-in logic to infer new relationships from existing data, minimizing manual data maintenance as with the case of relational data models.

Whether you’re looking to enhance your data governance, improve the discoverability of insights, or develop smarter applications, the Knowledge Graph Engine will be a game-changer in how you think about and use data.

Generative AI Toolkit: Unlocking Innovation & Productivity

Alongside the Knowledge Graph Engine, we’re also launching the Generative AI Toolkit for SAP HANA Cloud. As the world of AI continues to evolve, our toolkit is designed to make AI more accessible and usable for our customers. The Generative AI Toolkit is a powerful solution designed to accelerate AI-driven development and data analysis. Users can for example, effortlessly query datasets using natural language and receive results through intuitive AI-driven interactions.

Key Features:

Open-Source Python Library & Integration: Leverage advanced generative AI capabilities through Python, enabling intuitive development and smooth integration for AI-driven tasks.
Natural Language-Driven Analysis & Automated Code Generation: Perform dataframe analysis with natural language queries and automatically generate code based on best practices and business scenarios code templates.
Retrieval Augmented Generation & LLMs Integration: Support for RAG with multiple vector stores and integration with large language models (LLMs) via the Generative AI Hub SDK for more accurate and context-aware AI solutions.
Unlocking SAP HANA Cloud’s Database AI Engine Capabilities: Utilizing targeted tools to generate machine learning models for scenarios like Time Series Forecasting on presented business data.
Streamlined Machine Learning Application Development: Accelerate the infusion of machine learning scenarios, like classification, regression or time series forecasting into CAP applications with agentic, natural language assistance. This allows for a much faster getting-started experience, simplifying the overall embedded AI application development process.

As organizations seek to empower their workforce with smarter, AI-powered tools, the Generative AI Toolkit offers the scalability and flexibility needed to drive innovation, improve decision-making, and accelerate time-to-value across the organization.

Keep the Promise

Both the Knowledge Graph Engine and the Generative AI Toolkit were born out of feedback from our customers and years of research into the challenges businesses face in a data-driven world. In our previous announcements and at past SAP TechEd events, we made bold commitments to revolutionize how organizations use data and AI. We’ve delivered on those promises with these two cutting-edge solutions.

In line with our dedication to providing continuous value, we’ve ensured that the Knowledge Graph Engine and Generative AI Toolkit are production-ready, offering you robust, scalable, and secure platforms to build your next-generation applications and analytics.

Early Adopter Care Program: Join us on the Journey

As we mark the general availability of these solutions, we also want to extend an invitation to our customers and partners to join our Early Adopter Care (EAC) Programs. These exclusive programs offer personalized support and guidance as you explore and implement the Knowledge Graph Engine or Elasticity in SAP HANA Cloud.

Being part of the EAC programs means you’ll get early access to the latest features, tools, and documentation, as well as direct engagement with SAP experts who can assist you in optimizing the use of these technologies for your specific needs. Additionally, you’ll be able to influence the direction of future updates and enhancements, ensuring that SAP HANA Cloud continues to meet the evolving demands of your business.

How to Get Involved?

Visit the EAC program pages to learn more about how to enroll:

By joining the EAC programs, you will be at the forefront of the next wave of innovation, helping us shape the future of data management and AI.

Looking Ahead

I’m incredibly excited about the opportunities these new capabilities bring to the table. From smarter data connectivity to AI-driven productivity, SAP HANA Cloud is setting the stage for a new era of innovation in data and AI.

We believe these advancements will not only help organizations thrive in an increasingly digital world, but also empower individuals across industries to make better, more informed decisions, faster. These technologies will be pivotal in shaping the future of how businesses interact with data, AI, and the world around them.

We’re committed to supporting your digital transformation journey every step of the way, and we can’t wait to see how you leverage these new capabilities to achieve greater success.

Thank you for being part of the SAP community. The journey has only just begun!

Three Reasons SAP BusinessObjects Still Matters in 2025

2025-04-07T17:00:00.028000+02:00

In a previous article, I answered the question "Is BusinessObjects Still Relevant in 2025?" with a resounding yes, giving a number of reasons why the platform still matters. In this article, I provide three reasons why the SAP BusinessObjects platform still matters in 2025.

Rich Semantic Layer

The first reason SAP BusinessObjects still matters in 2025 is its rich semantic layers. Whether we are talking about a classic BusinessObjects universe or the semantic layers provided by BW (SAP Business Warehouse),SAP HANA, or SAP Datasphere, they enable self-service analytics. These semantic layers need to be redesigned or fudged when building analytics with a non-SAP toolset. Competing semantic layers often lack the flexibility and power already provided by SAP BusinessObjects. The time and effort involved in reinventing the semantic layer should be carefully estimated before reaching for another analytics tool. And in the case of the universe, it provides a layer of isolation between the data model and the user. This isolation is why organizations can move their enterprise data warehouse from an on-premise platform to a cloud platform, carefully update the existing universe with zero disruption to the user community.

Scheduling and Publishing

The second reason SAP BusinessObjects still matters in 2025 is its scheduling and publishing capabilities. Information distribution is often an immature feature in "modern" analytics tools, whose vendors are quietly introducing "new" features like- wait for it- tables for users who prefer a "classic" BI experience. Of course, third parties are bringing extensions to fill these gaps. But getting the data into a table is only part of the solution - being able to burst the information in an efficient and personalized way is equally important.

Data Mode

The third reason SAP BusinessObjects still matters in 2025 is its new Data Mode. Introduced in BI 4.3 and enhanced in BI 2025, Data Mode brings self-service data blending and data cleansing to SAP BusinessObjects - Web Intelligence (WebI) users without introducing a separate product or vendor for that purpose. And Data Mode isn't simply a feature. It's a sign of SAP's renewed commitment to the SAP BusinessObjects platform. As witnessed in last month's release of SAP BusinessObjects Business Intelligence 2025, SAP has done far more than just slap a new About Box on an aging product. From Fiori to UI5 charting to Data Mode to cloud database support, the SAP BusinessObjects platform has been thoroughly modernized where it really counts

Conclusion

It's not wrong to desire to pick the best tool for the job, or to have multiple tools to choose from. But analytics leaders need to look honestly at some of the poorly designed, poorly performing deliverables that have been created on their watch with the "latest and greatest" tools. And reconsider the total cost of ownership and return on investment in choosing a familiar yet thoroughly modern solution that they are often tempted to overlook.

HANA Cloud’s VECTOR EMBEDDING in CAP and Comparison with OpenAI Embedding

2025-04-09T03:12:15.016000+02:00

Introduction

In the Q1 2024 release of SAP HANA Cloud, the REAL_VECTOR datatype was introduced to address the increasing demand for efficient storage and processing of high-dimensional vector embeddings. These embeddings are key in a variety of AI and machine learning applications, particularly in tasks such as semantic understanding and similarity search.

For developers using the SAP Cloud Application Programming (CAP) model, the @cap-js/hana package is the recommended tool for connecting to SAP HANA Cloud and utilizing its vector engine capabilities. In practice, we can define entities in our CDS data model that include elements of type cds.Vector. Once deployed, these elements are automatically mapped to the REAL_VECTOR datatype in the underlying SAP HANA table.

entity Books : managed {
    key ID        : Integer;
        title     : String(111) @mandatory;
        descr     : String(5000);
        embedding : Vector
}

SAP AI Core and the CAP LLM Plugin collaborate to bring advanced AI capabilities, focusing on large language models (LLMs) and vector embeddings, into CAP applications.

Custom handlers, written in NodeJS or Java, are used within CAP applications to extend and customize the framework's default behaviour. Specifically, custom handlers are utilized for tasks like embedding generation and other AI-driven operations.

getEmbedding = async (content) => {
    const vectorPlugin = await cds.connect.to("cap-llm-plugin");
    const embeddingModelConfig = await cds.env.requires["gen-ai-hub"][
      "text-embedding-ada-002"
    ];
    const embeddingGenResp = await vectorPlugin.getEmbeddingWithConfig(
      embeddingModelConfig,
      content
    );
    return embeddingGenResp?.data[0]?.embedding;
  };


this.after(["CREATE", "UPDATE"],"Books", async (res) => {
      const embedding = await this.getEmbedding(res.descr);
      await UPDATE(Books, res.ID).with({
        embedding: embedding,
      });
    })

Once the embeddings are stored in the REAL_VECTOR columns, they enable similarity searches, such as COSINE_SIMILARITY and L2DISTANCE. These searches allow for measuring the relationships between vectors, leading to more accurate data analysis and comparisons.

getSimilaritySearchForAlgorithm = async (searchWord, algoName) => {
    const vectorPlugin = await cds.connect.to("cap-llm-plugin");
    const embedding = await this.getEmbedding(searchWord);
    const entity = cds.entities["descr"];
    const similaritySearchResults = await vectorPlugin.similaritySearch(
      entity.toString().toUpperCase(),
      entity.elements["embedding"].name,
      entity.elements["descr"].name,
      embedding,
      algoName,
      5
    );
    return similaritySearchResults
};

The Q4 2024 release of SAP HANA Cloud introduced the VECTOR_EMBEDDING functionality, which allows for the generation of vector embeddings directly within the database. The available model is SAP_NEB.20240715, featuring a vector dimension of 768 and a token limit of 256. It supports the following languages: German (de), English (en), Spanish (es), French (fr), and Portuguese (pt). This is a significant advancement because, previously, embedding generation often relied on external services like SAP AI Core or CAP LLM Plugin.

In this blog post, I’ll show you how easy it is to use this new feature in CAP, enabling more efficient and seamless management of vector embeddings within the SAP HANA Cloud ecosystem.

Prerequisite

During the SAP HANA Cloud instance provisioning and configuration process, the option to enable advanced settings for Natural Language Processing (NLP) is available. This allows for the use of text embedding and text analysis models. This capability is also accessible with the SAP HANA Cloud trial.

Let's get Started.

Defining Entities with Vector Data Types

As mentioned above, we can define entities in our CDS data model that include elements of type cds.Vector and Once deployed, these elements are automatically mapped to the REAL_VECTOR datatype.

Calculated elements in the CAP enable you to define fields in your data model where the values are automatically derived from other elements or expressions and "On-write" (stored) calculated Elements are computed when the entity is created or updated, and their values are then stored in the database.

So, when defining entities with a vector data type, you can set up the embedding as a calculated field. This ensures that the embedding is computed and stored in the database whenever the corresponding field is created or updated.

entity Books : managed {
    key ID        : Integer;
        title     : String(111) @mandatory;
        descr     : String(5000);
        embedding : Vector = VECTOR_EMBEDDING(
            descr, 'DOCUMENT', 'SAP_NEB.20240715'
        ) stored;
}

To quickly check the HANA-specific SQL commands that would create your database tables based on your data model, use cds compile. It shows you the generated SQL DDL statements.

cds compile db/schema --to sql --dialect hana

By examining the SQL generated by cds compile, you can see how text embeddings are automatically generated during data input and updates. This process is similar to what is described in the Hana documentation (Reference: How to Automatically Create Vector Embeddings). Please note : while the generated columns approach is simpler, it does not support referencing NCLOB columns.

COLUMN TABLE cap_vector_embedding_Books (
  createdAt TIMESTAMP,
  createdBy NVARCHAR(255),
  modifiedAt TIMESTAMP,
  modifiedBy NVARCHAR(255),
  ID INTEGER NOT NULL,
  title NVARCHAR(111),
  descr NVARCHAR(5000),
  embedding REAL_VECTOR GENERATED ALWAYS AS (VECTOR_EMBEDDING(descr, 'DOCUMENT', 'SAP_NEB.20240715')),
  PRIMARY KEY(ID)
)

Define Entity to Perform Similarity Search

Exposing entities as views with parameters is a key feature of SAP CDS that enhances data modelling flexibility, reusability, and efficient data retrieval. This allows you to easily create an entity designed to perform similarity searches, such as COSINE_SIMILARITY and L2DISTANCE, to assess the relationships between vectors.

entity Search(query : String) as
        select from db.Books {
            ID,
            title,
            descr,
            :query as searchWord : String,
            cosine_similarity(
                embedding, to_real_vector(
                    vector_embedding(
                        :query, 'QUERY', 'SAP_NEB.20240715'
                    )
                )
            ) as cosine_similarity : String,
            l2distance(
                embedding, to_real_vector(
                    vector_embedding(
                        :query, 'QUERY', 'SAP_NEB.20240715'
                    )
                )
            ) as l2distance : String, 
        }
        order by
            cosine_similarity desc
        limit 5;

Similarly To quickly check the HANA-specific SQL commands that would create your database view based on your data model, use cds compile. It shows you the generated SQL DDL statements.

cds compile srv/service --to sql --dialect hana

By reviewing the generated SQL DDL statements for creating a VIEW, you can see how generated columns or triggers are used to automatically compute the cosine similarity and L2 distance between the embedding column and the parameterized query. This process is similar to what is described in the Hana documentation ( Reference : How to Write Queries)

VIEW EmbeddingStorageService_Search(IN query NVARCHAR(5000)) AS SELECT
  Books_0.ID,
  Books_0.title,
  Books_0.descr,
  :QUERY AS searchWord,
  cosine_similarity(Books_0.embedding, to_real_vector(vector_embedding(:QUERY, 'QUERY', 'SAP_NEB.20240715'))) AS cosine_similarity,
  l2distance(Books_0.embedding, to_real_vector(vector_embedding(:QUERY, 'QUERY', 'SAP_NEB.20240715'))) AS l2distance
FROM cap_vector_embedding_Books AS Books_0
ORDER BY cosine_similarity DESC
LIMIT 5

Finally, the service.cds looks like:

using {cap.vector.embedding as db} from '../db/schema';

service EmbeddingStorageService {
    
    entity Books                            as
        projection on db.Books
        excluding {
            embedding
        };

    entity Search(query : String) as
        select from db.Books {
            ID,
            title,
            descr,
            :query as query : String,
            cosine_similarity(
                embedding, to_real_vector(
                    vector_embedding(
                        :query, 'QUERY', 'SAP_NEB.20240715'
                    )
                )
            ) as cosine_similarity : String,
            l2distance(
                embedding, to_real_vector(
                    vector_embedding(
                        :query, 'QUERY', 'SAP_NEB.20240715'
                    )
                )
            ) as l2distance : String, 
        }
        order by
            cosine_similarity desc
        limit 5;

}

Test & Output:

You can use the REST Client extension in Visual Studio Code to execute HTTP requests define in .http files. For testing the CAP services you can create new file with the .http extension and copy past the following code

@server = http://localhost:4004/odata/v4/embedding-storage

### post books
#  post_book
POST {{server}}/Books
Content-Type: application/json

{
    "ID":252,
    "title":"Wuthering Heights",
    "descr":"Wuthering Heights, Emily Brontë's only novel, was published in 1847 under the pseudonym \"Ellis Bell\". It was written between October 1845 and June 1846. Wuthering Heights and Anne Brontë's Agnes Grey were accepted by publisher Thomas Newby before the success of their sister Charlotte's novel Jane Eyre. After Emily's death, Charlotte edited the manuscript of Wuthering Heights and arranged for the edited version to be published as a posthumous second edition in 1850."
}

### Similarity Search
#  similarity_search
@query = Catweazle British
GET {{server}}/Search(query='{{query}}')/Set

By using the first POST request, you'll be able to create a new record in the Books entity.

When a new book record is added, any calculated store column has its value determined by an associated expression. This expression can utilize other book properties, constants, and functions. The outcome of this calculation is immediately persisted in the store column of the book's database record.

In your case, it will evaluate the VECTOR_EMBEDDING on the description using the 'SAP_NEB.20240715' model and store the result directly in the Books table.

Following the same approach, adding further records will trigger the generation of embeddings for each new entry. For this blog post, we will utilize the Books.csv file provided in the SAP-samples/cloud-cap-samples repository.

To enable similarity searches, the entities are accessible via a view that accepts parameter. The output of this view includes the COSINE_SIMILARITY and L2DISTANCE scores, which indicate the degree of similarity between the input query and the Book description vectors.

We can observe that creating an entity with an embedding field and performing similarity searches by exposing it as a view is a straightforward approach that avoids the need for custom handlers. Nevertheless, the necessity of implementing custom handlers is entirely dictated by the specific demands of your project.

SOURCE CODE

Comparing HANA and Azure OpenAI Text Embeddings

As discussed earlier in this blog, before SAP HANA Cloud introduced its native VECTOR_EMBEDDING functionality, developers within the SAP ecosystem had to rely on external providers for text embeddings. A key provider in this space is Azure OpenAI, which offers the text-embedding-ada-002 model, widely used for its strong balance between performance and cost-effectiveness. Let’s compare the embeddings generated by HANA’s native VECTOR_EMBEDDING with those produced by Azure OpenAI’s text-embedding-ada-002 model.

Update Entities Definition

We'll add a new embedding element of type cds.Vector into the Books entity definition, which will be used to store embeddings from OpenAI’s text-embedding-ada-002 model.

entity Books : managed {
    key ID        : Integer;
        title     : String(111) @mandatory;
        descr     : String(5000);
        embedding : Vector = VECTOR_EMBEDDING(
            descr, 'DOCUMENT', 'SAP_NEB.20240715'
        ) stored;
        embedding_openai : Vector;
}

We will also update the entity to enable similarity search by adding two new elements to store the cosine similarity and L2 distance calculated using OpenAI.

entity Search(query : String) as
        select from db.Books {
            ID,
            title,
            CONCAT(SUBSTRING(descr,0, 50),'...') as description,
            :query as query : String,
            cosine_similarity(
                embedding, to_real_vector(
                    vector_embedding(
                        :query, 'QUERY', 'SAP_NEB.20240715'
                    )
                )
            ) as cosine_similarity : String,
            l2distance(
                embedding, to_real_vector(
                    vector_embedding(
                        :query, 'QUERY', 'SAP_NEB.20240715'
                    )
                )
            ) as l2distance : String,
            0.0 as cosine_similarity_openai : String,
            0.0 as l2distance_openai : String
        }
        order by
            cosine_similarity desc
        limit 5;

Add Custom Event Handler

Until now, we haven't created a handler for entities, as it wasn't necessary for the SAP HANA Cloud native VECTOR_EMBEDDING functionality. However, to retrieve the embedding from OpenAI and retrieve the results of a similarity search based on COSINE_SIMILARITY and L2DISTANCE algorithms, we now need to create.

module.exports = class EmbeddingStorageService extends cds.ApplicationService { init() {

  const { Books, Search } = cds.entities('EmbeddingStorageService')

  this.before (['CREATE', 'UPDATE'], Books, async (req) => {
    console.log('Before CREATE/UPDATE Books', req.data)
  })
 
  this.after ('READ', Search, async (search, req) => {
    console.log('After READ Search', search)
  })

  return super.init()
}}

Generating Embeddings with the CAP LLM Plugin

As mentioned earlier, the CAP LLM Plugin, is commonly used for generating embeddings. To generate a vector for the content, we need to connect to the cap-llm-plugin with the appropriate model configuration in order to retrieve the embedding.

getEmbedding = async (content) => {
  const vectorPlugin = await cds.connect.to("cap-llm-plugin");
  const embeddingModelConfig = await cds.env.requires["gen-ai-hub"][
    "text-embedding-ada-002"
  ];
  const embeddingGenResp = await vectorPlugin.getEmbeddingWithConfig(
    embeddingModelConfig,
    content
  );
  return embeddingGenResp?.data[0]?.embedding;
};

Performing Similarity Searches

The CAP LLM Plugin also furnishes APIs tailored for executing similarity searches based on the entity structure of the CAP model. The plugin supports the COSINE_SIMILARITY and L2DISTANCE algorithms and will continue to support future available algorithms too. The following code snippets highlight the steps involved in retrieving the results of a similarity search based on COSINE_SIMILARITY and L2DISTANCE algorithms.

getSimilaritySearch = async (searchWord) => {
  var [cosineSimilarities, l2Distances] = await Promise.all([
    this.getSimilaritySearchForAlgorithm(searchWord, "COSINE_SIMILARITY"),
    this.getSimilaritySearchForAlgorithm(searchWord, "L2DISTANCE"),
  ]);

  return cosineSimilarities.map((item, i) =>
    Object.assign({}, item, l2Distances[i])
  );
};

getSimilaritySearchForAlgorithm = async (searchWord, algoName) => {
  const vectorPlugin = await cds.connect.to("cap-llm-plugin");
  const embedding = await this.getEmbedding(searchWord);
  const entity = cds.entities["Books"];
  const similaritySearchResults = await vectorPlugin.similaritySearch(
    entity.toString().toUpperCase(),
    entity.elements["embedding_openai"].name,
    entity.elements["descr"].name,
    embedding,
    algoName,
    5
  );

  return similaritySearchResults.map(result => {
    return Object.assign({}, {
      [algoName.toLowerCase() + "_openai"]: result.SCORE,
      "ID": result.ID
    });
  })
};

Update Custom Event Handler

Finally, we can update the handlers using the following code snippet. Whenever a book is created or updated, it will generate an embedding and update the Book entity with the vector data and during a read operation using the search entity, it will perform both the similarity search and update the entity records accordingly.

this.after (['CREATE', 'UPDATE'], Books, async (req) => {
    const embedding = await this.getEmbedding(req.descr);
    await UPDATE(Books, req.ID).with({
      embedding_openai: embedding,
    });
  })
 
  this.after ('READ', Search, async (searches,req) => {
    const [{ query: query }] = req.params;

    const scores = await this.getSimilaritySearch(query);
    searches.map((search) => {
      const score = scores.find((score) => score.ID === search.ID);
      search.cosine_similarity_openai = score.cosine_similarity_openai;
      search.l2distance_openai = score.l2distance_openai;
    });
    return searches;
  })

Test & Output:

Now test it again, you can create new file with the .http extension and copy past the following code:

@server = http://localhost:4004/odata/v4/embedding-storage

### update books
#  update_book
PATCH {{server}}/Books(ID = 271)
Content-Type: application/json

{
    "title":"Catweazle",
     "descr":"Catweazle is a British fantasy television series, starring Geoffrey Bayldon in the title role, and created by Richard Carpenter for London Weekend Television. The first series, produced and directed by Quentin Lawrence, was screened in the UK on ITV in 1970. The second series, directed by David Reid and David Lane, was shown in 1971. Each series had thirteen episodes, most but not all written by Carpenter, who also published two books based on the scripts."
}

### Similarity Search
#  similarity_search
@query = describe me the British television series Catweazle
GET {{server}}/Search(query='{{query}}')/Set

In first call you will update an existing record in Books entity, by sending PATCH request to the specific record's endpoint. As we implemented a custom handler for the after UPDATE Event your custom logic perform after the book record has been updated in the database. In your case, it will generate an vector embedding from OpenAI embedding model and update the Book entity with the vector data.

To perform similarity searches leveraging the OpenAI embedding model, we've implemented a custom handler that executes after the Search entity is read in our service. This handler, triggered by accessing a parameterized view, calculates the COSINE_SIMILARITY and L2DISTANCE to determine the relationship between the search query and the book description vectors, subsequently modifying the results object.

SOURCE CODE

Analysis of the Model Comparison:

While cosine similarity focuses on the angle between vectors, and L2 distance focuses on the magnitude of the difference, they are related. If two vectors have a small angle between them (high cosine similarity), their difference vector will likely have a smaller magnitude (low L2 distance). Conversely, if the angle is large (low cosine similarity), the difference vector will likely have a larger magnitude (high L2 distance).

However, it's important to remember that this relationship isn't always perfectly strict. Factors like the dimensionality and scaling of the vector space can introduce some variations. The slight variations in the trends for the HANA embedding model indicate that while the inverse relationship may not be perfectly linear in this specific dataset, it still demonstrates a meaningful and insightful pattern.

Metric	Hana (Text Embedding)	OpenAI (Text Embedding)
Mean Cosine Similarity	0.464	0.7285
Std Dev Cosine Similarity	0.1732	0.0914
Mean L2 Distance	0.9673	0.7254
Std Dev L2 Distance	0.185	0.145

Above graph shows an snapshot of comparison of similarity metrics based on the dataset we have used. The key parameters compared are mean cosine similarity (0.7285 vs. 0.464), mean L2 distance (0.7254 vs. 0.9673) and standard deviations for both cosine similarity (0.0914 vs. 0.1732) and L2 distance (0.145 vs. 0.185). These are some of the parameters to analyze to check to decide on preference of embedding model. However, it's important to note that, without understanding the specific task and desired outcome of the models, it's not possible to determine which model is "better" based solely on this comparison of similarity metrics.

Conclusion:

Despite the overall performance, there might be some potential positive aspects or situations where Hana Embedding Model could be considered:

Potentially Simpler and Faster: If It has a significantly simpler architecture than OpenAI embedding model, it might be faster to train and generate embeddings. In resource-constrained environments or applications where speed is critical and a slight drop in accuracy is acceptable, Hana embedding model could be a viable option.
Potentially Lower Computational Cost: A simpler model often translates to lower computational cost for both training and inference. If your infrastructure has limitations, Hana embedding model might be more feasible to deploy and run at scale.
Potentially More Interpretable (Depending on the Model): If it is based on a more interpretable architecture, it might be easier to understand why it produces certain embeddings, which can be valuable for debugging and analysis.
Performance on Specific Data Subsets: it's possible that on a very specific subset of the data it might perform comparably or even slightly better than OpenAI. This would require a more granular analysis of the performance across different categories or types of text within your dataset. In this specific small dataset, this isn't evident, but it's a possibility in larger, more diverse datasets.

Reference & Further Reading

Selective Data Migration with Mix and Match Conversion Strategy to SAP S/4HANA

2025-04-10T16:52:28.809000+02:00

Why Convert to S/4HANA?

End of ECC Support

SAP will end mainstream support for ECC by 2027. This means no more updates, patches, or bug fixes, making it crucial to migrate to stay current and secure. Without regular updates and support, businesses using ECC will face increased security risks and potential operational disruptions.

Digital Transformation

S/4HANA supports modern digital business processes with advanced technologies like AI, machine learning, and IoT. These capabilities enable businesses to automate routine tasks, gain deeper insights from their data, and innovate more rapidly. By adopting S/4HANA, companies can stay ahead in the digital transformation journey.

Performance

The in-memory database of SAP HANA offers significantly faster data processing and real-time analytics. This enhanced performance allows businesses to make quicker, data-driven decisions, improve operational efficiency, and enhance customer experiences. The ability to process large volumes of data in real-time is a game-changer in today’s fast-paced business environment.

Competitive Edge

Migrating to S/4HANA ensures your business remains competitive by leveraging the latest innovations and capabilities. Staying on the cutting edge of technology provides organizations with the tools needed to outperform competitors, adapt to market changes swiftly, and meet evolving customer demands. Embracing S/4HANA is not just a technical upgrade; it’s a strategic move towards sustained growth and competitiveness.

What is the Conversion?

The conversion from SAP ECC to S/4HANA involves transitioning from the traditional ECC system to the next-generation S/4HANA system, which is built on the SAP HANA in-memory database. This process includes migrating data, adapting custom code, and configuring new functionalities.

Some Advantages of Converting to S/4HANA

Unlocking the Full Potential of Modern Business Operations

Improved Performance:
- Real-Time Analytics: The in-memory database architecture of SAP HANA enables real-time data processing and analytics
- Faster Transactions: Enhanced processing speeds lead to quicker transaction times and improved overall system performance.
Simplified Data Model:
- Reduced Complexity: S/4HANA's simplified data model eliminates redundant tables, reducing data footprint and maintenance.
- Integrated Analytics: Combines transactional and analytical data, providing a unified view of business operations.
Enhanced User Experience:
- SAP Fiori: Offers a modern, intuitive, and customizable user interface, improving user productivity and satisfaction
- Mobile Access: Enables access to business processes from any device, enhancing flexibility and responsiveness.
Advanced Capabilities:
- Embedded Analytics: Integrates advanced analytics and machine learning capabilities directly into business processes
- Future-Proofing: Ensures compatibility with future SAP innovations and updates, supported until at least 2040

Options to move to SAP S/4HANA

If you want to migrate to SAP S/4HANA on Premise you do have the following three options

System Conversion ("Brownfield-Approach")
New Implementation ("Greenfield-Approach")
Selective Data Transition to SAP S/4HANA

Comparison of time and Complexity Features

Feature	System Conversion (Brownfield)	New Implementation (Greenfield)	Selective Data Transition
Implementation Time	Shorter	Longer	Moderate
Business Process Optimization	Limited	Extensive	Customizable
Data Migration	Full	None	Selective
Disruption to Operations	Minimal	High	Moderate
Customization Retention	Yes	No	Selective
Suitable for Established Processes	Yes	No	Yes

System Conversion ("Brownfield-Approach")

In the Brownfield Approach, you convert your existing system to an SAP S/4HANA system using the Software Update Manager for an in-place conversion. If your current system is not yet on SAP HANA, you can combine this approach with a database migration to SAP HANA. Below is the highlighted process for the Brownfield approach

New Implementation ("Greenfield-Approach")

The Greenfield approach in SAP implementation involves starting from scratch with a new system, rather than upgrading or converting an existing one. This method allows for a fresh start, enabling the design and implementation of new processes without the constraints of legacy systems. Below is the highlighted process for the Greenfield approach

Selective Data Transition to SAP S/4HANA

The Selective Data Transition approach to SAP S/4HANA is a hybrid method that combines elements of both the Greenfield and Brownfield approaches. It allows organizations to selectively migrate data and processes from their existing SAP ERP systems to SAP S/4HANA. This approach is particularly useful for companies that want to retain certain historical data and customizations while redesigning other parts of their system. The Selective Data Transition to SAP S/4HANA comprises two sub-scenarios:

Shell Creation and Conversion
Mix and Match

High-Level Conversion Process lifecycle for Selective Data Transition (SDT) along with Shell System Conversion

Shell System Understanding

An SAP Shell system is a streamlined version of an SAP system that contains only the essential metadata, customizations, and repository data, excluding most application data and unnecessary clients. This approach is often used in scenarios where a full system copy is not required, such as for testing, development, or selective data transitions

Steps for Creating the Shell System

The Shell Creation process creates a new empty system based on a source system.
All client-independent objects (repository and client-independent customizing) will be identical to the source system. The new system will have the same support package level and add-ons.
All transaction and master data are stripped from the shell system as part of the SDT migration.
Only client 000 will exist in the new system.
The typical source system is the production system or a copy of it.
A shell system can be created using a remote client copy.

2. Shell Conversion:

SAP system will be converted to SAP S/4HANA using a standard system conversion with the Software Update Manager (SUM) or SUM with Database Migration Option (DMO).

3. S4HANA Goldan backup

As there is no application data (master data and transactional data) within this system, you can apply any necessary changes to the configuration of your new empty SAP S/4HANA system. For example:

You can adapt your financial model or controlling settings to meet future requirements.
Adopt the Fiori post-activation steps.
Implement add-ons as part of the post-conversion process.
Complete all post-conversion steps.
Apply all necessary post-functional and technical steps after the conversion process is completed over the shell system.

Note: Since we are saving this system as a golden copy, please ensure that all common post-conversion steps are completed before the golden backup. This will help reduce effort in future system implementations.

4. Target system build

The preserved golden backup taken in step (3) will be used to create the new target system for SAP S/4HANA

5. Data Migration

The next step will be data migration (step 5). For all areas where you didn't apply fundamental changes, it will be possible to migrate historical data and transactional and modify them on the fly to fit the new SAP S/4HANA structures. Data for these areas will be migrated at the table level, and the necessary on-the-fly changes for SAP S/4HANA will be executed using special afterburner programs. These programs are like those used during a standard system conversion to SAP S/4HANA

SAP has partners that provide features for slicing data based on various factors and tools for migrating data with data slices. Data slicing is entirely dependent on business needs and the data migration approach, utilizing the available tools for migration and slicing

SDT (Some data Slice criteria): Time-Slice Scenario, Selective Company Code Transfer, Slice on FI and Other related selections, ETC.
SDT Tools for migration/partner companies: SAP DMLT, SNP, CBS, Natuvion, ETC.

Mix and Match

The basic idea of the mix and match scenario is to start with a greenfield approach while retaining your old solution for specific areas where your current SAP ERP solution meets your future demands.

To retain parts of your current solution, you will first identify and stabilize these areas. (Remember: even in Agile projects, one of the biggest reasons for budget overruns or project failures is unclear or continuously expanding scope).

After identifying the solution areas, you want to bring over to your new SAP S/4HANA system, you need to decide how to perform this migration

There are two possibilities:

You manually add the configuration to the new SAP S/4HANA solution.
You migrate/transport the solution using some tooling

Option 1: Manual Configuration

You will need experienced consultants to manually configure the solution in the target system.

Option 2: Migration/Transport Using Tooling

Process a): If there are no or minor changes between SAP ERP and SAP S/4HANA, you can transport the customizing directly from SAP ERP to SAP S/4HANA and manually rework the customizing in the target system.
Process b): If there are significant changes between SAP ERP and SAP S/4HANA, you can perform a Shell Creation of your SAP ERP system and convert this to SAP S/4HANA. This provides a basis from which you can transport the solution to your new SAP S/4HANA system. In this case, you will also manually rework and fit the transported elements within your SAP S/4HANA system.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Use case for (Mix and Match Approach) Option 2: Migration/Transport Using Tooling “Process b”

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Landscape conversion Approach (MIX and Match) (ECC to S4HANA Conversion with SDT)

Use case highlights as below

Best example of the Mix and Match approach.
Selective data migration conversion approach.
We created the Shell system and converted it to the S/4HANA system.
Performed the post-conversion steps and then took the golden backup to prepare the subsequent systems.
For the selective data migration, we used the DMLT Tool from SAP.
As per the Mix and Match approach, current ECC changes and processes were aligned with manual retrofit changes moved to the S/4HANA system.
According to the project plan, along with testing phases like SIT, UAT, and Dress Rehearsal, we converted the ECC to the S/4HANA system using the Selective Data Migration approach.

DMLT Tool information:

The SAP Data Management and Landscape Transformation (DMLT) tool is designed to help organizations manage and transform their data landscapes efficiently. Here are some key aspects of the SAP DMLT tool:

Data Migration: DMLT facilitates the smooth transition to SAP S/4HANA by offering selective data transition, new implementation, and system conversion options
System Landscape Optimization: The tool supports mergers, acquisitions, divestitures, and complex organizational restructurings by transforming SAP data and system landscapes
Data Unification and Harmonization: DMLT helps in analyzing and cleaning incorrect data from customers, partners, and products. It unifies and standardizes financial and controlling processes, improving data quality and governance
System Consolidation: By reviewing and consolidating system landscapes, DMLT reduces the cost of operating distributed systems and facilitates better data management and reporting
Holistic Approach: DMLT offers a comprehensive approach to data-related needs, including data quality, data integration, and information lifecycle management

This ensures a successful migration using proven technologies and methodologies. Overall, SAP DMLT is a powerful tool that helps organizations optimize their data management processes and transition smoothly to advanced IT landscapes

Control system Role in the DMLT tool:

A control system is a system that is already on the S/4HANA target version and is used for the controlled logic program, which provides the selective data migration decision from the source to the target system

Technical Challenges with DMLT Tool during conversion process:

During the shell creation, SAP generally does not include third-party components and Y and Z custom tables. Therefore, we must provide the list of third-party add-ons to SAP to include in the shell creation.
Technical tables are loaded only once during the shell creation. After the shell creation, deltas should be managed by other tools or manually.
- Examples of technical data: RFC, LSMW, WE21, WE20, printers, certificates, jobs, job variants, user master data. All delta data should be managed outside the DMLT process.
All Y and Z tables and t-codes must be checked in the shell system along with all entries.
We must align with SAP on how the Y and Z table data will be managed during the DMLT process (e.g., including tables in the DMLT process or through a retrofit process).
SAP must provide the DMLT tool transport, which we need to align. However, during the tool setup, we may encounter some errors in the SAP system.
In this approach, you might have to redefine all interface connections along with AL11 and scripts, if any

Conclusion

This blog provides an understanding of the SAP S/4HANA conversion process methods available in the SAP ecosystem. It primarily focuses on the Selective Data Migration approach, which is part of the Mix and Match and Shell Conversion ap

Flip Virtual Tables in Calculation Views from ECC to S/4HANA

2025-04-10T16:54:07.764000+02:00

As SAP prepares to end support for ECC by 2027, many companies are making the crucial transition to S/4HANA. This shift promises enhanced capabilities and performance, but it also brings its own set of challenges. One of the most significant hurdles involves the calculation views that rely on virtual tables from the ECC system. After migration, these views need to be redirected to the new S/4 system.

For many organizations, BW hybrid modeling is a common practice. This approach uses virtual tables from the ECC source system within their calculation views. With numerous calculation views depending on these ECC virtual tables, the need to point them to the new S/4 system without disrupting functionality becomes paramount.

SAP Note 3103359 clearly states that there is no straightforward way to change the adapter and remote source database without dropping and recreating the remote source. This raises a critical question: **Do we need to adjust all the calculation views?**

Adjusting all calculation views is a daunting task. It requires extensive manual effort and rigorous testing, which inevitably adds to the project's cost. The prospect of manually updating each view can be overwhelming for many organizations.

However, there is a workaround that can simplify this process. By deleting the virtual table and recreating it within the same schema, organizations can effectively redirect their calculation views to the new S/4 system.

Example: Redirecting Virtual Tables in HANA System

Let's take a closer look at a practical example. Below, we see the HANA system with the "ECC" schema. There are two virtual tables, "V_EKKN" and "V_EKKO" which refer to the ECC tables "EKKN" and "EKKO".

Now, we want to point these tables to the new S/4 system. We can achieve this by following these steps:

Delete the virtual tables.
Recreate the virtual tables.

These steps can be performed by running the following script:

DROP TABLE <schema_name>."<virtual_table_name>" CASCADE;
CREATE VIRTUAL TABLE "<schema_name>"."<virtual_table_name>" at "<remote_source>"."<database_name>"."<remote_schema_name>"."<remote_table_name>";

DROP TABLE ECC."V_EKKN" CASCADE;
CREATE VIRTUAL TABLE "ECC"."V_EKKN" at "S4X"."S4DB"."SAPPROD"."EKKO";

DROP TABLE ECC."V_EKKO" CASCADE;
CREATE VIRTUAL TABLE "ECC"."V_EKKO" at "S4X"."S4DB"."SAPPROD"."EKKO";

By executing this script, we can see that the virtual tables are now pointing to the new S/4 system "S4X."

By deleting the virtual table and recreating it within the same schema, organizations can seamlessly redirect their calculation views to the new S/4 system without needing to adjust the calculation view itself. This method, while still requiring careful execution, can significantly reduce the complexity and effort involved.

Reference links:
SAP Help portal for Managing Virtual Tables
SAP Note 2542963 - Is it possible to rename a Remote Source in SAP HANA?

SAP Developer News April 10th, 2025

2025-04-10T21:10:00.041000+02:00

DESCRIPTION

Podcast: https://podcast.opensap.info/sap-developers/2025/04/10/sap-developer-news-april-10th2025/

Shaping the Future of Data and AI with SAP HANA Cloud:

Shaping the Future with Data and AI: SAP HANA Cloud Knowledge Graph Engine and Generative AI Toolkit: https://community.sap.com/t5/blogs/blogarticleprintpage/blog-id/technology-blog-sap/article-id/180448

Generative AI Toolkit for SAP HANA Cloud: https://github.com/SAP/generative-ai-toolkit-for-sap-hana-cloud/blob/main/README.md

LangChain integration for SAP HANA Cloud: https://pypi.org/project/langchain-hana/

Invitation to join HANA Tech Con: https://www.linkedin.com/feed/update/urn:li:activity:7316037624064819201/

SAP Open Source Webinar Apeiro Reference Architecture

Registration page: https://events.sap.com/apeiro-reference-architecture/en/registration.aspx

LinkedIn post: https://www.linkedin.com/posts/michael-ameling_neonephos-gardener-gardenlinux-activity-7312819875003985923-D7wZ/?utm_source=share&utm_medium=member_desktop&rcm=ACoAACgkg4UBF6JXngL8eiZlhbT4UQWinLeQkAo

ApeiroRA: https://apeirora.eu/

NeoNephos Foundation: https://neonephos.org/

CAP March 8.9 Release

Release Notes for March 8.9: https://cap.cloud.sap/docs/releases/mar25

Major Release migration Guide: https://cap.cloud.sap/docs/releases/mar25#prepare-for-major-release

Free SAP Certification and practice systems for students and lecturers:

- Blog post for student learning hub: https://community.sap.com/t5/beginner-corner-blog-posts/free-sap-certification-and-practice-systems-for-students-amp-lecturers/ba-p/14052493

- What is SAP: https://community.sap.com/t5/beginner-corner-blog-posts/how-and-where-do-you-start-a-consulting-career-in-sap/ba-p/221488

Welcome to SAP BusinessObjects BI 2025

Welcome to SAP BusinessObjects BI 2025: https://community.sap.com/t5/technology-blogs-by-sap/welcome-to-sap-businessobjects-bi-2025/ba-p/14040695

CHAPTER TITLES

0:00 Intro

00:10 Shaping the Future of Data and AI with SAP HANA Cloud

1:55 SAP Open Source Webinar Apeiro Reference Architecture

3:13 CAP March 8.9 Release

4:55 Free SAP Certification and practice systems for students and lecturers

7:50 Welcome to SAP BusinessObjects BI 2025

TRANSCRIPTION

[Into] This is the SAP Developer News for April 10th, 2025.

[Witalij] We are delighted to inform you that two cool technologies, the Knowledge Graph Engine and the Generative AI Toolkit, have been launched in SAP HANA Cloud. The Knowledge Graph Engine enables you to model and analyze complex relationships within data stored in SAP HANA Cloud. Native RDF triple store support enables semantic data storage and querying using the resource description framework. It also enhances LLM responses by grounding them in business -specific knowledge stored within the knowledge graph, providing more accurate and contextually relevant outputs. A lot of smart words, I know, so you better try it yourself. I plan to make the new SAP CodeJam exercises available shortly. Let me know in the comments if you are interested. The Generative AI Toolkit simplifies the development of intelligent data applications by providing a natural language interface for creating in-database machine learning models, reducing the need for extensive coding from your cert. Although, following recent updates in the long-chain review process, our engineering team has published a new long-chain integration for SAP HANA Cloud package. You can now easily leverage features like in database embeddings, full-text search, and vector indexes directly within LongChain. Interested in discussing this and other aspects of SAP HANA? The very first HANA TechCon will take place at SAP Headquarter in Germany on July 10th. Please check the published agenda and register if you are interested. I'm looking forward to meeting you there.

[Ajay] Hello everyone, I have got some interesting news for the open source enthusiast. There is an upcoming SAP Open Source webinar on the topic Apeiro Reference Architecture, Strengthening Digital Sovereignty for Europe. It is on May 8th at 3 p.m. CESD. The Apeiro Reference Architecture is a part of initiative and aim to build a high-performance cloud edge infrastructure with open source standards. Peter Giese, head of SAP Open Source Program Office, will give you an overview on why and how this project has been transferred into the new NeoNephos Foundation, which will serve as a neutral entity dedicated to fostering, open collaboration, and governance. There is also a LinkedIn post by Michael Ameling explaining the NeoNephos Foundation, where several flagship open source projects such as Gardena, Garden Linux, and more are transferred to NeoNephos through this initiative. Vasu Chandrasekhara, Chief Product Owner of Apeiro, will explain the landscape of Apeiro reference architecture with an overview of the architecture and the components. Please find the registration links to the webinar in the description below.

[Kevin] A March 2025 release of the SAP Cloud Application Programming Model is out. The new version 8.9 brings important changes to you. But before we dive in, I just wanted to remind you that there is a new CDS parser version coming out with the next major version 9.0. So make sure to comply to all the new changes and that your projects keep on running. You can use the new parser now by setting the new parser flag to true to try them out in your project today. So you might as well go ahead and do it. A new change with 8.9 brings you actions to generated child entities when defining a composition of aspect. This is really nice bringing you action capabilities when reflecting document structures in your domain models. Recursive hierarchies and Fiori Tree Table support is out now available in beta, allowing you to serve read requests for the SAP Fiori Tree Table control. Version 8.9 is introducing a new CDS command, which is cds-up. CDS-up will build and deploy your CAP applications to Cloud Foundry. And if you need to deploy to Kyma, simply add the argument dash dash to Kubernetes to the end of the command. And the last thing I want to mention is that you can now use the cds deploy to HANA command for deploying to Kubernetes as well. So just add the argument of dash dash on Kubernetes this at the end of the command for the magic to begin. And there are more changes in the current release, which we aren't covering here. So make sure to read the official release notes on the CAP documentation page. And as always, the link is in the description below.

[Ebere] Hi, everyone. I've got some exciting news to share, especially if you’re a student just like me, working with SAP, or just curious to know what SAP is all about, the Learning Hub Student Edition is live. And the best part of it, it is free for all actively enrolled students. And here's what you can get. Full access to content available to customers, two certifications attempts, access to live expert sessions, hands-on cloud-based training systems, plus content to help you get certified over time. If you’re wondering, what is SAP? Don't worry, I got it covered. There is a blog post link below. Definitely give that a read if you are new to SAP world. And how do you get started? Let me break it down real quick.Create your SAP universal ID. This is like your learning passport, so make sure you use your email address of your university to do so. Verify that you are a student using the email address attached or tied with your enrollment. And while waiting for the activation link to get to your inbox, you can already start checking out the student's Zone and exploring courses and even start learning. You can also activate your subscription by checking your inbox to activate the link. If you don’t see it on your inbox, check the spam folder just in case. and you can also start you can only start learning and pick up from where you left or choose new topics from your student's zone account also you can also start practicing on the cloud-based training systems this is really really important because here you actually switch or you turn your theories into real skills and also you can also join live sections and ask expert questions and also learn what is happening in the field. When you think you're done or when you think you've learned enough, you can already book from the Learning Hub certification exams. Remember, you've got two free attempts. Before I go, just a few quick FAQs. Old student email address won't get you access, so make sure you are actively enrolled. And the subscription lasts for 12 months, But, if you're still enrolled, you can renew it. And remember, the certification exam is only for two attempts if only you pass on the first try. So, you're still unsure where to start? I got it covered. You can check out the Discovery SAP for students or position in SAP business suits. And remember, if you ever get stuck, the SAP community is always there for you. For more information, please check out the blog post. Happy learning and goodluck on your SAP journey. Ciao.

[Witalij] Because of all the troubles, I'm misinforming you that SAP Business Objects BI 2025 is now available. But thanks, Josh, for reminding me. SAP Business Objects BI 2025 brings a fresh set of enhancements that make your business analytics more efficient, user-friendly, and future-proof. BI 2025 offers powerful enhancements designed to help you help your business users make faster and more informed decisions. This release isn't just about the new features. It's about ensuring that your BI platform remains secure, flexible, and ready for the future. Please check the Welcome to SAP Business Objects BI 2025 blog post and share your thoughts in the comments there. And by the way, Greg, Eric, yes, I still have it.

Understanding & Tips and Tricks for CDC ( Change Data Capture)

2025-04-15T07:08:58.081000+02:00

What is Change Data Capture (CDC)?

Change Data Capture (CDC) is a design pattern that tracks changes (inserts, updates, deletes) in a database and makes those changes available to downstream systems in real-time or near-real-time.

How CDC Works

Detects row-level changes in source databases
Processes these changes into structured events
Delivers them to data warehouses, streams, or services

Best way to think of CDC (Change Data Capture) on your data coming from Source to Target

Live Playground to understand Change Data Capture (CDC) Playground

https://www.change-data-capture.com

Tips and Tricks for CDC ( Change Data Capture)

Use Time Stamps for Efficient Data Extraction:
- Ensure your source tables have update and create time stamps. This allows you to efficiently track changes and extract only the modified rows.
Leverage Change Logs:
- Utilize the change logs maintained by your RDBMS to capture detailed audit trails of data modifications. This can help in identifying changes more accurately.
Optimize Performance with Incremental Extraction:
- Implement source-based CDC to improve performance by extracting only the changed rows, rather than the entire dataset.
Set Up Proper Indexing:
- Ensure that your tables are properly indexed on the columns used for CDC. This can significantly speed up the data extraction process.
Use SAP Operational Data Provisioning (ODP):
- For integration with Azure Data Factory, use the SAP ODP framework to replicate delta changes efficiently.
Monitor and Tune CDC Processes:
- Regularly monitor the performance of your CDC processes and tune them as necessary to ensure optimal performance.

Benefits of SAP HANA CDC

Improved Performance:
- By capturing only the changes, CDC reduces the amount of data that needs to be processed, leading to faster data integration and reduced load on the source systems.
Real-Time Data Integration:
- CDC enables near real-time data integration, ensuring that your data warehouse or analytics systems are always up-to-date with the latest changes.
Reduced Data Latency:
- With CDC, data latency is minimized as changes are captured and propagated almost immediately, which is crucial for real-time analytics and reporting.
Cost Efficiency:
- By processing only the changed data, CDC reduces the computational and storage costs associated with full data loads.
Enhanced Data Accuracy:
- CDC ensures that only the most recent and relevant data is captured, improving the accuracy and reliability of your data.

Implementing these tips and leveraging the benefits of CDC can significantly enhance your data management and integration processes in your database like SAP HANA. If you have any specific questions or need further details, feel free to ask!

Overview and architecture of the SAP CDC capabilities - Azure Data Factory | Microsoft Learn

Learning Journey - Using Source-Based Changed Data Capture (CDC)

Learning Journey - Using Target-Based Changed Data Capture (CDC)

Exploring New SQL on Files Use Cases with SAP HANA Cloud QRC 04/2024 and QRC 01/2025

2025-04-15T10:42:07.034000+02:00

Related Blogs:

Unlocking the True Potential of Data in Files with SAP HANA Database SQL on Files in SAP HANA Cloud

As I mentioned in my blog last year, the SQL on Files capability in SAP HANA Cloud has unlocked new potential by providing direct read-only SQL access to files stored in SAP HANA Cloud, data lake Files since SAP HANA Cloud QRC 03/2024.

Starting with SAP HANA Cloud QRC 04/2024, this capability has been further extended to support direct access to the most common external object storages listed below, in addition to SAP HANA Cloud, data lake Files.

Amazon S3
Azure Blob Storage or Azure Data Lake Storage (ADLS) Gen2
Google Cloud Storage (GCS)

Although the method for creating a remote source varies by cloud service provider, the process for creating virtual tables, the supported file formats, and the read-only access remain the same as when using SQL on Files with SAP HANA Cloud, data lake Files. Since these aspects are already explained in my previous blog, let me focus on the differences in this one.

Additionally, as you may have already noticed from the above diagram, read-only access to external Delta Sharing is also supported from SAP HANA Cloud QRC 04/2024 with a dedicated Delta Sharing adapter.

And when it comes to CSV formats, two new options are now included to provide better flexibility and reduce the effort required when reading files in CSV format, starting with SAP HANA Cloud QRC 04/2024.

Another piece of good news is that the existing limitation on Delta tables, which currently supports only reader version 1 for both direct file access and Delta sharing, has been lifted starting with SAP HANA Cloud QRC 01/2025.

Alright, with these points in mind, let's dive deeper into the details with some examples.

Accessing external object storages

As highlighted, starting from SAP HANA Cloud QRC 04/2024, the SAP HANA Database SQL on Files feature has begun to support direct read-only access to files (CSV, Parquet, and Delta tables) stored in external object storage.

The supported external object storage options are, as mentioned above, Amazon S3, Azure Blob Storage or Azure Data Lake Storage (ADLS) Gen2, and Google Cloud Storage (GCS).

For Amazon S3 and GCS, both path-style and virtual-host-style endpoints are supported (GCS also supports path-style regional endpoints). For Azure, both standard and DNS endpoints are supported.

The methods for creating a remote source differ according to the specific standards and integrations established by each cloud service provider. The example below shows how to create a remote source to Amazon S3.

-- create a remote source to Amazon S3
CREATE REMOTE SOURCE <remote_source_name> ADAPTER "file" CONFIGURATION '
provider=s3;
endpoint=<s3_endpoint>;'
WITH CREDENTIAL TYPE 'PASSWORD' USING 'user=<access_key>;password=<secret_key>;

As described, the file adapter is still in use but now with the parameter provider=s3.

A remote source can also be created via the SAP HANA Database Explorer UI, and the example below shows how to create a remote source to Google Cloud Storage via the UI.

Please refer to the links below for further details on creating a remote source for each external object storage.

SAP HANA Cloud, SAP HANA Database SQL on Files Guide: Create an Amazon S3 Remote Source
SAP HANA Cloud, SAP HANA Database SQL on Files Guide: Create a Microsoft Azure Storage Remote Source
SAP HANA Cloud, SAP HANA Database SQL on Files Guide: Create a Google Cloud Storage Remote Source

Although the process for creating a remote source differs, as mentioned above, the method to create a virtual table is the same as with SAP HANA Cloud, data lake Files and can be found in my previous blog.

Accessing external Delta Sharing

Also, starting from SAP HANA Cloud QRC 04/2024, SAP HANA Database SQL on Files has started to support external Delta Sharing.

This means you can create a remote source to Databricks Delta Sharing and also create read-only virtual tables by pointing to the tables exposed by Delta Sharing protocol.

-- create a remote source to Delta Sharing
CREATE REMOTE SOURCE <remote_source_name> ADAPTER "deltasharing" CONFIGURATION '
provider=databricks;
endpoint=<endpoint in config.share>;'
WITH CREDENTIAL TYPE 'OAUTH' USING 'access_token=<bearerToken in config.share>';

As described, a dedicated deltasharing adapter is used with the parameter provider=databricks, instead of the file adapter.

Similar to external object storage access, the remote source to Databricks Delta Sharing can also be created via the SAP HANA Database Explorer UI.

For this read-only access, Delta tables must be added to the Delta Sharing of Databricks. By creating a new recipient with a token and token lifetime, the credential (config.share) should be downloaded. Please also note that, unlike direct file access in SQL on Files, the method of creating virtual tables follows the same approach as with other SAP HANA smart data access (a.k.a., SDA) adapters.

-- create a virtual table
CREATE VIRTUAL TABLE VT_TAB AT <remote_source_name>.<share_name>.<schema_name>.<table_name>;

Although Delta tables in files can be directly accessed with the file adapter, support for the Delta Sharing protocol via a dedicated deltasharing adapter can offer more flexibility if Delta Sharing is already in use and there is a requirement to use it as a unified access protocol within your organization.

Please also refer to the link below for further details.

SAP HANA Cloud, SAP HANA Database SQL on Files Guide: Create a Databricks Delta Sharing Remote Source

Two new options for CSV format

For better usability and reduced effort, two new options were introduced with SAP HANA Cloud QRC 04/2024.
One is SKIP FIRST N ROWS, which literally skips the first N number of rows when creating a virtual table by pointing to CSV files. Obviously, if the number of rows is less than N, an error will be thrown.

-- skip the first 5 rows
CREATE VIRTUAL TABLE TEST_SCHEMA.TAB1 (
    A INT,
    B INT
    ) AT TEST_REMOTE."/path/to/csv/files1"
  AS CSV SKIP FIRST 5 ROW;

This is useful if there are multiple header rows, or if you intentionally ignore certain rows for any business reasons.

Another option is COLUMN LIST IN FIRST ROW, which is similar to the above option but can be used to more precisely recognize the first row as the header and ensure the match between virtual table field names and the field names in the file header.

-- recognize the first row as the header
CREATE VIRTUAL TABLE TEST_SCHEMA.TAB2 (
    A INT,
    B INT
    ) AT TEST_REMOTE."/path/to/csv/files2"
  AS CSV COLUMN LIST IN FIRST ROW;

In this case, since the case sensitivity of field names is also compared, if the field name of the virtual table does not match the field name in the file header, an error is thrown.

In addition, the VIRTUAL_TABLE_FILES system view has been extended to include these two new configuration values. Please also refer to the link below for further details.

SAP HANA Cloud, SAP HANA Database SQL on Files Guide: Create a Virtual Table

Latest reader version support for Delta table

As briefly mentioned above, before QRC 01/2025, if the reader version of the Delta table was higher than 1, a virtual table could not be created. In most use cases, supporting reader version 1 was sufficient because, when converting Parquet files into Delta tables, the lowest possible reader version is generally chosen.

Nevertheless, if a specific data type like TimestampNtz (Timestamp without timezone) is included in your data or specific Delta table features like the deletion vector are enabled, a higher reader version like 3 is required, and SQL on Files could not read these before.

However, starting with SAP HANA Cloud QRC 01/2025, SQL on Files has begun supporting reader versions 2 and 3, meaning the existing limitation has now been fully lifted.

Conclusion

In conclusion, the enhanced capabilities of SAP HANA Database SQL on Files in SAP HANA Cloud provide more flexible and efficient ways to manage your data. With the ability to access vast volumes of data from well-managed object storages, you can fully leverage the robust features of the SAP HANA Cloud as a platform. This includes in-memory processing, elasticity, the Predictive Analysis Library (PAL), machine learning capabilities, and more.

Furthermore, these advancements facilitate seamless data integration with SAP applications like S/4HANA, empowering you to derive even greater value by combining your object storage data with data from SAP applications. If you're interested in reading data from SAP applications like S/4HANA in SAP HANA Cloud, please refer to my blog, Taking Data Federation to the Next Level: Accessing Remote ABAP CDS View Entities in SAP HANA Cloud.

As we remain committed to innovation, stay tuned for upcoming updates that will continue to expand and enrich your SAP HANA Cloud experience.

Terminate Long-Logged-In Users via ABAP Program

2025-04-16T10:37:47.230000+02:00

How to Automatically Terminate Long-Logged-In Users via ABAP Program

Introduction:

In this blog post, we'll explore how you can implement an ABAP solution to scan user sessions and terminate those that have been active for more than one hour. Although SM04 transaction provides the option to log off the users manually, doing this automatically through ABAP programming adds efficiency.

Step1: To fetch the data from T-Code (SM04) which show the details of user and their systems through submit of standard program (RSM04000_ALV_NEW). CL_SALV_BS_RUNTIME_INFO=>SET is used to grab ALV output.

Note: If the class-method is not called, u can’t fetch the data through submit program.

Step 2: Retrieve the data using CL_SALV_BS_RUNTIME_INFO=>GET_DATA_REF and move to internal table as mentioned in the below the image.

Declaration of internal table with limited fields which is used in perform.

Step 3: Get the user log on time from the table(USR02)

Step 4: Loop the data which is fetched from SM04 and read the record of user from USR01. Calculate the system time with logon time. Check the time which is more than an hour and perform the delete_user which can terminate the users who have been logged in more than an hour.

Log off the users through SM04:

Step 1: Go to transaction SM04.

Data of users :

Step 2: Select the user which need to log off and follow the same as image shown.

Step3: If u want to log off the user, click YES.

This is manual process to log off the users.

Conclusion.

By developing a custom ABAP report, we can identify user sessions that have been active for more than one hour and terminate them automatically. To make this solution fully automated, the ABAP report can be scheduled as a background job to run at regular intervals—for example, once every hour. This scheduled job will routinely scan for long-running sessions and take appropriate action without any manual oversight, making it a proactive approach to session management in SAP.

Evolution of Pricing in SAP

2025-04-16T11:29:09.793000+02:00

In this blog we will discuss about pricing concept in SAP, Evolution from ECC to S/4HANA with key benefits.

ECC 16 pricing fields in detail are as follow:

Steps: Place the condition in sequence.
Counter: If there is no space between two steps to add one more condition type.
Condition type: Specifies the type of the pricing element.
Description: To describe the condition types in sales document pricing.
From: Help to determine the base value for calculating the condition type value in sales document pricing.
To: Used to cumulate the value of multiple steps that are in a sequence.
Manual: If we check manual, then the system will not determine the condition type automatically into the sales document.
Requirement: If we check required, then system will not allow to save the sales document if the condition type is missing
Statistical: If we check statistical then it will have two effects, value doesn’t have any effect on net value & not be posted into accounting.
Print: Controls whether to print the condition type into output printout or not.
Subtotal: Used to store the value of condition type in some temporary tables and fields for the purpose of further calculations.
Requirements: The system will check while determining condition type into sales document, if condition fulfilling then condition type determine into sales document.
Alternate calculation type (Formula): Use alternate calculation type, if the calculation part of condition type is not standard.
Alternate Base type (Formula): Use alternate base type when the base value of the condition type is not standard.
Account Key: It is a parameter to determine revenue G/L account while posting invoice values into accounting.
Accrual Key: Keeping some amount aside from each transaction into provisional account to meet the further requirements of rebate settlement.

Now SAP S/4HANA pricing procedure has expanded from 16 pricing field to 18 pricing field are follow:

1. Statistical and Relevant for Account Determination:

Statistical: It does not impact the net price but is recorded for reporting purposes.

Relevant for Account Determination: It still flows into financial accounting, even though it’s statistical. We can use this field indicator to define that the statistical price condition is posted to account-based Profitability Analysis (CO-PA) as journal entry to an extension ledger of Financial Accounting.

Example: Imagine you are delivering the free delivery above 200 rupees, previously this delivery cost maybe be ignored in financial posting because it has marked as Statistic.

Now, you can post it to an extension ledger in CO-PA, giving you a clear picture of how much free delivery impacts your bottom line.

Scenario: In this scenario Tax will display and post, but no impact on Net price:

Condition	Statistical	Account Relevant	Impact on Net Price	G/L Posting	Use Case
ZXYZ -Tax	✅ Yes	✅ Yes	❌ No	✅ Yes	Legal tax display + accounting

This shows the Tax ZXYX on invoice, it will be posted to a special liability GL account and will not impact the product net price (i.e., customer doesn’t pay directly).

In system:

✅ Visible on sales order and invoice as a separate line

✅ Posted to G/L account (e.g., Environmental Provisions) using Account Key ZEV

❌ Does not affect the net selling price (because it’s statistical)

2. Access level (New in 2023): Access levels is used to control the access of your business users to price elements in business documents.

Access levels consist of a four-digit number and a description, you can use all numbers from 0001 to 9999.

Default value 0000 grants full access to all users.

For example, the profit margin in a sales order can be displayed only by a sales manager.

Business Need: Can we hide certain price details (like Profit Margin or Internal Cost) from some users in sales documents or invoices?

Solution: In SAP S/4HANA, introduced Role-Based Access level to Price Elements. This controls who can view or edit specific price conditions (e.g., profit margin) in sales orders, returns, or invoices — either at header or item level. You can set access as no access, display only, or editable, based on the user’s role.

Recommendation by SAP: The higher the sensitivity, the higher is the number of the access level.

Benefit of S/4HANA Pricing:

Improved performance as pricing is stored in the single table PRCD_ELEMENTS, it leads to execution of sales order creation and billing creation.
Reduces errors by replacing multiple condition tables into one condition table.
Enhanced user experience with Fiori apps.
Supports Real-time profitability analysis.
Enhanced customization, increased range of digits, can create more Access sequences, Condition tables, Formula/Routine and Condition type counter.

Key Components of Pricing Procedure:

Condition Records: It stores the pricing information, taxes, and discounts. (e.g., prices, surcharge, taxes, discounts)
Condition Types: Define types of pricing components, such as- PR00 Basic price, K004 material discount. Each pricing type represents a particular pricing element.
Pricing Procedures: A pricing procedure is a collection of condition types arranged in a specific sequence, it defines the rules and logic for pricing that how to calculate conditions where conditions are mandatory, optional, or statistical.
Access Sequences: It searches the strategy for finding valid condition records.
Condition Tables: It specifies the combination of fields used to maintain condition records (e.g., customer and material, sales organization).
Master Data Integration: Pricing leverages data from Customer master and material master.

Difference between ECC pricing & S/4HANA pricing:

Feature	ECC Pricing	S/4HANA Pricing
Tables Used	Uses multiple tables: KONH, KONP, KONV	Merged into single table: PRCD_ELEMENTS
Pricing Execution Speed	Slower due to multiple tables joins.	Faster due to simplified data model (single table)
Pricing procedure Fields	16 fields	18 fields
Memory Consumption	Higher (due to table redundancy)	Lower (HANA in-memory capabilities)
Customer Pricing Procedure Length	1 digit	1 to 2 digits
Document Pricing Procedure Length	1 digit	1 to 2 digits
Access Sequence Capacity	Max 99 condition tables	Max 999 condition tables
Condition Table Range	Table numbers: 1–999 only	Table numbers: 1–999 + 483 additional tables
Condition Type Counter	2 digits	2 to 3 digits
Formulas/Routines	2 to 3 digits (e.g., 123)	3 to 7 digits (e.g., 1234567)

Conclusion: The evolution from ECC to S/4HANA in pricing procedure is not just as technical upgrade but it is transformation that brings greater flexibility, transparency, improved performance, time saving and error free. with the expansion from 16 to 18 pricing fields, the system now supports more accurate financial reporting and role-based access control through features like statistical posting to CO-PA and Access Levels.

SAP Developer News April 17th, 2025

2025-04-17T21:10:00.033000+02:00

DESCRIPTION

Podcast: https://podcast.opensap.info/sap-developers/2025/04/17/sap-developer-news-april-17th-2025/

Upcoming Removal of SAP Fiori Themes and UI5 Renovate Preset Config Tool

SAP Fiori Belize Theme Removal Announcement: https://community.sap.com/t5/technology-blogs-by-sap/announcement-removal-of-belize-theme-of-sap-fiori/ba-p/14061924
Upcoming Removal of SAP Fiori Themes Belize and Blue Crystal: https://community.sap.com/t5/technology-blogs-by-sap/upcoming-removal-of-sap-fiori-themes-belize-and-blue-crystal/ba-p/14063111
UI5 Renovate Preset Config Tool: https://community.sap.com/t5/technology-blogs-by-sap/stay-up-to-date-with-the-ui5-renovate-preset-config/ba-p/14070649

New SQL on Files Use Cases with SAP HANA Cloud QRC 01

Exploring New SQL on Files Use Cases with SAP HANA Cloud QRC 04/2024 and QRC 01/2025: https://community.sap.com/t5/technology-blogs-by-sap/exploring-new-sql-on-files-use-cases-with-sap-hana-cloud-qrc-04-2024-and/ba-p/14076223
Tutorials to Explore SAP HANA Cloud, SAP HANA Database SQL on Files: https://developers.sap.com/tutorials/hana-dbx-sof.html

SAP BTP Cloud Foundry: switching to higher security level Root Certificate Authority

Blog post https://community.sap.com/t5/technology-blogs-by-sap/sap-btp-cloud-foundry-switching-to-higher-security-level-root-certificate/ba-p/14061965

Migration Tooling now supporting migration via the pipeline approach

Blog post https://community.sap.com/t5/application-development-and-automation-blog-posts/introducing-the-new-project-creation-wizard-within-sap-build-lobby/ba-p/14079038

CHAPTER TITLES

0:00 Intro

0:10 Upcoming Removal of SAP Fiori Themes Belize and Blue Crystal

0:53 UI5 Renovate Preset Config Tool

1:41 New SQL on Files Use Cases with SAP HANA Cloud QRC 01

2:59 SAP BTP Cloud Foundry: switching to higher security level Root Certificate Authority

3:46 Migration Tooling now supporting migration via the pipeline approach

TRANSCRIPTION

[Into] This is the SAP Developer News for April 10th, 2025.

[Michelle] SAP has announced the removal of the Belize and Blue Crystal themes from its Fiori user interface, starting in the later half of this 2025. Belize along with Blue Crystal, will be phased out to make way for more modern themes like Quartz and Horizon These new themes promise enhanced design and usability, giving users a more streamlined interface SAP urges users to transition to these new themes to benefit from current and future improvements. By embracing Quartz and Horizon, organizations can ensure they remain aligned with SAP's innovation focus. if you want to learn more about the removal of these themes and to prepare for it. checkout the links below. In other news, SAP has introduced the UI5 Renovate Preset Config a tool designed to keep SAP UI5 projects effortlessly up-to-date. It automates the dependency update process, allowing developers and organizations to remain current with the latest framework improvements and security patches. The UI5 Renovate Preset Config minimizes manual intervention saving time while improving application stability This tool is an efficient and straightforward solution for developers eager to enhance project reliability with regular updates! for more information check below for link to blog post about this new tool. As always, thanks for watching and happy coding!

[Witalij] The SQL on files capability in SAP HANA Cloud has unlock new potential by providing direct read-only SQL access to files stored in SAP HANA Cloud and data lake files since QRC03 2024. Starting with the QRC 04 2024 of SAP HANA Cloud, this capability has been further extended to support direct access to the most common external object storages in addition to SAP HANA Cloud-owned data lake files, namely Amazon S3, Azure Blob Storage or ADLS Gen2, and Google Cloud Storage. With the ability to access vast volumes of data from managed object storage, you can fully leverage the robust features of SAP HANA Cloud as a platform. This includes in-memory processing, machine learning capabilities, predictive analytics library, and so forth. Another piece of good news is that the existing limitations on delta tables, which previously supported only Reader version 1 for both direct file access and delta sharing, has been lifted starting with the SAP HANA Cloud QRC01 this year. For hands-on exercises, check the updated tutorials at Explore SAP HANA Cloud, SAP HANA Database SQL on files.

[Mamikee] The SAP process orchestration to SAP integration suite now support automatic creation of integration artifacts supporting the pipeline for cloud integration. This update simplifies the shots from SAP process orchestration by using a pattern based method that automatically generate integration artifacts and makes your integration flow cleaner, smarter, and way easier to manage. Now, the pipeline model offers better error handling, fewer queues to juggle, and reusable components that save you time. No more stretching over rigid templates. In fact, it identifies supported patterns, making more scenarios eligible for migration. To learn more about this, check the link in the description below.

[DJ] If you're a user of the SAP Business Technology platform, whether using applications or services offered by SAP or by others, this is an important item, especially if you manage trust certificates yourself. There's a change to the root certificate authority, CA, used to issue certificates on SAP domains for Cloud Foundry, basically the TLS handshake between your browser and the server. This is for environments on BTP. This change is due to a recommendation from the Bundesamt für Sicherheit in der Informationstechnik, that's the German Federal Office for Information Security. In essence, there's an old G2 certificate, the DigiCert Global Route CA, and that's being replaced by a newer DigiCert TLS RSA 4096 Route G5. You can manage Route Certificate Authority certificates yourself, or, and this is what we recommend because it makes it a lot easier, you can use the BTP Trust Store. In any case, you must make sure that your trust store does contain this new G5 certificate, especially if you do manage the trust store yourself. Anyway, for all the details, check out the blog post that's linked in the description.

🇪🇸 Getting started with Generative AI Hub on SAP AI Core CodeJam

2025-04-21T09:33:21.102000+02:00

On the 30th of May, there will be an SAP CodeJam event on the topic of Getting started with Generative AI Hub on SAP AI Core and Python. You do not need to have any background in the topic, but a lot of curiosity!

RSVP Here -> https://community.sap.com/t5/sap-codejam/getting-started-with-generative-ai-hub-on-sap-ai-core/ec-p/14082043#M820

In this CodeJam you will learn how to use Generative AI Hub on SAP AI Core to implement a retrieval augmented generation (RAG) use case to improve the responses of large language models (LLMs) and reduce hallucinations. You will learn how to deploy an LLM and the orchestration service on SAP AI Core and query it via SAP AI Launchpad and the Python SDK. Furthermore, you will learn about the most important genAI concepts and create and use embeddings to improve your RAG response and build your own chatbot with memory.

This is an in-person event only (not virtual) and is planned for Friday, May 30th, 2025 at the SAP office in Madrid 🇪🇸 from 09:30 - 14:30 local time. The language of the event and the content will be English.

Linux AI Agent: Automating SAP HANA Installation

2025-04-22T10:53:02.548000+02:00

The New SAP AI Agent

I created an SAP AI agent. Again. This time, I challenged myself with an operating system-level task and specifically came up with the idea of installing the SAP HANA database using an AI Agent.

If you're already familiar with installing SAP HANA, you know it involves a lot of hands-on work in the terminal. Sure, you could use a graphical interface or automation tools, but for now, think of this as a proof of concept to show what this agent can do. We're aiming for a versatile tool here, not something that's only good with SAP HANA installation, but something that can handle a wide range of applications and tasks right from the Linux terminal.

The Linux Agent

For various tasks, we already know that being able to work in a terminal like a human is more effective than just executing commands, even though the latter might be easier to implement. Don't get me wrong - I've always said that while using human-like interfaces might not be the most efficient way for artificial intelligence to function, it's still essential because not everything can be done differently.

When working in a Linux terminal, the main tools you rely on are your eyes to see what's happening and your fingers to quickly press the keys. And actually, agents can be equipped with the same tools and expected to work in a manner similar to humans. I've equipped the agent with these three tools. Why three not two? To keep things simple:

We have a tool that lets you input text. It's pretty straightforward and obvious, so there's not much to delve into here.
The other tool may be slightly less obvious and is used for handling special keys and their combinations. For example, Backspace, Delete, Print Screen, Escape, Ctrl-C, Ctrl-V, and others. It is clear that this is not plain text, and for instance, the ESC key is invaluable when using VI. In fact, not only there.
And finally, we have a tool to preview the current contents of the terminal window. Something obvious to use during long-running operations when it is not really expected for the agent to press any keys.

OpenAI Agents SDK

In contrast to my SAP GUI AI Agent, where LangChain was used, this time I opted for the OpenAI Agents SDK framework. Why this one and not LangChain? The answer is simple. In such a dynamically changing industry, you have to try different solutions. And since the OpenAI framework doesn't tie me to models from just this company, but is universal, I decided to try it out. I conducted my initial tests specifically on the DeepSeek V3 model. Of course, when choosing other models, we lose some functionality, but the most important part is available.

Model Context Protocol

Lately, there's been quite a buzz about MCP, and for good reason - it's truly revolutionary. The power of agents comes from their tools, but what makes MCP stand out is its standardization and reusability. By designing an MCP server to work with a Linux terminal, I know I'll be able to use it later with another agent or even a regular chat client. Even the OpenAI desktop application will soon support MCP servers.

I'm confident that MCP will have a world-changing impact. We can use them for a range of tasks without always depending on external agents, just by adding tools to our favorite client. Keep in mind, MCP isn't just about tools; it also includes prompts and other resources. It's clear that someone designed it with the future in mind, potentially to replace whole external agents. I get the feeling that developing well-designed MCP servers will become more critical than creating agents. That's why it's essential to get to know this technology now, and that's why I opted for MCP.

The LLM

Although the OpenAI Agents SDK allows me to use virtually any model, I decided to test using GPT 4.1. This model was unveiled to the world just a few days ago. It is, in a sense, the successor to GPT-4o, but it is better at understanding commands and using tools, which is critical for agent applications.

Since my agent is an experimental solution, the choice of a new model was quite obvious. However, we should be aware that it is not a reasoning model and does not belong to the top tier of the best LLMs. It is, however, economical and very fast, which is evident in the agent's operation.

Hardware Environment

If you're familiar with my approach to miniPCs, it won't be a shocker that I picked this platform for testing. Sure, the cloud is super convenient and definitely the future, but running a system at home still has its perks. I used the Firebat AK2 with 16GB of RAM and installed Red Hat Enterprise Linux version 8.8 on it. It's free for developers and supported for SAP HANA.

Reflections on Safety

Safety in artificial intelligence is a crucial aspect that's often overlooked, but it's definitely something we should focus on. See how we are increasingly easily relinquishing control over our computers connected to the network to artificial intelligence. This approach carries some risk because we still don't completely understand how large language models operate. If you're curious about this topic, I strongly recommend checking out my earlier post: Beyond the Black Box: The Illusion of Control.

Here's the ironic part: having so many MCP servers available and being so easy to use can actually create problems. This happens when artificial intelligence is set up by people who don't fully understand the risks involved.

Summary

The video I recorded as part of the Agent tests is short, but be aware that these are just initial attempts. I'm surprised at how well it went the first time. While I haven't tested DeepSeek on an SAP HANA installation, only on simpler tasks, I see that GPT 4.1 handles agent tasks much better.

We're just starting out on this journey, but it's already clear how much potential there is. I see my agents as a proof of concept that shows how current models can handle a wide range of tasks they weren't specifically trained for. This flexibility is exactly what sets the agent-based approach apart from traditional automation. Sure, automating SAP HANA installations is straightforward and efficient. However, in this case, the agent executes a task efficiently even without prior training. This suggests it could tackle many other tasks just as well.

Where is this going to take us? We'll find out. But one thing's for sure - we're living in interesting times.

I invite you to join the discussion and feel free to ask any questions. If there's enough interest, I'd be more than happy to create a follow-up video.

SAP Datasphere External Access Overview: APIs, CLI and SQL

2025-04-22T10:55:27.647000+02:00

Querying and Managing SAP Datasphere with Python, Postman, Open SQL and the Command Line Interface

Introduction

This blog post aims to provide an overview of different external tooling options for SAP Datasphere as these resources are scattered across different Help pages, the Business Accelerator Hub and other community Blogs. This blog post does not aim to be exhaustive or cover every detail, but lists the different possibilities to perform actions in Datasphere or create, read, update or delete objects and data using external tools like Postman, Python, the CLI or open SQL.

Here is an overview of Topics covered in the Blog Post:

Rest API: TLS server certificates API; Connections API; Data Sharing Cockpit API and SCIM 2.0 API for user management
Command Line Interface: Manage User Access, Spaces, Modeling Objects, the Data Marketplace, Tasks and Task Chains as well as Connectivity
OData API: Get assets from the SAP Datasphere Catalog; Consume the datasets and metadata from consumable data assets.
Open SQL schema and ODBC/JDBC: Query the SAP HANA Cloud database with database users using SQL Statements like SELECT; CREATE; UPDATE; INSERT and ALTER Tables and Views.

Every Agenda Item is split into two sections: What? and How? The first section explains which use cases are supported by the shown technology and provides links to the relevant documentation. Second, a simple example is shown for each technology using the appropriate tools like Postman, Python, SQL or the CLI. This example can then easily be adapted and extended for future options explained in the What? section following the documentation. Additionally further links are provided to gain a deeper understanding of possible use case scenarios. Often the same action can be achieved by multiple options. E.g. It is possible to create and list connections via the CLI or via the REST API.

Of course, there is also the option to integrate SAP Datasphere directly with third party applications via e.g. OData or ODBC/JDBC connections or pushing the data to target systems like AWS S3 or GCP Cloud Storage using Replication Flows. However, these options are not part of this blog post, because application specific scenarios must be considered.

REST API

REST APIs are based on a standard architecture that uses HTTP methods like GET, POST, PUT, and DELETE. They allow you to perform actions in Datasphere regarding User & Role Management, Connection & Certificate Management and the usage of the Data Sharing Cockpit. You can call them via an API Platform such as Postman or via Programming Languages like Python or Type Script.

What?

Use the Certificates API to create, read and delete TLS server certificates to Datasphere - https://api.sap.com/api/CertificateManagement/overview
Use the Connections API to list, validate, delete, update or create connections in a space - https://api.sap.com/api/ConnectionManagement/overview
Use the Data Sharing Cockpit API to maintain your data provider profile; create and edit data product; manage licenses; create and publish releases and manage contexts - https://api.sap.com/api/DataSharingCockpit/overview
Use the SCIM 2.0 API to create, read, modify and delete users; add roles and get information on the identity provider - https://help.sap.com/docs/SAP_DATASPHERE/9f804b8efa8043539289f42f372c4862/1ca8c4a9467f43df9ae6d4ed3734f05a.html

How?

Using the REST API involves generally two steps.

Creating an OAuth 2.0 Client to Authenticate Against SAP Datasphere
Using Postman or another technology to obtain an access token and then calling the REST API with the access token

1 Creating an OAuth 2.0 Client to Authenticate Against SAP Datasphere

To create an OAuth2.0 Client users need the DW Administrator role. Under System -> Administration -> App Integration a new OAuth Client can be added.

In the OAuth Client configuration enter a name and choose the following settings:

Purpose: API Access
Access: Select the appropriate access (e.g. User Provisioning if you want to use the SCIM 2.0 API)
Security: Client Credentials
Token Lifetime

Click Add and copy the Client ID and Client Secret from the next screen (the client secret can only be copied now and you need to create a new client if you lose it!).

Additionally, please copy the Authorization URL and Token URL from the App Integration overview as they are needed later to authenticate.

2 Using Postman or another technology to obtain an access token and calling the REST API

Now we can use the Authorization URL, Token URL, Client ID and Client Secret to first obtain an access token and then call the REST API that we need.

The next steps will be shown in (1) Postman and (2) Python as two alternative approaches

(1) Postman: Create a collection and a new GET request within that collection. Provide the copied token URL and add grant_type=client_credentials in the Parameters.

Then switch to the Authorization tab use Authorization type Basic Auth and enter the Client ID as username and the client secret as password. You can now send the request and get the access_token with its lifetime as result. Copy the token for the next step.

Now we can call the REST API endpoint as documented in the resources linked in the What? Section (mainly https://api.sap.com/package/sapdatasphere/rest). For this simple example we will just get a list of connections from one of the Datasphere space by calling this endpoint:

Follow these steps:

Create a new GET request in Postman
Enter the URL https://<host>/api/v1/datasphere/spaces/<spaceId>/connections
- Host refers to the URL of your Datasphere Host; it can be copied from the browser. Copy everything until the first “/”
- The space ID is the ID of the space from which you want to get the connections. You can see available spaces if you navigate to Space Management in Datasphere.
Add x-sap-sac-custom-auth=true in the Headers section in Postman
Specify the Auth Type as Bearer Token by using the Token from the previous step.

Once you send the request you get a list of all connections from that space returned. You can now adapt this example to use any other REST API mentioned above by simply changing the URL + HTTP Method to the one specified in the documentation and adding relevant parameters.

Note: If you want to use the SCIM 2.0 API to create, modify or delete users you also need a so called CSRF Token by calling https://<host>/api/v1/csrf with the obtained access token and x-sap-sac-custom-auth=true and x-csrf-token=fetch as Headers.

(2) Python: The steps in Python are like the ones in Postman, using the same credentials and parameters. First an access_token needs to be obtained and then the connections API is called.

To simplify the scenario the Authorization URL, Token URL, Client ID and Client Secret are stored in global variables. However, in a productive scenario a secret store or environment variables should be used instead.

For this demonstration Python 3.9 is used. The only import that is needed is the requests library to make the API calls. Additionally, the credentials are stored as variables as mentioned. That is all for the setup.

import requests

token_url = xxx
username = xxx
password = xxx

To get the list of connections again two calls are made. (1) to get the access_token and (2) to the actual connections API endpoint. Each call is a function.

The get_token function specifies the authentication context using the client ID as username and the client secret as password. Via the request library the token_url is called with the defined authentication context. If the call was successful (code 200) the access_token is read from the API response.

def get_token():
    
    # Use basic authentication with username and password
    auth = requests.auth.HTTPBasicAuth(username, password)
    
    # API call
    response = requests.get(token_url, auth=auth)
    
    # Check result and return
    if response.status_code == 200:
        return response.json()['access_token']
    else:
        print("HTTP Error occurred", response.status_code)

In the second function the first function is called to get the access_token. Then it is passed on with the header x-sap-sac-custom-auth=true to the URL https://<host>/api/v1/datasphere/spaces/<spaceId>/connections as shown in the postman section.

def get_connections():
    connection_url = xxx

    # Get Token
    bearer_token = get_token()
    
    # Define headers and use bearer  token as authentication method
    headers = {
        "Authorization" : f"Bearer {bearer_token}",
        "x-sap-sac-custom-auth" : "true",
        "Content-Type": "application/json"
    }
    
    # API call
    response = requests.get(connection_url, headers=headers)
    
    # Check result and return
    if response.status_code == 200:
        return response.json()
    else:
        print("HTTP Error occurred", response.reason)

As with the Postman option you can now adapt this example to use any other REST API mentioned above by simply changing the URL + HTTP Method to the one specified in the documentation and adding relevant parameters.

To use the SCIM API, first a csrf token needs to be generated and then send in combination with the bearer token as headers to the Endpoint for deleting, modifying or creating users in the system. Instead of calling the API Endpoints directly from requests, a session needs to be created via requests.Session() to obtain the csrf token and call the create user Endpoint from one session.

Command Line Interface (CLI)

The Command Line Interface is a business user friendly toolset to achieve a variety of tasks in Datasphere without needing to write any code or use API platforms. Datasphere end users can run simple one-line statements in the command line after authenticating to perform admin tasks as well as to work with modeling objects in Datasphere.

What?

Work with Global & Scoped Roles; List, Add, Remove Users from Global and Scoped Roles; Manage Users https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/3a3d0ef3d4954797acac12afbcf9ab5d.html
List, Read, Create, Update and Delete Spaces; Manage Space Users and Database Users; Set Space Priorities and Statement Limits - https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/5eac5b71e2d34c32b63f3d8d47a0b1d0.html
List, Read, Create, Update and Delete Modeling Objects via the Command Line - https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/6f5c65f209004751aa48f9682ee2ec45.html
Manage the Data Providers, Products, Licenses, Releases and Contexts of the Data Marketplace via the Command Line - https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/5a815f6c21e9468eb96d0be95b9d2def.html
Manage Tasks and Task Chains via the Command Line - https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/2b26a31f197444dea314495bc0008eae.html
List, Upload and Delete TLS Certificates; List, Read, Create, Validate, Edit and Delete Connections - https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/8eb811898d1049fbb426339e44a2eb70.html

How?

To use the command line with SAP Datasphere it is recommended to use an OAuth 2.0 Client with interactive usage. The setup is quite similar to the one shown above for REST API, but some parameters have to be set up differently.

Again, to create an OAuth2.0 Client users need the DW Administrator role. Under System -> Administration -> App Integration a new OAuth Client can be added.

In the OAuth Client configuration different settings are used to use that Client with the CLI instead of REST API:

Purpose: Interactive Usage
Authorization Grant: Authorization Code
Redirect URI: This is the URI the user will be redirected to after authorization. For the Command Line Interface, you can simply start a localhost server on your machine using http://localhost:8080
Token Lifetime
Refresh Token Lifetime

Click Add and copy the Client ID and Client Secret from the next screen (the client secret can only be copied now and you need to create a new client if you lose it!).

Additionally, please copy the Authorization URL and Token URL from the App Integration overview as they are needed later to authenticate.

Using the Command Line Interface directly

To use the CLI for Datasphere Node.js >= 18 and <= 22 as well as npm >= 8 and <= 10 need to be installed. Npm is automatically installed with Node.js. Node.js can be downloaded from https://nodejs.org/

You can test the installation by running the following commands in your command line

$ node -v
$ npm -v

Then run the following command to install the datasphere related commands:

$ npm install -g /datasphere-cli

Some packages will be installed and you can check the successful installation by running

$ datasphere –version

Here is the summary of the installation commands:

Next step is to log in to Datasphere. As a best practice, make sure to always clean host, cache and secrets before logging in again. However, if this is the first time you are using the CLI no credentials will be available. Run these commands

$ datasphere config host clean
$ datasphere config cache clean
$ datasphere config secrets reset

Then log in to Datasphere using the datasphere login command. You will be prompted with the necessary credentials for log in. For this step you need the host URL you see when opening Datasphere in your browser (see above). Additionally, client ID and client secret that are copied from the OAuth client are needed.

$ datasphere login
✔ URL of the system to connect to: … <host>
✔ Please enter your client ID: … <client ID>
✔ Please enter your client secret: … <client secret>

After entering the client secret a browser window to the redirect URI provided in the OAuth Client will open and the log in will be automatically handled. You can continue in the CLI, where you are logged in now. Just start running commands to Datasphere documented in the What? Section above. Here is a simple example to get all the spaces in your Datasphere Tenant.

$ datasphere spaces list -H <host>

Here are all commands:

Note: For now you need to provide the host URL of the Datasphere Tenant via the -H option in every command. But the host can also be set as default and then it must not be used in every statement:

$ datasphere config host set <host>

And now you can run

$ datasphere spaces list

Once you are done, log out from Datasphere again

$ datasphere logout

There are many more options when using the CLI to work with Datasphere. E.g. the credentials can be stored in a secrets file, so you don’t have to paste them every time you log in. Additionally, a refresh token can be extracted once you are logged in and passed on when running a command, so you do not have to log in at the beginning of every session. Please refer to the documentation for more details.

Using the Command Line Interface within a scripting language

While this introduction uses the CLI directly, it should be mentioned that users can also use scripting languages like Python to automate the CLI usage by calling the commands directly from their code. In Python, the subprocess library is used to call CLI commands from code. This is a simple example to log in and list all available spaces in Datasphere. The credentials are stored in a JSON file:

import subprocess

subprocess.run("datasphere config host clean", shell=True)
subprocess.run("datasphere config cache clean", shell=True)
subprocess.run("datasphere config secrets reset", shell=True)

subprocess.run("datasphere login --host <host> --options-file ./dsp_cli_secrets.json", shell=True)

subprocess.run("datasphere spaces list -H <host>", shell=True)

Using this option is handy if you want to automate running CLI commands and chaining multiple commands together. An example would be to first get all spaces, then use a for loop to get all connections per space and use another command to validate all the connections.

Here is a great blog post that shows how you can combine the power of CLI and Python to generate views in Datasphere: https://community.sap.com/t5/technology-blogs-by-sap/sap-datasphere-view-generation-with-python-and-the-command-line-interface/ba-p/13558181

OData API

So far REST APIs and the CLI are shown to perform certain actions in Datasphere. But what if you want to consume and report on data within the Datasphere Tenant? In that case, you can use the next two options: OData API and the OpenSQL schema can be used.

What?

Catalog: List Spaces and Assets exposed for consumption
- https://help.sap.com/docs/SAP_DATASPHERE/43509d67b8b84e66a30851e832f66911/7a453609c8694b029493e7d87e0de60a.html#loio7a453609c8694b029493e7d87e0de60a__section_catalog_service
- https://api.sap.com/api/DatasphereCatalog/resource/SAP_Datasphere_Consumption_Catalog
Consumption: Retrieve Analytic Models, retrieve views exposed for consumption, retrieve metadata of Assets
- https://help.sap.com/docs/SAP_DATASPHERE/43509d67b8b84e66a30851e832f66911/7a453609c8694b029493e7d87e0de60a.html#loio7a453609c8694b029493e7d87e0de60a__section_analytical_data
- https://api.sap.com/api/DatasphereConsumption/overview

How?

As mentioned, the OData API is mainly used to consume objects from Datasphere. It can be accessed directly from the browser, via an API Platform like Postman, a scripting language like Python or it can be used by 3rd party applications like SAP Analytics Cloud and PowerBI to consume data from Datasphere in a reporting scenario.

Using the Browser to consume OData Requests

If you are privileged to see objects in Datasphere you can directly access the OData API from the browser, you are already logged in to Datasphere, since the log in context is just reused for the OData API without any additional setup. If there is e.g. an analytic model that you build and want to consume via OData, you can directly do so by just opening a new window in the browser and pasting the OData request URL. The OData request URL can be crafted yourself by referring to the documentation or you can use the Generate OData request available from the Datasphere UI for all objects exposed for consumption.

To generate the OData request, open the asset from the data builder. If it is exposed for consumption (Analytic Models are exposed by default; Views have a switch in the details pane to expose them) an icon appears in the header section under “Tools” to generate the OData request.

A pop-up opens to customize the OData URL. In the top, there is a selection if the actual data of the object should be received or its metadata. Additionally, variables and query parameters can be defined – as shown below. If the generate OData request is opened to retrieve data and the default settings are used the OData request URL look like this:

Exposed Views: <host>/api/v1/dwc/consumption/relational/<space Id>/<object technical name>/<object technical name>
Exposed Analytic Models: <host>/api/v1/dwc/consumption/analytical/<space Id>/<object technical name>/<object technical name>

As one can see the relational URL is used for views while the analytical is used for analytic models. This is because analytic models allow for more features like restricted measures and exception aggregations that are processed like a multidimensional statement and are dependent on the aggregation state defined via the variables and parameters. The relational URL for views just receives the result in a row-by-row fashion. Here is a Blog post exploring the differences in more detail: https://community.sap.com/t5/technology-blogs-by-sap/sap-datasphere-analytical-and-relational-odata-apis/ba-p/13573797

Changes you do to the variables and query parameters are reflected in the OData request URL. Variables are defined during the objects modeling process. If a default value is set for a variable it is used by default. If no default value is set, you must set a value for the variable to call the OData request. On the other hand, query parameters are not defined in the modeling process but are used in only that specific OData request. Query parameters are standard URL parameters used to filter, sort and limit the result set. Here is an overview of the usable query parameters

$select – return only specified columns
$filter – restrict result according to the provided criteria
$orderby – sorts the result by the specified column
$count – returns the count of the number of records
$top – limits the number of returned records to <n>
$skip – excludes the first <n> items
sap-language – returns the data in the specified language (if available)

In this example an analytic model to monitor task chain runs is shown. It has one variable INCLUDE_FAILURES_ONLY with the default value YES and query parameters are set to show only task chain steps with replication flows, ordered by the end date of the task chain run and limited to show only the last 100 results.

To see the result of the OData request you can click Preview or copy the URL to a new browser window. If everything is configured correctly a value array of objects is shown, where each object is one result row. As mentioned, this works without any additional setup since the log in context from your browser is reused.

Note: There is the limitation, that a user can send a maximum of 300 OData requests per minute. Pagination is used by default with 50.000 records per page. Via $skip and $top parameters client-side pagination can be implemented.

Using Postman to consume OData Requests

So far, we have seen, how to consume data in Datasphere via the OData API directly from the browser without any additional setup. However, if you want to use a 3rd party tool like Postman or Python to call OData requests you have to setup an OAuth client under System -> Administration -> App Integration with Interactive usage and a redirect URI. Please refer to the How? section under Command Line Interface (CLI) as the setup is the same. Once you got the client id, client secret, authentication and token URL you can continue in Postman by creating a new GET request. Copy & Paste the URL that you generated via the generate OData request function in Datasphere. You will see that the query parameters are automatically shown in the Parameters section of Postman.

The main step to use OData from Postman is to setup the Authorization correctly. Use Auth Type = OAuth 2.0 and header Prefix = Bearer (default).

Then configure a New Token in the section below.

Grant type = Authorization Code
Callback URL = <Your Callback URL set in the OAuth client (http://localhost:8080)>
Auth URL = <Authorization URL copied from System -> Administration -> App Integration>
Access Token URL = <Token URL copied from System -> Administration -> App Integration>
Client ID = <Client ID copied from the OAuth client>
Client Secret = <Client secret copied from the OAuth client>
Client Authentication = Send as Basic Auth Header.

Then you have to click on “Get New Access Token”.

Postman automatically handles the redirect and opens an embedded browser where you need to log in with your business user, using your normal Datasphere credentials. If the log in is successful, an Access Token Is generated, and you can click on Use Token.

The Token has the lifetime defined in the OAuth client. After it is expired you just click on get new access Token again and use the new Token. Having the Token, you can now send the API request to consume an analytic model or exposed view. Here we consume the same analytic model as we did when we generated the OData request and opened the URL in the browser.

That’s it for consuming data via OData in the browser and via Postman. Of course, you can also use Python or another language to replicate Postman’s behavior in handling the redirect URI and authorization against Datasphere by calling the authorization URL and handling the callback via the redirect URI. Since this approach involves a bit more coding than the other consumption options, I will publish the scenario in a separate Blog post. Please see my upcoming blog post [TBD].

HANA Database Explorer & Open SQL Schema

The last option to consume data from Datasphere via external tools is using SQL Statements to access the HANA Cloud Database underneath Datasphere directly instead of querying the objects in the modeling UI of Datasphere. Because SQL provides two-dimensional results in a row-by-row fashion, only views and tables can be consumed. Analytic models are multidimensional statements, so the OData API should be used to consume them instead. Third party tools like Tableau and Power BI use a JDBC/ODBC connection to the HANA Cloud to report on data exposed in views. This section shows how you can use this connection to consume and create objects of the HANA Cloud via the HANA Database Explorer and via Python.

What?

Access the space schema and read data from the space using SQL SELECT statements and
- https://help.sap.com/docs/SAP_DATASPHERE/be5967d099974c69b77f4549425ca4c0/3de55a78a4614deda589633baea28645.html
- https://userapps.support.sap.com/sap/support/knowledge/en/3428316
Create Tables and Views to write data to a space
- https://help.sap.com/docs/SAP_DATASPHERE/be5967d099974c69b77f4549425ca4c0/3de55a78a4614deda589633baea28645.html
- https://userapps.support.sap.com/sap/support/knowledge/en/3428316
Write SQL Script Procedures and add them to a task chain - https://help.sap.com/docs/SAP_DATASPHERE/c8a54ee704e94e15926551293243fd1d/59b9c773035a48c5beb54ce9bb29f1d8.html
Run Machine Learning Algorithms from HANA APL & PAL via open SQL

How?

To consume data via ODBC/JDBC you need either a database user or a database analysis user. The database user is limited to read from and/or write to an Open SQL schema with restricted access to the space schema whereas the Database analysis users have read only access to all space schemas (if configured).

To simplify the scenario, we will use a database analysis user for this introduction, because this user has default access to all the objects in Datasphere. To create a database analysis user, you need to be an administrator. Go to System -> Configuration -> Database Access -> Database Analysis User. Then click create to create a new user.

The Database Analysis User starts with “DWCDBUSER#” and you have to provide a custom suffix for the user. Additionally enable Space Schema Access to also consume data that is available in the spaces of your datasphere tenant. Click Create

After the user is created the database Host Name, Port, Password and Username are displayed. Copy all four credentials, as they are needed to log into the HANA cloud. In difference to OAuth Client used for the CLI and OData API you can simply request a new password for this user if you lose the old one.

That is the setup for now if you want to access the HANA Cloud via the HANA Cockpit or HANA Database Explorer. To consume data in Datasphere, the HANA Database Explorer can be used. Open your HANA Database explorer (e.g. via Space Management -> Edit -> Database Access -> Select a Database User -> Open Database Explorer). To use the created Database Analysis user the HANA Database Instance has to be added again with the analysis user. Click on the “+” sign in the upper left corner, select SAP HANA Database as Instance type and paste the credentials from the user creation.

You see the added connection in the upper left section of the HANA Database Explorer. The explorer helps you by automatically creating Select statements. To find an element to consume open the Instance and the Catalog Option. Choose Tables to consume Tables or Views to consume Views. By default, all Tables / Views from all schemas are shown. You can search for a specific Table / View or filter by schema. The schema option shows all schemas available on the HANA Cloud, but you can simply search for the name of one of your spaces that holds the object you want to consume.

In this example I select the table SalesOrders_TestUpload by selecting Tables from the catalog and filtering for my Space schema COE_EMEA_DW_DM and searching for test. By right clicking the table you have multiple options. To just consume the data via OpenSQL click “Generate SELECT Statement” and execute the statement. Data will be shown.

Instead of using the HANA Database Explorer you can also consume data in the HANA Cloud from any other Database client (e.g. DBeaver) or from a scripting language like Python by defining a connection to that HANA Database. To consume assets from Python you first need to allowlist your environment’s external IP address. Go to System -> Configuration -> IP Allowlist -> Trusted IPs and add your external IP address. You can get your external IP Address by running the command curl ifconfig.me on Linux/macOS or opening a website like https://ifconfig.me/. If you are using a VPN client, investigate the settings of your client and check if the external IP address is provided there. For testing IP range 0.0.0.0/0 can be used.

Then install the hdbcli extension module for Python. This defines the necessary API specification to directly send SQL queries to the HANA Cloud. You can find the documentation here: https://pypi.org/project/hdbcli/ and install it in the command line via the command:

$ pip install hdbcli

In a new Python file import the dbapi from hdbcli and define the connection with the four credentials from the Database Analysis user similarly to the HANA Database Explorer. Write a SQL command or copy the one generated from the HANA Database Explorer and execute it. Finally, run cursor.fetchall() to get a row-by-row result. Here is the full code

from hdbcli import dbapi	

conn = dbapi.connect(
    address="<HANA host>", 
    port=443, 
    user="DWCDBUSER#<suffix>", 
    password="<DB Analysis User Password>",
)

sql = 'SELECT * FROM "COE_EMEA_DW_DM"."SalesOrders_TestUpload"'
cursor = conn.cursor()
cursor.execute(sql)
cursor.fetchall()

Certainly, you can now start creating more complex SQL statements or working with the result set by e.g. passing the fetchall() command into a pandas DataFrame. As shown in the What? Section there are many more possibilities when working with the HANA Open SQL schema, like running SQL script procedures in Task Chains or executing HANA machine learning algorithms that are beyond this introductory blog post. Take a look at the articles linked in the What? Section to get you started with additional scenarios.

Conclusion

That’s it for this introduction to external access to Datasphere. By now you should have a good understanding of all the major ways to interact with Datasphere from tools like the CLI; Postman and Python. As mentioned in the beginning Replication Flows, OData and the ODBC/JDBC connection can also directly be used by 3rd party application to retrieve data from Datasphere. This was not part of this blog post as the configuration differs between all the possible targets. There is extensive documentation and there are many community blogs available to explain application specific setups.

If you have questions or noticed a scenario I didn’t cover, feel free to leave a comment below the blog post.

Cheers, Henri

Troubleshooting SAP OS Collector (saposcol) Issues and Resolving ST06 Data Access Problems

2025-04-22T18:52:18.629000+02:00

Introduction:

In this article, I will share my experience troubleshooting issues related to the SAP OS Collector (saposcol) not running, which led to challenges in accessing ST06 data and analyzing system performance. We will explore the common symptoms such as shared memory errors and missing dump status data, discuss the underlying causes, and outline the steps taken to resolve the issue. This guide aims to help SAP administrators quickly identify and address similar issues in their environments.

Key Notes:

Always Use the Latest saposcol
SAP recommends using the latest version of saposcol. It is part of the SAPHOSTAGENT.SAR package.
Root Privileges Required for saposcol
To start saposcol, ensure you have root privileges, not the SIDadm user. This is because saposcol collects all OS statistics.
Location of saposcol Executables
To start or check the status of saposcol, navigate to the following directory: /usr/sap/hostctrl/exe
useful commands for saposcol:
./saposcol -s (to check the status)
./saposcol -l (to start)
./saposcol -k (to stop)
./saposcol -d (to access saposcol by dialog mode, once ./saposcol -d executed then type help, it will show the list of dialog mode commands in the dialog mode)

Solution:

In my case, I used saposcol’s dialog mode and executed the launch command to start it. After starting, I encountered a NipConnect error that kept displaying. Initially, I thought it was a persistent error, so I waited without closing the command prompt. After approximately 5 minutes, saposcol successfully started.

If you're still facing issues when checking the history of collected saposcol data in ST06, follow the solutions provided in the SAP Notes listed below.

Recommended SAP Notes:

👉 SAP Note 1915341 – ST06: No historical data available

👉 SAP Note 2338673 – SM12: Lock entry remains in system

Additional Steps:

In my case, I resolved the issue by restarting the sapstart service. The command for restarting the service is as follows:

sapcontrol -nr <instance nr> -function RestartService

After restarting the service, I was able to see the ST06 history for previous hours.

Additional Useful SAP Notes:

Conclusion:

By following the steps outlined above and referring to the relevant SAP Notes, I was able to resolve the issue and restore access to ST06 data. If you encounter similar problems, I hope this guide will provide you with the necessary steps to troubleshoot and fix the issue in your SAP environment.

How to restore HANA Backup with ENCRYPTION ROOT KEYS in Target System.

2025-04-24T13:35:08.558000+02:00

Source System Details. Target System Details.

DB SID HXP. DB SID is HXD.

Prerequisites--- 1. root key password require. 2. HANA Studio require.

We are restoring HXP backup in HXD system, but we don’t have root key then how to restore backup in Target (HXD) HANA DB.

Facing the error message below in HANA Studio while restoring backup in HANA HXD.

Backup rootkey 3ef7cae23ce897c7748807d98689abc1bfd3d265654140b244b4ae8918c6e096 is required for decryption of backup files, but not found in key store. Please make sure you have provided the necessary keys before

recovery.,

No root key matching hash=3ef7cae23ce897c7748807d98689abc1bfd3d265654140b244b4ae8919c6e096

Reason is During the HANA DB installation below option selected Yes and HXP backup was taken with encryption key.

While restore HANA DB backup in Target system you have to provide root key password other wise backup will not restore and throw the error.

Source System Step (HXP)

Solution

Step 1.

Take the backup of the ssfs keys.

We have taken backup with ssfs_backup

Step 2.

From SYSTEM DB we need to run below query.

We need to set ENCRYPTION root key password.

ALTER SYSTEM SET ENCRYPTION ROOT KEYS BACKUP PASSWORD (password)

Step 3.

Now Validate the password from below query in HANA Studio.

ALTER SYSTEM VALIDATE ENCRYPTION ROOT KEYS BACKUP PASSWORD (password)

Step 4.

Now run this query for checking DB ID in tenant DB.

SELECT DATABASE_NAME, 
	CASE WHEN (DBID = '' AND 
	DATABASE_NAME = 'SYSTEMDB') 
	THEN 1
	WHEN (DBID = '' AND 
	DATABASE_NAME <> 'SYSTEMDB') 
	THEN 3
	ELSE TO_INT(DBID)
	END DATABASE_ID 
FROM (SELECT DISTINCT DATABASE_NAME, SUBSTR_AFTER (SUBPATH,'.') AS DBID FROM SYS_DATABASES.M_VOLUMES);

HXP DB ID is 3.

Step 5.

hdbnsutil -backupRootKeys /backup/backup/rootkey/rootkey_HXP.rkb --dbid=3 --type='BACKUP'

We used here dbid 3 which we checked in Tenant DB earlier screen shot.

Step 6.

This is rootkey location and root key copy from HXP to HXD.

/backup/backup/rootkey

Note: Copy the rootkey_HXP.rkb file from HXP (Source ) to HXD (Target)

Target System Step (HXD)

Step 7. (HXD Target DB)

Check the encryption status

SELECT * FROM SYS_DATABASES.M_ENCRYPTION_OVERVIEW

Encryption Status TRUE Coming. It means now Encryption is ok and we can restore backup easlily.

Step 8.

SELECT DATABASE_NAME, 
	CASE WHEN (DBID = '' AND 
	DATABASE_NAME = 'SYSTEMDB') 
	THEN 1
	WHEN (DBID = '' AND 
	DATABASE_NAME <> 'SYSTEMDB') 
	THEN 3
	ELSE TO_INT(DBID)
	END DATABASE_ID 
FROM (SELECT DISTINCT DATABASE_NAME, SUBSTR_AFTER (SUBPATH,'.') AS DBID FROM SYS_DATABASES.M_VOLUMES);

We can see DB ID 3 same coming which was coming in HXP (Source DB )

DB ID is 3.

Step 9.

We have taken backup with ssfs_backup.

Step 10.

Step 11.

Go to same location /backup/backup/rootkey and run below command for import

/backup/backup/rootkey

hdbnsutil -recoverRootKeys rootkey_HXP.rkb --dbid=3 --password="password" --type='BACKUP'

Step 12.

Now you can start Restore again it will work.

Thanks for watching it. 😊

SAP Datasphere CLI & Python: Exporting Modeling Objects to CSV Files for Each Artifact

2025-04-29T18:57:17.498000+02:00

Introduction

In this blog post, we'll explore how to use Python alongside SAP Datasphere CLI to extract modeling objects and export them to CSV files. The script allows users to handle artifacts such as remote tables, views, replication flows, and more, for each space in SAP Datasphere.
This solution is particularly useful for automating repetitive tasks and ensuring structured data handling across different modeling objects

Prerequisites

Steps to install SAP Datasphere CLI:

https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/f7d5eddf20a34a1aa48d8e2c68a44e28.html/

https://community.sap.com/t5/technology-blogs-by-sap/sap-datasphere-external-access-overview-apis-cli-and-sql/bc-p/14086942#M180986/

Step-by-Step Process

Step 1: Prepare Login.Json file

Create OAuth Client with Purpose as Interactive Usage and Redirect URL as http://localhost:8080

Get the value of all below fields from the OAuth Client and prepare the Login.json file.

{
"client_id": "",
"client_secret": "",
"authorization_url": "",
"token_url": "",
"access_token": "",
"refresh_token": ""
}

Step 2: Create Model_Object.py file with below code

dsp host : give URL of Datasphere Tenant.

secrets_file : Give Path of Login.json file.

import subprocess
import pandas as pd
import sys

def manage_Modeling_Object(Modeling_Object):
    # Step 1: Login to Datasphere using host and secrets file
    dsp_host = '<URL of Datasphere>'
    secrets_file = '<path>/Login.json'
    command = f'datasphere login --host {dsp_host} --secrets-file {secrets_file}'
    subprocess.run(command, shell=True)  # Execute the login command
    
    # Step 2: Retrieve a list of all spaces in JSON format
    command = ['datasphere', 'spaces', 'list', '--json']
    result_spaces = subprocess.run(command, capture_output=True, shell=True, text=True)  # Run the command and capture output
    
    # Step 3: Parse the list of spaces from the command's output
    spaces = result_spaces.stdout.splitlines()  # Split output into individual lines
    
    ModelingObject_data = []  # Initialize a list to store Modeling Object data
    
    # Step 4: Check if the Modeling Object is 'spaces'
    if Modeling_Object == 'spaces':
        for space in spaces:
            if space == "[" or space == "]":
                continue  # Skip brackets in the JSON output
            space_id = space.strip()  # Extract space ID
            
            # Add space details to the data list
            ModelingObject_data.append({
                'Space ID': space_id.replace('"', '').replace(',', ''),
                'Technical Name': space_id.replace('"', '').replace(',', ''),
                'TYPE': Modeling_Object[:-1].upper()  # Set the TYPE as uppercase version of the input Modeling Object name
            })
    
    # Step 5: Process Modeling Objects for each space
    else:
        for space in spaces:
            if space == "[" or space == "]":
                continue  # Skip brackets in the JSON output
            space_id = space.strip()  # Extract space ID
            
            # Step 6: Retrieve Modeling Objects for the current space
            command = ['datasphere', 'objects', Modeling_Object, 'list', '--space', space_id.replace('"', '').replace(',', '')]
            result_ModelingObject = subprocess.run(command, capture_output=True, shell=True, text=True)  # Run the command
            
            # Step 7: Parse the Modeling Object data from the output
            ModelingObject_info = result_ModelingObject.stdout.splitlines()  # Split output into individual lines
            print("Checking "+Modeling_Object.upper()+" for space : "+space_id.replace('"', '').replace(',', ''))  # Log the space being checked
            
            # Step 8: Process each Modeling Object
            if len(ModelingObject_info) > 1:
                for flow in ModelingObject_info:
                    if '{' in flow or '}' in flow or '[' in flow or ']' in flow:
                        continue  # Skip brackets or braces in the output
                    cleaned_flow = flow.replace('"technicalName":', '').replace('"', '').strip()  # Clean up the output
                    
                    # Step 9: Add Modeling Object details to the data list
                    ModelingObject_data.append({
                        'Space ID': space_id.replace('"', '').replace(',', ''),
                        'Technical Name': cleaned_flow,
                        'TYPE': Modeling_Object[:-1].upper()  # Set the TYPE as uppercase version of the input Modeling Object name
                    })
    
    # Step 10: Write the collected data into a CSV file
    if ModelingObject_data:
        df = pd.DataFrame(ModelingObject_data)  # Create a DataFrame from the data list
        df.to_csv(Modeling_Object.upper()+'.csv', index=False)  # Save the DataFrame to a CSV file without the index
        print("Space vise all "+Modeling_Object.upper()+" have been written to "+Modeling_Object.upper()+".csv.")  # Log success message
    else:
        print("No Modeling Objects found.")  # Log message if no data was collected
    
    print('------------------------------------------------------------------------------------------------------------------------------------')  # Separator for readability
        
if __name__ == "__main__":
    # Check if an argument is provided via the command line
    if len(sys.argv) > 1:
        # Pass the first argument to the method
        manage_Modeling_Object(sys.argv[1])
    else:
        print("Please provide a Modeling Object name as an argument.")  # Log error message if no argument is provided
        
# Execute for predefined Modeling Objects
manage_Modeling_Object('remote-tables')
manage_Modeling_Object('local-tables')
manage_Modeling_Object('views')
manage_Modeling_Object('intelligent-lookups')
manage_Modeling_Object('data-flows')
manage_Modeling_Object('replication-flows')
manage_Modeling_Object('transformation-flows')
manage_Modeling_Object('task-chains')
manage_Modeling_Object('analytic-models')
manage_Modeling_Object('data-access-controls')

Step 3: Open command prompt and execute the Model_Objects.py file

Once the program execution is done it will generate CSV files for all the Datasphere artifactes mention in python code

each CSV file will have 3 columns :

1) Space ID : Name of the space

2) Technical Name : Exact Technical Name of Object

3) Type : Type of Object ( i.e view, local-table, remote-table, replication flw etc)

Conclusion

This script demonstrates how Python and SAP Datasphere CLI can collaborate to streamline artifact management and export data systematically. By following the steps provided, users can extend or adapt the code to suit their requirements.

SAP Datasphere CLI & Python Automation : Extract All Artifacts of Datasphere in CSV files.

2025-05-05T10:47:20.047000+02:00

Introduction

In this post, we'll look at how to use Python with SAP Datasphere CLI to extract data objects and save them as CSV files. The script helps you manage items like remote tables, views, replication flows, and more for every space in SAP Datasphere. It's a great tool for automating repeated tasks and keeping data organized across different objects.

Usecases:

1. Validate Namig Convesion : Generated files can be used as source in Datasphere to validate the naming convension for all artifactes
2. Identify and Delete unncessary objects from Datasphere Tenant.

Prerequisites

Steps to install SAP Datasphere CLI:

https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/f7d5eddf20a34a1aa48d8e2c68...

https://community.sap.com/t5/technology-blogs-by-sap/sap-datasphere-external-access-overview-apis-cl...

Step-by-Step Process

Step 1: Prepare Login.Json file

Create OAuth Client with Purpose as Interactive Usage and Redirect URL as http://localhost:8080

Get the value of all below fields from the OAuth Client and prepare the Login.json file.

{
"client_id": "",
"client_secret": "",
"authorization_url": "",
"token_url": "",
"access_token": "",
"refresh_token": ""
}

Step 2: Create Model_Object.py file with below code

dsp host : give URL of Datasphere Tenant.

secrets_file : Give Path of Login.json file.

import subprocess
import pandas as pd
import sys

def manage_Modeling_Object(Modeling_Object):
    # Step 1: Login to Datasphere using host and secrets file
    dsp_host = '<URL of Datasphere>'
    secrets_file = '<path>/Login.json'
    command = f'datasphere login --host {dsp_host} --secrets-file {secrets_file}'
    subprocess.run(command, shell=True)  # Execute the login command
    
    # Step 2: Retrieve a list of all spaces in JSON format
    command = ['datasphere', 'spaces', 'list', '--json']
    result_spaces = subprocess.run(command, capture_output=True, shell=True, text=True)  # Run the command and capture output
    
    # Step 3: Parse the list of spaces from the command's output
    spaces = result_spaces.stdout.splitlines()  # Split output into individual lines
    
    ModelingObject_data = []  # Initialize a list to store Modeling Object data
    
    # Step 4: Check if the Modeling Object is 'spaces'
    if Modeling_Object == 'spaces':
        for space in spaces:
            if space == "[" or space == "]":
                continue  # Skip brackets in the JSON output
            space_id = space.strip()  # Extract space ID
            
            # Add space details to the data list
            ModelingObject_data.append({
                'Space ID': space_id.replace('"', '').replace(',', ''),
                'Technical Name': space_id.replace('"', '').replace(',', ''),
                'TYPE': Modeling_Object[:-1].upper()  # Set the TYPE as uppercase version of the input Modeling Object name
            })
    
    # Step 5: Process Modeling Objects for each space
    else:
        for space in spaces:
            if space == "[" or space == "]":
                continue  # Skip brackets in the JSON output
            space_id = space.strip()  # Extract space ID
            
            # Step 6: Retrieve Modeling Objects for the current space
            command = ['datasphere', 'objects', Modeling_Object, 'list', '--space', space_id.replace('"', '').replace(',', '')]
            result_ModelingObject = subprocess.run(command, capture_output=True, shell=True, text=True)  # Run the command
            
            # Step 7: Parse the Modeling Object data from the output
            ModelingObject_info = result_ModelingObject.stdout.splitlines()  # Split output into individual lines
            print("Checking "+Modeling_Object.upper()+" for space : "+space_id.replace('"', '').replace(',', ''))  # Log the space being checked
            
            # Step 8: Process each Modeling Object
            if len(ModelingObject_info) > 1:
                for flow in ModelingObject_info:
                    if '{' in flow or '}' in flow or '[' in flow or ']' in flow:
                        continue  # Skip brackets or braces in the output
                    cleaned_flow = flow.replace('"technicalName":', '').replace('"', '').strip()  # Clean up the output
                    
                    # Step 9: Add Modeling Object details to the data list
                    ModelingObject_data.append({
                        'Space ID': space_id.replace('"', '').replace(',', ''),
                        'Technical Name': cleaned_flow,
                        'TYPE': Modeling_Object[:-1].upper()  # Set the TYPE as uppercase version of the input Modeling Object name
                    })
    
    # Step 10: Write the collected data into a CSV file
    if ModelingObject_data:
        df = pd.DataFrame(ModelingObject_data)  # Create a DataFrame from the data list
        df.to_csv(Modeling_Object.upper()+'.csv', index=False)  # Save the DataFrame to a CSV file without the index
        print("Space vise all "+Modeling_Object.upper()+" have been written to "+Modeling_Object.upper()+".csv.")  # Log success message
    else:
        print("No Modeling Objects found.")  # Log message if no data was collected
    
    print('------------------------------------------------------------------------------------------------------------------------------------')  # Separator for readability
        
if __name__ == "__main__":
    # Check if an argument is provided via the command line
    if len(sys.argv) > 1:
        # Pass the first argument to the method
        manage_Modeling_Object(sys.argv[1])
    else:
        print("Please provide a Modeling Object name as an argument.")  # Log error message if no argument is provided
        
# Execute for predefined Modeling Objects
manage_Modeling_Object('remote-tables')
manage_Modeling_Object('local-tables')
manage_Modeling_Object('views')
manage_Modeling_Object('intelligent-lookups')
manage_Modeling_Object('data-flows')
manage_Modeling_Object('replication-flows')
manage_Modeling_Object('transformation-flows')
manage_Modeling_Object('task-chains')
manage_Modeling_Object('analytic-models')
manage_Modeling_Object('data-access-controls')

Step 3: Open command prompt and execute the Model_Objects.py file

Once the program execution is done it will generate CSV files for all the Datasphere artifactes mention in python code

each CSV file will have 3 columns :

1) Space ID : Name of the space

2) Technical Name : Exact Technical Name of Object

3) Type : Type of Object ( i.e view, local-table, remote-table, replication flw etc)

Conclusion

🚀 SAP Green Token: Goodbye Blockchain, Hello Faster and More Efficient Auditability

2025-05-06T09:00:00.027000+02:00

Blockchain has been a long-standing and energy-efficient component of SAP Green Token. As we continuously strive to increase the value to our customers and partners, Green Token also continues to evolve. As part of Green Token’s continuous evolution, the strategic decision has been made to sunset the blockchain component.

🔍What were the initial drivers to adopt blockchain?

When Green Token was first developed in 2019, the vision was to create a blockchain network where companies in the value chain could collaborate on end-to-end traceability and transparency across the supply chain.

❓Why are we now removing it?

The reality is that the market and supply chains have responded differently to the level of transparency required and the level of transparency that is willingly disclosed beyond tier-1 supply chain partners.

Even though the transition to a sustainable economy is accelerating, the adoption of extensive transparency milestones across the end-to-end supply chain still lags. A recent study by McKinsey (2024) showed that even though tier-1 transparency continues to increase, deeper tier-n visibility is decreasing.

In fact, blockchain fatigue has become a well-known reality across the industry. According to Gartner, 90% of blockchain-based supply chain initiatives were expected to suffer from blockchain fatigue by 2023, placing the technology firmly in the Trough of Disillusionment on the hype cycle. This mirrors direct feedback from our customers—interest in blockchain networks simply didn’t materialize.

✅What do customers actually want?

The main need and ask for Green Token is and always has been: chain-of-custody accounting of sustainable materials across a single organization (including intra-company transfers) and its tier-1 material movements.

Which is also the main value proposition Green Token continues to offer today.

At the end of the day, organizations and value chains really value two things:

Auditability and traceability of transactions
Fast and efficient performance without unnecessary complexity

🔄What’s changing under the hood?

As part of ongoing improvements, Green Token is being migrated to SAP HANA, enhancing speed, efficiency, and security while maintaining full auditability.

From the start, Green Token has been built on the principle of digital twins, which act as a real-world representation of materials, transactions, and ownership. The blockchain component supplemented auditability capabilities, but the foundation was—and remains—in Green Token’s persistence layer, including its digital twin and audit logging system.

By removing the blockchain, we’re increasing Green Token’s performance:

Transactions are faster due to optimized data storage and retrieval
Security and traceability remain intact, ensuring compliance with reporting requirements

🔐Trust without blockchain?

Blockchain is often used to represent trust. But trust isn’t built by technology alone—it’s built through verifiable data, traceability, and robust audit trails.

SAP Green Token continues to provide the same level of data integrity and transaction traceability, ensuring continuous compliance with changing regulations and standards—without the overhead of blockchain.

💡So what is a token?

With that, SAP Green Token remains SAP Green Token. After all, tokenization ≠ blockchain.

A token is a representation of something in the real world, and that is exactly what we continue to provide.

A token is like a marble in a bucket—you can still see how many there are, where they came from, and where they go, without needing a blockchain.

With the migration to SAP HANA, the digital twin tokens will reside in SAP HANA along with audit logs supported by SAP BTP, providing transparency over all transactions.

🚛For certified material movements, we’re just getting faster

When it comes to certified sustainable material movements and chain-of-custody accounting, SAP Green Token remains a secure and auditable solution for supply chain compliance—just faster and more efficient going forward.

🔹 If you're already live or currently implementing Green Token, no action is required—this change will not impact your setup or ongoing implementation.

Hi! 👋 I’m Gloria Figaroa. Since 2015, I live and breathe sustainability and supply chain management. I’ve had the pleasure to work on Green Token for the past 4+ years as a Product Manager. Now in my latest role as Product Marketing Manager, I take care of everything Go-to-Market. 🌍

Follow me to stay up to date on the SAP Green Token solution, its newest features and functions, and detailed overviews of its capabilities. You will hear it first here! 📢

Data volume problems during an SAP S/4HANA Conversion

2025-05-07T12:06:41.075000+02:00

An SAP S/4HANA conversion refers to migration from a legacy SAP ERP system, such as SAP ECC, to the modern SAP S/4HANA business suite. This process is essentially a platform upgrade that includes adapting data system configurations, and custom developments to ensure compatibility with S/4HANA. Click here to read more about the features of S/4HANA.

There are three approaches available when embarking on this journey (read a more in-depth discussion about them here)

Greenfield approach - means starting from a clean slate.
Brownfield approach - is more like a system upgrade.
Hybrid approach - lets you select the best parts of Greenfield and Brownfield implementations.

Many customers embarking on this transformation journey opt for the brownfield conversion approach. The brownfield approach retains your historical data which can lead to the technical challenge we will discuss and solve in this blog.

SAP provides a process for the conversion to SAP S/4HANA, this is illustrated in the figure below which gives an overview of the tools, the phases, and the activities involved in the process. The activity that we will be focusing on is t5 (Software Update Manager (SUM)), specifically the “Data Conversion” task:

For further reading:

Conversion phases and related activities are documented in the “Conversion Guide for SAP S/4HANA and SAP S/4HANA Cloud Private Edition 2023”, which can be found under the Implement section of the SAP S/4HANA | SAP Help Portal.
For information specific to accounting components please refer to the following SAP note 2332030 - Conversion of accounting to SAP S/4HANA - SAP for Me.

The “Data Conversion” task is the conversion of your data into the new data structure used by SAP S/4HANA.

At this point you could be wondering “What new data structure used by SAP S/4HANA?”, well I’m glad you asked, let me explain.

The data model used in SAP S/4HANA has been simplified drastically when comparing it with the data model used in SAP ERP. If you look at the Finance section of the below diagram you will see that many of the legacy tables have been consolidated into a simplified data model focused on the ACDOCA table.

Furthermore, the effects of Accounting Data Model Changes and how they impact the conversion are described as follows in an extract from the Converting Your Accounting Components to SAP S/4HANA guide:

To use Finance in SAP S/4HANA you must migrate the existing user data from the General Ledger, Asset Accounting, Controlling and Material Ledger. The data migration is necessary because Finance in SAP S/ 4HANA rests on a uniform data model for all accounting areas. The comprehensive ACDOCA data table contains all line-item documents. After the migration, all postings of the named applications are written into the ACDOCA table.

The following tables are replaced by views using the same technical names:

The line item, totals tables and application index tables of the General Ledger (GLT0, BSIS, BSAS and FAGLFLEXA, FAGLFLEXT, FAGLBSIS, FAGLBSAS).
The totals tables and application index tables of Accounts Receivable and Accounts Payable (KNC1, KNC3, LFC1, LFC3, BSID, BSIK, BSAD, BSAK).
The line item and totals tables of Controlling (COEP for certain value types, COSP and COSS).
The material ledger tables for parallel valuations (MLIT, MLPP, MLPPF, MLCR, MLCD, CKMI1, BSIM).
The Asset Accounting tables (ANEK, ANEP, ANEA, ANLP, ANLC).

This is the crux of our challenge: data that was previously distributed across multiple tables in the legacy system is now consolidated into a single table in the S/4HANA environment, that is ACDOCA.

Although this may not appear to be an issue under normal circumstances, in high-volume scenarios, technical limitations of the HANA database platform can arise when a single table (or partition) approaches a very large number of records. Specifically, each table or partition in HANA is limited to a maximum of 2 billion entries.

The solution: table partitioning!

Let’s do a deep dive into partitioning and its role during conversions by exploring a few questions.

Question 1: What is table partitioning?

According to SAP Help Portal: The partitioning feature of the SAP HANA database splits column-store tables horizontally into disjunctive sub-tables or partitions. In this way, large tables can be broken down into smaller, more manageable parts. Partitioning is typically used in multiple-host systems, but it may also be beneficial in single-host systems.

Question 2: Why do we need to do partitioning?

As mentioned earlier, there is a limitation of 2 billion records per table (partition) on the HANA database. This is described in more detail in SAP Note 2154870 - How-To: Understanding and defining SAP HANA Limitations - SAP for Me.

If the limit is reached the following errors could be generated:

2048: column store error: [17] Docid overflow in current delta.
2055: maximum number of rows per table or partition reached: '<schema>.<table>'
Maximum number of rows per partition reached.
Max row count exceeded.
exception 1: no.3100025: Allowed rowcount exceeded.

Below is an example of a short dump with the error “maximum number of rows per table or partition reached” that can occur in step ID MUJ “Data Migration into Unified Journal: Line Items”:

Question 3: How do we know that we need to do partitioning for the conversion?

The Readiness Check which forms part of the Preparation Phase can guide the need for partitioning. It details the table records in the existing tables in ECC (classic General Ledger or New General Ledger), and also gives an estimate of the number of records expected in ACDOCA. See the table below as an example:

The calculation is not exact, but it already gives you an idea that the number of records will exceed 1 billion for ACDOCA, in this case, partitioning is required.

Further Reading: Great community blog - SAP S/4HANA Simplification Item Check - How to do it right.

Note: Ensure that all relevant notes are implemented to get thorough analysis and results.

Question 4: How should we partition the tables?

Luckily there is quite a bit of guidance on partitioning in general, as well as specific guidance for Finance Tables, please review the following:

SAP Note 2044468 - FAQ: SAP HANA Partitioning
SAP Note 2289491 - Best Practices for Partitioning of Finance Tables - SAP for Me
For scale out scenarios read SAP Note 3325787 - Partition-Distribution of Financial Data in SAP S/4HANA
Guidance provided in SAP Customizing Implementation Guide:

When designing a partitioning strategy, it is important to analyze how the data is distributed over Fiscal Periods and Years, and to anticipate future data growth.

Note: Pay attention to the following guidance from SAP Note 2289491 - Best Practices for Partitioning of Finance Tables when creating future partitions à Too many empty partitions can affect the HANA optimizer and cause poor performance for queries which don't apply partition pruning.

Question 5: When should partitioning be done during the conversion process?

The partitioning of the ACDOCA table, in the target system, can be done once the SUM tool has reached the “business downtime” phase (this phase is known as DOWNCONF_DTTRANS) on the source system (in SUM uptime phase).

Note 1: Other tables are partitioned automatically by the SUM tool.

Note 2: If partitioning isn’t performed before the data migration it is recommended to reset the migration rather than trying to apply partitioning with an already populated ACDOCA. Once records exist in ACDOCA, applying partitioning becomes significantly slower. It's recommended to reset the migration (Step ID MUJ is where ACDOCA begins to be populated) - this clears the records in ACDOCA — then proceed with partitioning.

Before we wrap up, we would like to mention that it is very important to focus on Data Volume Management before conversion.

Benefits are:

Reduced data footprint which results in a shorter conversion duration.
Reduced memory requirement for the new SAP S/4HANA system.

The following guide and notes contain useful reading on the topic:

Hope you enjoyed this blog! Please feel free to leave suggestions and comments.

This blog was co-authored by @DotEiserman.