SAP Community - SAP Connectivity service

Surviving and Thriving with the SAP Cloud Application Programming Model: Deployment to SAP BTP, Kyma runtime

2023-03-07T15:51:05+01:00

Greetings and salutations, fellow CAP enthusiasts! We're back with another episode of "Surviving and thriving with the SAP Cloud Application Programming Model" (#CAPTricks). Last time, we covered the wild and bumpy ride of getting your app up and running on your local machine, complete with all the twists and turns of talking to on-premise systems via the Cloud Connector and whatnot.

But now, I've been experimenting with the SAP BTP, Kyma runtime, and wanted to see just how quick and easy it is to take our "trusty old CAP app" from the Cloud Foundry environment over to the Kyma landscape - it's been quite a journey for me personally. Now, if you're still thinking this series is going to be a neat and tidy chronology, I hate to disappoint - I just write about what pops up on my radar. (But don't worry, I've got much more like integration and UI tests on the horizon, I just need to do some more research before I feel confident enough to share my findings. 😉)

I've been primarily working with the SAP BTP, Cloud Foundry runtime, but we have seen a growing demand and need for applications to be deployed and running on Kubernetes. SAP provides a managed Kubernetes-based runtime called SAP BTP, Kyma runtime which has started as the open-source project Kyma. With loads of discussions on which SAP BTP environment to choose, this blog post will dive into the technical aspects of deploying a CAP application on the SAP BTP, Kyma runtime.

Full disclosure as usual: I'm not exactly a Kubernetes pro, especially not in a production setting or within the context of SAP. My experience has mostly been limited to playing around with hosting dedicated GitHub Actions runners for small projects or doing "Hello World" stuff. But, the recent changes in the CAP on Kyma movement caught my eye because it involves a new approach to deployment and service binding. That's why I decided to dive a little deeper and learn more about it.

Common Ground: SAP BTP, Cloud Foundry runtime and SAP BTP, Kyma runtime

I'm not here to delve into the differences between SAP BTP, Kyma runtime and SAP BTP, Cloud Foundry runtime or to argue for one over the other. Instead, my focus is on the technical aspects of deploying a CAP application to Kubernetes.

Kubernetes? Yes, the SAP BTP, Kyma runtime is essentially "a fully managed Kubernetes runtime based on the open-source project "Kyma". This cloud-native solution allows the developers to extend SAP solutions with serverless Functions and combine them with containerized microservices. The offered functionality ensures smooth consumption of SAP and non-SAP applications, running workloads in a highly scalable environment, and building event- and API-based extensions", so the service description in the SAP Discovery Center.

To gain a better understanding, I recommend checking out available resources such as the SAP Community Call with piotr.tesny and jamie.cawley titled "Understand the value proposition of SAP BTP, Kyma runtime by example," or the "2 Minutes of Cloud Native" playlist by my friend and former colleague, kevin.muessig. You could also start with some well-known Kubernetes courses.

Before discussing the differences between the runtimes, let's first highlight their similarities. Both runtimes operate on the SAP Business Technology Platform and have an entry point in the platform, making this their commonality.

At first glance, both platforms may appear similar, as they provide an abstraction layer for developers to focus on application development and deployment, without the need to worry about the underlying infrastructure. Additionally, both platforms support containerization, allowing applications to be packaged in a consistent and portable way, making them easier to manage and deploy.

Deployment differences: SAP BTP, Cloud Foundry runtime and SAP BTP, Kyma runtime

Let's talk about the deployment differences between SAP BTP, Cloud Foundry runtime and SAP BTP, Kyma runtime. Obviously, the command line tools to interact with the environments are different. While the Cloud Foundry CLI was the go-to tool to interact with the SAP BTP, Cloud Foundry runtime, kubectl is the tool you'll need to get cozy with in the world of SAP BTP, Kyma runtime.

So, let's quickly recap how you'd deploy a CAP application to SAP BTP, Cloud Foundry runtime:

Develop your CAP Full-Stack application (often underrated in the whole process 😉)

Build an mta.yaml deployment descriptor with the help of cds add mta

Create an MTA archive with the Cloud MTA Build Tool: mbt build

Deploy the MTA archive using a CF CLI plugin called multiapps: cf deploy

The MTA (Multi-Target Application) is technically just a ZIP file that the SAP BTP, Cloud Foundry runtime (specifically the deploy service) picks up and generates the container for you. The platform takes care of managing the service instances, like creating, updating, and deleting them as needed. As a developer, all you have to do is provide a properly structured ZIP file - the MTA archive. The SAP BTP, Cloud Foundry runtime and its components create the container and the image (Diego cells and droplets) automatically for you.

The deployment of applications to Kubernetes-based environments (like SAP BTP, Kyma runtime) is a little different but provides more flexibility. However, as they say, with great power comes great responsibility. In a nutshell:

Develop your CAP Full-Stack application

Add a Helm Chart configuration for a common CAP application with the help of cds add helm

Configure the right values for the provided Helm Chart of a common CAP application

Create the needed Docker Images with the help of Paketo: pack build

Install the Helm Chart to the Kubernetes Cluster: helm install

Paketo, Helm? You might be wondering about Paketo and Helm. Don't worry; we'll cover that in a bit. The main difference is that with Kubernetes-based environments, you're responsible for creating the application (Docker) image and not any platform. Kubernetes picks up the Docker images and creates the container, while Cloud Foundry takes care of the entire process from the MTA archive to running a container.

At first, this might sound complicated (it did for me!) but it's not as tough as it seems. Let's dive into the two tools, Paketo and Helm, that will help with your deployment tremendously.

What's Paketo?

Before we dive into Paketo in detail, let's take a step back to better understand where it's coming from. When a developer deploys an application to Cloud Foundry, the container and image to run the application are created by Cloud Foundry, using buildpacks. So, what exactly are buildpacks? According to the official Cloud Foundry documentation: The official Cloud Foundry documentation states the following:

"Buildpacks provide framework and runtime support for apps. Buildpacks typically examine your apps to determine what dependencies to download and how to configure the apps to communicate with bound services.

When you push an app, Cloud Foundry automatically detects an appropriate buildpack for it. This buildpack is used to compile or prepare your app for launch."

In short: You provide the application. Cloud Foundry does the rest to figure out what's needed and creates the container and image to run your stuff.

Kubernetes, on the other hand, relies on Docker images to run applications or workloads in general. This means that the responsibility shifts to you, as the developer, to build the Dockerfile. While this is manageable, there are more convenient ways. And that's where Paketo comes in. Paketo provides buildpacks without actually using Cloud Foundry.

Built by the Cloud Foundry buildpacks team, Paketo allows you to build Docker images with a single command and without any additional configuration. Using the Paketo CLI, called pack, you can quickly create Docker images for your application. Want to see it in action? Here we go:

pack build ghcr.io/maxstreifeneder/verificationapp-srv:test --path gen/srv \ 

--builder paketobuildpacks/builder:base \

--publish

The command "pack build" is used to create a Docker image. The command specifies the following options:

ghcr.io/maxstreifeneder/verificationapp-srv:test is the name and tag of the Docker image that will be created.

--path gen/srv specifies the path where the application artifacts are located (cds build --production has generated the needed artefacts in there)

--builder paketobuildpacks/builder:base specifies which builder to use for the image. In this case, the "base" builder is being used from the Paketo Buildpacks project. (basically, which base OS to pick)

--publish specifies that the image should be published to the specified container registry after it is built.

The command will analyse the contents of the specified path ("gen/srv"), and will use the Paketo builder to determine which dependencies are required to run the application. Once the dependencies have been determined, the necessary packages will be installed and the Docker image will be created.

After the image is created, it will be tagged with "ghcr.io/maxstreifeneder/verificationapp-srv:test" and published to the specified container registry.

Initially, pack analyses the type of project that needs to be handled:

pack cli detected which buildpacks are needed

Given that the analysis has determined this to be a node-based project, the pack CLI will then examine the package.json file to determine the specific version of Node.js that needs to be installed:

After determining that this is a node-based project, pack examines the package.json file to determine which version of node needs to be installed. There are various ways to install dependencies, and pack ensures to choose the most suitable one based on the available files and directories. For example, it checks if a node_modules directory already exists with the dependencies installed, or if npm ci is a viable option due to the presence of a package-lock.json file.

pack cli detected npm install as the best dependency installation process

While there are various aspects to consider, such as the configuration of the operating system and setting environment variables, it can be unclear what the resulting Docker image will look like. To shed some light on this, I like to reverse engineer things whenever possible. But how can you reverse engineer a Docker image to its individual layers or even a Dockerfile? Although Paketo's methodology may pose some limitations, you can still deconstruct the image into its layers using tools like dive.

It's relatively simple:

dive image-name:image-tag

dive tool in action

In the GIF above, you can see the layers of the image on the left side and the impact of each layer on the file system (highlighted in yellow) on the right side. This provides a good idea of what Paketo does in general and gives one the opportunity to further inspect the built Docker image - for instance its different layers. If you find the previous method too detailed or excessive, pack CLI also offers a simpler alternative to inspect the built Docker image at a higher level:

pack inspect-image <image-name>:<tag>

pack inspect to see the buildpack details (also remote vs. local)

You can use this feature to view the list of buildpacks that have been identified and will be included in the Docker image you have built. This Docker image can then be deployed to the SAP BTP, Kyma runtime.

All of these Docker images will later on end up in your values.yaml file for the Helm chart (next paragraph 😉) and are picked up by the SAP BTP, Kyma runtime to run the application.

You can apply the same approach to deploy SAP HANA artifacts: use the pack CLI to build the corresponding Docker image and then include its reference in the values.yaml file for the Helm installation.

Installation of pack cli: https://buildpacks.io/docs/tools/pack/
Paketo homepage: https://paketo.io/
pack build: https://buildpacks.io/docs/tools/pack/cli/pack_build/
pack inspect: https://buildpacks.io/docs/tools/pack/cli/pack_inspect/
dive tool: https://github.com/wagoodman/dive

What's Helm?

"The package manager for Kubernetes", so the short description on helm.sh.

Helm is a package manager for Kubernetes, which makes it easy to install, upgrade, and manage applications and services in a Kubernetes cluster. When it comes to SAP Cloud Application Programming Model, Helm can be used to package and deploy CAP applications to a Kubernetes environment like SAP BTP, Kyma runtime.

Using Helm to deploy CAP applications to Kubernetes offers numerous benefits. Firstly, it simplifies the deployment process by automating the installation and configuration of applications and services, reducing the amount of manual work required. Additionally, it enables version control, so you can track changes and rollback to previous versions if necessary.

Deploying a CAP application without Helm would typically involve writing a complete Kubernetes manifest to create the corresponding resources. However, using Helm's templating function, you can avoid doing that for every single CAP application. Do you want to do that for every single CAP application? My rough guess: Most likely not. The CAP team has already taken good care of the common configuration with the help of Helm template functions and much more, so you only need to focus on the specifics for your application.

Here's a concrete example:

Let's take a values.yaml file in one of my sap-samples repositories. The values.yaml file only contains a basic configuration about the application itself and which service instances and bindings are needed and is the key file in your deployment configuration. With helm install you could already install this chart on your SAP BTP, Kyma runtime but you won't really find out what exactly you are deploying. I wanted to know more. I know that Kubernetes resource definitions are by far more complex than this particular values.yaml file for Helm - so where's the magic happening that generates a Kubernetes resource out of that Helm chart? Here's my Helm chart (Link😞

values.yaml file

When you run cds add helm, it generates more than just the values.yaml file for your CAP application. It also creates several .tpl files that contain template helpers/functions used to generate various Kubernetes/Kyma resoucres, such as APIRules and Deployments, in addition to the basic CAP application.

To see the complete deployment configuration, you can use helm template <directory>. This command takes the template helpers (.tpl files) into account and renders the chart to standard Kubernetes resource templates. You could basically use these templates with kubectl apply -f <helm-template-result-file> without using Helm, but you won't get the benefits of Helm. I've shared the results of running "helm template" here.

part of the result for helm template

Why write hundreds of lines of Kubernetes resource definitions when you can do it all with just a few dozen lines in a Helm chart for your CAP application? That's the power of using Helm. And if you've already installed a Helm chart and want to see what you've got, just use helm get manifest <release name> to get the full installation manifest. It's that simple!

part of the result for helm get manifest

Adding a new service instance is as easy as creating a new key in the values.yaml file and filling in the required properties (such as serviceInstanceFullname, serviceOfferingName, servicePlanName, etc.).

If you're familiar with SAP BTP, Cloud Foundry runtime, you'll recognise these properties, as ServiceInstances and ServiceBindings work on the same principle in both environments - thanks to the SAP BTP Service operator!

You can find a list of possible properties for creating ServiceInstances and ServiceBindings in the GitHub repository of the SAP BTP service operator.

Lastly, create a new .yaml file with the name of the key you provided in the values.yaml file. The file's content is simply "{{- include "cap.service-instance" . }}", and one of the Helm template helpers provided by CAP will pick it up and recognise it as a ServiceInstance and ServiceBinding.

template/_helpers.tpl picks up the yaml files

How the required yaml file per ServiceInstance should look like

In other words, the name you choose for the key in the values.yaml file to define a service instance can be anything you like, as long as you create a corresponding file with that name in the directory, and include the phrase I mentioned earlier. The whole process of using template functions, different yaml files, and getting the desired results can be a bit overwhelming at first, but once you've got the hang of it... 😉

Helm flow (source: opensource.hcltechsw.com)

Overall, using Helm to deploy CAP applications to Kubernetes can save time and effort, improve consistency and reliability and makes it easier to manage.

How do I deploy my application now?

Create a namespace in your Kyma cluster. (kubectl create namespace)

Make sure that you have created your SAP HANA Cloud instance before and have mapped it to your namespace using the SAP HANA Cloud tools: https://blogs.sap.com/2022/12/15/consuming-sap-hana-cloud-from-the-kyma-environment/

Add a Helm chart and the required helpers with cds add helm

Use Paketo to build the Docker images for your db and srv artefacts

Deploy the UI bits to the HTML5 Application Repository with a provided Docker image from the responsible team: sapse/html5-app-deployer. An example of how that could work is part of the sample repository I've provided: Dockerfile and preparation scripts

In case you are publishing your Docker images to a private Container registry, make sure you create secret in your Kyma cluster and provide in the Helm chart

Use Helm upgrade <release name> <chart directory> --install --debug to start the deployment (in helm jargon installation)

That's it in a nutshell.

Helm: https://helm.sh/
Helm template functions: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/
CAP with Helm: https://cap.cloud.sap/docs/guides/deployment/deploy-to-kyma#deploy-using-cap-helm-chart

Can I use the Cloud Connector with SAP BTP, Kyma Runtime?

Yes, you can use the Cloud Connector with SAP BTP's, Kyma Runtime, but the Connectivity Service operates slightly different than for SAP BTP's, Cloud Foundry runtime. In Kyma, the Connectivity Proxy (read more about it in one of my previous blog posts) is created per cluster, whereas in Cloud Foundry, it already exists per region.

The Kyma reconciler, which is a scheduled job that runs frequently, checks if you have created a ServiceInstance and ServiceBinding for the Connectivity Service. The Connectivity Proxy will be provisioned into the kyma-system namespace in the runtime but only once - so don't create more than one ServiceInstance and ServiceBinding for the Connectivity service! It can be accessed from within the Kyma runtime using the URL connectivity-proxy.kyma-system.svc.cluster.local:20003. By using cds add connectivity, the Helm chart is updated with the necessary secrets for the Connectivity Proxy as volumes, so that the CAP application knows how to access the on-premise system through the SAP Cloud Connector using the Connectivity Service and its Connectivity Proxy.

connectivity-proxy-0 is the Pod name in namespace kyma-system

When creating the ServiceInstance and ServiceBinding for the Connectivity Service, it's crucial to ensure that this process isn't part of any app deployment, as it's a one-time job and secret rotation (caused by multiple ServiceBindings) could cause inconsistencies for the Connectivity Proxy Pod. Additionally, it's important to make sure that only one connectivity-proxy Pod exists in the kyma-system namespace, as this can impact connectivity to the on-premise system. If issues arise, the Container Logs of this Pod can provide useful insights.

Important: It's also worth noting that Istio injection must be enabled in the namespace where the ServiceInstance and ServiceBinding are created, which can be done with the command

kubectl label namespace <namespace> istio-injection=enabled

How does CAP know where to get the service binding information from?

On a slightly different note, as someone who has worked with the SAP BTP, Cloud Foundry runtime for quite some time, I'm well acquainted with VCAP_SERVICES. This environment variable is set for each app instance and contains all service binding information as a JSON. Many SDKs heavily rely on fetching this information to establish connections with specific backing services. Initially, in the early days of SAP BTP, Kyma runtime, and CAP, faking the VCAP_SERVICES environment variable was necessary to allow CAP to access the necessary credentials to connect to other services.

However, thanks to the SAP BTP Service operator, this is no longer necessary. Secrets and ConfigMaps are now automatically provided as volumes, eliminating the need to fake the VCAP_SERVICES environment variable.

the deployment configuration after running helm template

Ok, fair. Since I always like to understand how things work actually - where's the piece of code that differentiates between apps running on SAP BTP, Cloud Foundry runtime and apps running on SAP BTP, Kyma runtime? Basically, if there's no VCAP_SERVICES it's going to read the service information from the mounted volume path:

As I said, little excursion 😉

SAP Event Mesh and SAP S/4HANA on-premise - little detour needed

If you need to receive events from an SAP S/4HANA on-premise system in your CAP application, there are several configuration steps and artefacts required on the SAP S/4HANA on-premise side to enable this connection. Fortunately, there's a simplified process offered through a dedicated transaction called /IWXBE/CONFIG.

In essence, you just need a service key and copy its contents to this transaction, and all the necessary artefacts are automatically created for you. The service key contains information about authorization, applied namespaces, and effective endpoints for the SAP Event Mesh instance. However, this only applies to service keys created in the SAP BTP, Cloud Foundry runtime. For the SAP BTP, Kyma Runtime, which uses Kubernetes Secrets as its service key equivalent, an additional step is required to transform it into the correct format.

Secret for the SAP Event Mesh service binding

jq to the rescue! Since the output of the above shown secret can be formatted as json, we can make use of jq to transform it into the right format again. The jq call is slightly too complex for a one-liner (for instance not all of the properties of that secrets are needed, base64 decoding is required, etc.), that's why I've put into a dedicated file:

#!/usr/bin/env jq



def props: "^(management|messaging|tags|uaa)$";



.data | with_entries(select(.key != ".metadata") |

  .value|=@base64d

  | if .key | test(props) then .value |= fromjson else . end

)

So, to get the right JSON for the SAP S/4HANA on-premise transaction, you can simply execute:

kubectl get secrets/your-messaging-secret -o json | jq -f ./kyma/eventmeshkey.jq

As usual, the above shown jq file is also part of the repository I'm currently working on: Enhance core ERP business processes with resilient applications on SAP BTP

Mind your DB Pool configuration!

I didn't want to dedicate a whole paragraph to it, since I can barely explain the background - my application didn't contain a pool configuration. It worked on SAP BTP, Cloud Foundry but it didn't on SAP BTP, Kyma runtime. Please make sure that you provide a proper DB Pool configuration, otherwise your CAP application might not be able to establish connections to your SAP HANA Cloud instance and something like this might appear in your application logs:

TimeoutError: Acquiring client from pool timed out. Please review your system setup, transaction handling, and pool configuration. Pool State: borrowed: 0, pending: 0...

I have no idea why that happens on SAP BTP, Kyma runtime and not on SAP BTP, Cloud Foundry (same application, just packaged differently - Paketo instead of MTA), but my rough guess would be that it has to do with the buildpacks being used and a different default configuration for the underlying npm module generic-pool. I couldn't figure it out, I'm sorry. 😉

Thank you so much for taking the time to read my blog post! Although I'm a big fan of the simplicity of the SAP BTP, Cloud Foundry runtime, I'm thoroughly enjoying my experience with SAP BTP, Kyma runtime. This post is not meant to be a pro-contra comparison, but rather a collection of my thoughts and observations while working on some sample repositories.

I welcome any comments and feedback you may have, whether it's questions, personal experiences, or just your thoughts. Let's continue the conversation in the comments!

SAP Data Intelligence Python Operators and Cloud Connector – TCP

2023-04-14T09:20:38+02:00

The Transmission Control Protocol (TCP) is a widely used protocol that provides a reliable and ordered delivery of data between applications running on different hosts. It serves as the foundation for many technologies and plays a crucial role in modern IT infrastructure.

SAP Data Intelligence is a powerful platform that allows you to integrate various systems and data sources to gain insights and make informed decisions. Its custom Python operators offer even more flexibility, enabling you to connect to systems beyond what the standard shipped operators support.

In this blog post, we'll guide you through the steps to establish connectivity to an on-premises TCP-based system using Python in SAP Data Intelligence.

Access TCP-based resources:

Scenario: We aim to establish a connection with a TCP-based system using the Python operator in SAP Data Intelligence. Many databases but also other protocols like SFTP etc. are based on TCP. To achieve this, we will configure the cloud connector, which involves a series of steps. For the purposes of this demonstration, we will use an example of a "Hello TCP" server that is running on my local computer. The ultimate objective is to access this server from within SAP Data Intelligence using the Cloud Connector.

Szenario Architecture

Prerequisite:

Before proceeding with the steps outlined in this guide, it is essential to have an instance of the SAP Cloud Connector installed. While it is possible to install the cloud connector on a server, for the purposes of this demonstration, we will be using a Windows machine. We recommend following the instructions provided in this blog (https://blogs.sap.com/2021/09/05/installation-and-configuration-of-sap-cloud-connector/) to install and configure the cloud connector.

The second requirement is a BTP subaccount with a Data Intelligence cluster. To this Subaccount we will connect the Cloud Connector.

1. Configuration:

The first step is to create a configuration in the Cloud Connector that connects to our subaccount and exposes the TCP resource. For the purpose of this demonstration, I ran a small Node.js server on my Windows machine that outputs "Hello TCP!".

telnet localhost 4444

Connecting to localhost ...

Hello TCP!

To create the configuration, navigate to the admin interface for the cloud connector and create a "Cloud to On-Premise" configuration.

Cloud Connector configuration

In the screenshot above, you can see that I exposed the internal host "localhost" with port 4444 via a virtual host called "virtualhost". This virtual host is the host that we will be requesting from the BTP side. Compared to HTTP resources, we do not need to explicitly expose paths.

BTP Cockpit Cloud Connector Resources

On the BTP end, we can check the cockpit and the connected cloud connectors in the respective menu tab. If you cannot see this tab, you may be missing some roles. It is important to note that we see the LocationID "FELIXLAPTOP", which is an identifier that distinguishes multiple cloud connectors connected to the same subaccount.

2. Creating a Data Intelligence Connection:

For our purposes, we do not want to hard-code the connection details, because we need a little help from the connection management to access the Connectivity Service of BTP. In the Connection Management application from SAP Data Intelligence we can create connections of all types. We create a connection of type HTTP with host, port and SAP Cloud Connector as the gateway.

Note: Not all connection types allow you to access via the Cloud Connector. See the official product documentation for details.

3. Developing a Custom Operator:

In the operators menu of Data Intelligence we create a new custom operator based on the Python3 operator.

Creating Custom Python Operator

We than change the configSchema.json of this operator to accept an HTTP Connection as a parameter. This file can be found in the repository under the following path.

Note: This is a bit hacky. Data Intelligence proxies the Connectivity Service from the BTP internally. We can reach it using the internal host provided in the connection details ("connectivity-proxy-service") of a HTTP Connection (because this connection supports the SAP Cloud Connector as a gateway - other connection types might work as well). It is important to notice that the behavior of this proxy is exactly the same as that of the Connectivity Service. In other environments, like Cloud Foundry, you can bind an instance of the Connectivity Service to your application and access the service that way. Data Intelligence also helps us with the authentication flow; in the connection details, we will find a valid JWT token to authenticate against the Connectivity Proxy.

vflow\subengines\com\sap\python36\operators\PythonCloudConnector\configSchema.json

The JSON file looks like this:

{

    "$schema": "http://json-schema.org/draft-06/schema#",

    "$id": "http://sap.com/vflow/PythonCloudConnector.configSchema.json",

    "type": "object",

    "properties": {

        "codelanguage": {

            "type": "string"

        },

        "scriptReference": {

            "type": "string"

        },

        "script": {

            "type": "string"

        },

        "http_connection": {

            "title": "HTTP Connection",

            "description": "HTTP Connection",

            "type": "object",

            "properties": {

                "configurationType": {

                    "title": "Configuration Type",

                    "description": "Configuration Type",

                    "type": "string",

                    "enum": [

                        " ",

                        "Configuration Manager",

                        "Manual"

                    ]

                },

                "connectionID": {

                    "title": "Connection ID",

                    "type": "string",

                    "format": "com.sap.dh.connection.id"

                },

                "connectionProperties": {

                    "title": "Connection Properties",

                    "description": "Connection Properties",

                    "$ref": "http://sap.com/vflow/com.sap.dh.connections.http.schema.json",

                    "sap_vflow_constraints": {

                        "ui_visibility": [

                            {

                                "name": "configurationType",

                                "value": "Manual"

                            }

                        ]

                    }

                }

            }

        }

    },

    "required": [

        "http_connection"

    ]

}

In the Python Operator we will be using the sapcloudconnectorpythonsocket library I created for this purpose. This needs to be installed in a custom Dockerfile. The reason we cannot use standard librarys like PySocks to connect to the connectivity service, is the custom authentication flow used.

This post will not elaborate further on how to link the Dockerfile to the Custom Operator using tags.

FROM $com.sap.sles.base



RUN python3 -m pip --no-cache-dir install 'sapcloudconnectorpythonsocket' --user

And finally, the actual script looks straightforward:

from sapcloudconnectorpythonsocket import CloudConnectorSocket



tcp_connection = api.config.http_connection



api.logger.info(str(tcp_connection))



def gen():

    api.logger.info("Generator Start")

    

    cc_socket = CloudConnectorSocket()

    cc_socket.connect(

        dest_host=tcp_connection["connectionProperties"]["host"], 

        dest_port=tcp_connection["connectionProperties"]["port"], 

        proxy_host=tcp_connection["gateway"]["host"], 

        proxy_port=20004, 

        token=tcp_connection["gateway"]["authentication"],

        location_id="FELIXLAPTOP"

    )

    

    cc_socket.send(b"")

    

    response = cc_socket.recv(4096)

    api.logger.info(response)

    

    api.logger.info("Generator End")



api.add_generator(gen)

First we take the connection details from the api.config.http_connection object. Then we hand those credentials to the library to connect a socket via the Cloud Connector proxy to our destination server. The logs reveal the various fields that are accessible.

{

   "configurationType":"Configuration Manager",

   "connectionID":"TCP_ONPREM_SYSTEM",

   "connectionProperties":{

      "authenticationType":"NoAuth",

      "host":"virtualhost",

      "port":4444,

      "protocol":"HTTP"

   },

   "connectionType":"HTTP",

   "gateway":{

      "authentication":"<token>",

      "host":"connectivity-proxy-service",

      "locationId":"",

      "password":"",

      "port":20003,

      "subaccount":"2343242-b1ea-415c-9a50-46347c32fa2a",

      "user":""

   },

   "type":"HTTP"

}

One thing to keep in mind here. Instead of the standard HTTP proxy port of the connectivity service, we use the SOCKS5 proxies port. Looking at a regular service key of a connectivity service instance will make things clearer:

{

    "clientid": "<clientid>",

    "clientsecret": "<clientsecret>",

    "url": "https://<subaccount>.authentication.sap.hana.ondemand.com",

    "identityzone": "<subaccount>",

    "tenantid": "<tenantid>",

    "tenantmode": "dedicated",

    "verificationkey": "<verificationkey>",

    "xsappname": "iueniuerncoirencorencercer!b3008|connectivity!b137",

    "uaadomain": "authentication.sap.hana.ondemand.com",

    "credential-type": "binding-secret",

    "onpremise_proxy_host": "connectivityproxy.internal.cf.sap.hana.ondemand.com",

    "onpremise_proxy_http_port": "20003",

    "onpremise_proxy_ldap_port": "20001",

    "onpremise_proxy_port": "20003",

    "onpremise_proxy_rfc_port": "20001",

    "onpremise_socks5_proxy_port": "20004",

    "token_service_domain": "authentication.sap.hana.ondemand.com",

    "token_service_url": "https://<subaccount>.authentication.sap.hana.ondemand.com"

}

We find the classic XSUAA style clientid, clientsecret and token_service_url to retrieve a token in the client credential flow (Data Intelligence does that for us). Plus, one can also see the different ports exposed by the Connectivity Service.

HTTP: 20003

SOCKS5/TCP: 20004

This is why we specify the SOCKS5 port specifically in the opening of the socket.

With the successful opening of a socket, we are now able to send requests to the TCP server. In the example I am sending a request with an empty body, my local server returns "Hello Tcp!" to every request.

4. Testing the Custom Operator:

Finally we can put the new operator in a empty graph and fill the http_connection parameter. In this instance, I provide the Connection ID of the connection I created previously.

Python Operator Parameters

When executing the graph we can have a look at the console and see the following entries:

Generator Start

Hello TCP!

Generator End

That means we were successful and the operator was able to request the local TCP server on my windows machine.

Hope you find the content of this blog helpful. Feel free to comment for further clarifications.

SFTP via Cloud Connector Python Operator in SAP Data Intelligence

2023-04-14T13:00:08+02:00

The (Secure) File Transfer Protocol is still a very common way to integrate files from different sources. SAP Data Intelligence supports many source systems for file operations out of the box. To allow for even more flexibility in the connection to SFTP servers, this blog post shows how to use the Python library Paramiko to read, write, list or delete files on remote sources. Even through the infamous Cloud Connector.

Connect Python Operator to SFTP via Cloud Connector:

Szenario: This blog post aims at establishing a connection to an on-premises SFTP server. We will show how to establish a TCP socket and use Paramiko to read, list, write or delete files. The TCP connection socket is the basis for this blog and I recommend my previous blog about the basics of establishing a TCP Socket from a python custom operator and a detailed explaination on the role of the Connectivity Service. https://blogs.sap.com/2023/04/14/sap-data-intelligence-python-operators-and-cloud-connector-tcp/

Szenario Architecture

Prerequisite:

All prerequisites from the previous blog post apply.

For this case, we additionally need a SFTP server running on the same host as the Cloud Connector. To establish a local test setup on my Windows machine, I use a very simple dockerized SFTP server. It exposes a SFTP server on the localhost on port 22. With the user foo and the password pass we are allowed to interact with the contents of the folder /upload.

docker run -p 22:22 -d atmoz/sftp foo:pass:::upload

1. Configuration:

The very first step in the integration is to configure the Cloud Connector to expose the SFTP server to the respective BTP subaccount. The configuration looks as follows:

Cloud To On-premises Configuration

The localhost:22 is exposed to a virtual host that we can see in the BTP Cockpit.

BTP Cockpit Cloud Connector Resources

2. Creating a Data Intelligence Connection:

For our purposes, we do not want to hard-code the connection details, because we need a little help from the connection management to access the Connectivity Service of BTP. In the Connection Management Application from SAP Data Intelligence we can create connections of all kinds of types. We create a connection of type HTTP with out host, port and we specify SAP Cloud Connector as gateway.

Data Intelligence Connection Management

Note: Not all connection types allow you to access through the Cloud Connector. See the official product documentation for details.

3. Developing a Custom Operator:

We use Paramiko and the sapcloudconnectorpythonsocket library. They need to be installed in a Dockerfile and added to the custom operator. Paramiko is a python library that helps to communicate with the SFTP server. The sapcloudconnectorpythonsocket library helps us to open a socket via the Cloud connector.

FROM $com.sap.sles.base



RUN python3 -m pip --no-cache-dir install 'sapcloudconnectorpythonsocket' --user

RUN python3 -m pip --no-cache-dir install 'paramiko' --user

Finally, let's have a look at the actual custom script. The first important part is opening the TCP socket as described in the previous blog. This TCP socket can be passed to Paramiko SSHClient to establish the connection. Since we do keep this really simple we put some Paramiko policies in to ignore the missing host keys. With the open ftp client, we can interact with the file server.

For the purpose of demonstration, we create a file in the /upload directory, called test_create_file.txt. We then create a new directory, list the contents of the /upload directory, read the content of the created file and remove it again.

Paramiko can do a lot more than the shown file operations, it implements the SSHv2 Protocol and can be used to execute shell commands remotely. Thus opening a big door for hacky integration scenarios - even fully on-prem 🙂 🙂

Check out the Paramiko documentation for more details.

import paramiko

import io

from sapcloudconnectorpythonsocket import CloudConnectorSocket



sftp_connection = api.config.http_connection



api.logger.info(str(sftp_connection))



def gen():

    api.logger.info("Generator Start")

    

    cc_socket = CloudConnectorSocket()

    cc_socket.connect(

        dest_host=sftp_connection["connectionProperties"]["host"], 

        dest_port=sftp_connection["connectionProperties"]["port"], 

        proxy_host=sftp_connection["gateway"]["host"], 

        proxy_port=20004, 

        token=sftp_connection["gateway"]["authentication"],

        location_id="FELIXLAPTOP"

    )

    

    ssh_client = paramiko.SSHClient()

    ssh_client.load_system_host_keys()

    ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())



    ssh_client.connect(

        sftp_connection["connectionProperties"]["port"], 

        username=sftp_connection["connectionProperties"]["user"],

        password=sftp_connection["connectionProperties"]["password"],

        sock=cc_socket

    )



    sftp = ssh_client.open_sftp()

    

    api.logger.info("SFTP Client open")

    

    file = sftp.file("/upload/test_create_file.txt", "a", -1)

    file.write("Hello SFTP.!\n")

    file.flush()



    sftp.mkdir("/upload/testdir")



    dir_objects = sftp.listdir_attr("/upload")

    file_names = [e.filename for e in dir_objects]

    api.logger.info(file_names)

    

    file_obj = io.BytesIO()

    sftp.getfo("/upload/test_create_file.txt", file_obj)

    api.logger.info(str(file_obj.getvalue()))

    

    sftp.remove("/upload/test_create_file.txt")



    api.logger.info("Generator End")

    

    api.send("file", api.Message(file_obj.getvalue(), {"file":{"path": f"/shared/pythoncloudconnector/test_create_file.txt"}, "file_name": "test_create_file.txt"}))





api.add_generator(gen)

4. Testing the Custom Operator:

To test the new custom SFTP operator, we create a Data Intelligence graph. In the example, I added an output port of type message.file to the custom operator. It connects to the write file operator, which is configured to write to the DI_DATA_LAKE. For the SFTP operator we add the Connection ID we created before as a parameter.

Data Intelligence Graph

In the logs of the Graph execution, we can see how the file and directory are created on the SFTP server. Additionally, we can also retrieve the file content "Hello SFTP.!".

Generator Start 

SFTP Client open 

['testdir', 'test_create_file.txt'] 

b'Hello SFTP.!\n' 

Generator End

At last we can check the Metadata Explorer and see the newly imported file.

Looking at the file content, we can see it worked well! We are able to connect to the SFTP server and integrate files with the standard operators.

Hello SFTP.!

Hope you find the content of this blog helpful. Feel free to comment for further clarifications.

Proxy Third-Party Python Library Traffic - Using SAP Cloud Connector in SAP Data Intelligence Python Operators

2023-04-24T12:50:54+02:00

Custom Python operators are the most convenient way to develop non-standard integrations in SAP Data Intelligence. The vast range of Python libraries available is virtually unlimited. In this blog post, I will demonstrate how to proxy TCP traffic of third-party libraries through the SAP Cloud Connector, enabling the integration of on-premises systems to be as effortless as internet-accessible ones.

Please note that this blog post is a continuation of my previous post, "Proxy Third-Party Python Library Traffic - General", where I provided a more comprehensive explanation of the approach and libraries used.

Szenario Architecture

Szenario: In this example, I will show how to connect to a PostgreSQL database via the Cloud Connector. The database is running on my Windows machine, and the Cloud Connector is connected to that machine. This replicates a real world on-premises setup.

Prerequisite:

1. Configuration:

The first step is to create a configuration in the Cloud Connector that connects to our subaccount and exposes the PostgreSQL host as a TCP resource.

Cloud Connector configuration

On the BTP end, we can check the cockpit and the connected cloud connectors in the respective menu tabs. If you cannot see this tab, you may be missing some roles. It is important to note that we see the LocationID “FELIXLAPTOP”, which is an identifier that distinguishes multiple cloud connectors connected to the same subaccount.

Registered Cloud Connector Resources

2. Creating a Data Intelligence Connection:

For our purposes, we do not want to hard-code the connection details, because we need a little help from the connection management to access the Connectivity Service of BTP. In the Connection Management application from SAP Data Intelligence we can create connections of all types. We create a connection of type HTTP with host, port and SAP Cloud Connector as the gateway.

In there we specify the internal host of the Cloud Connector resource, its port and the Postgres username and password.

Connection with Cloud Connector Gateway

Note: Not all connection types allow you to access via the Cloud Connector. See the official product documentation for details.

3. Developing a Custom Operator:

In the operators menu of Data Intelligence we create a new custom operator based on the Python3 operator.

We build a custom Dockerfile with the required libraries. We use psycopg2 to connect to Postgres, SocketSwap as a local proxy, and sapcloudconnectorpythonsocket to open a socket via the Cloud Connector.

FROM $com.sap.sles.base



RUN python3 -m pip --no-cache-dir install 'psycopg2-binary' --user

RUN python3 -m pip --no-cache-dir install 'sapcloudconnectorpythonsocket' --user

RUN python3 -m pip --no-cache-dir install 'SocketSwap' --user

Now to the key part, the custom script:

import psycopg2

from SocketSwap import SocketSwapContext

import multiprocessing

from operators.<your_operator>.socket_factory import socket_factory



multiprocessing.set_start_method("spawn")   # needed because DI Python Operator is itself executed in Threads



postgres_connection = api.config.http_connection



api.logger.info(str(postgres_connection))





def connect_postgres():

    """

    This function demos how to easily setup the local proxy using the SocketSwapContext-Manager.

    It exposes a local proxy on the localhost 127.0.0.1 on port 2222

    The connection factory is provided to handle the creation of a socket to the remote target

    """

    

    with SocketSwapContext(socket_factory, [postgres_connection], "127.0.0.1", 2222):

        api.logger.info("Proxy started!")

        # Set up a connection to the PostgreSQL database

        conn = psycopg2.connect(

            host="127.0.0.1",

            database="postgres",

            user="postgres",

            password="password",

            port=2222

        )

        api.logger.info("Postgres connected successfully!")



        # Create a cursor object to execute SQL queries

        cur = conn.cursor()



        # Execute a SELECT query to retrieve data from a table

        cur.execute("SELECT CURRENT_TIMESTAMP;")



        # Fetch all the rows returned by the query

        rows = cur.fetchall()



        # Print the rows to the console

        for row in rows:

            api.logger.info(str(row))



        # Close the cursor and connection

        cur.close()

        conn.close()





def gen():

    api.logger.info("Generator Start")

    

    connect_postgres()





api.add_generator(gen)

And we need a second file in which we put the function we are passing into the SocketSwap Proxy. Note: This function is placed in a seperate file on the top level, to make it pickleable.

from sapcloudconnectorpythonsocket import CloudConnectorSocket



def socket_factory(postgres_connection):

    cc_socket = CloudConnectorSocket()

    cc_socket.connect(

        dest_host=postgres_connection["connectionProperties"]["host"],      # virtualhost

        dest_port=postgres_connection["connectionProperties"]["port"],      # 5432

        proxy_host=postgres_connection["gateway"]["host"],                  # connectivity-service-proxy

        proxy_port=20004,                                                   # 20004 SOCKS5 Proxy Port of connectivity service

        token=postgres_connection["gateway"]["authentication"],             # auth token

        location_id="FELIXLAPTOP"

    )

    return cc_socket

In the custom operator, script we just use a generator to start the connect_postgres function. Inside that function, we use the standard psycopg functions to make a small SQL query to the database. The only difference here: We wrap it in the SocketSwapContext to redirect its traffic through a CloudConnectorSocket.

Note: The SocketSwapContext takes mainly 3 arguments: A socket_factory, a callable that returns a connected socket, socket_factory_args, the local proxy host and its port. As we can see in the example, I pass the socket_factory function to the context manager. This function uses the already introduced CloudConnectorSocket functionality to open a Python socket using the Cloud Connector as proxy server. Check out my previous post about TCP and the SAP Cloud Connector to learn more details about this.

The local proxy is setup automatically in a background daemon process and listens to the given host, port. Usually this will be the localhost and a free port. In your third-party library you can then connect to that proxy and its traffic will be redirected automatically to the address specified as destination in the cloud connector socket.

4. Test the custom operator:

We can wrap the new operator in a Data Intelligence Graph and fill the configuration parameter with the POSTGRES_ONPREM_SYSTEM connection ID.

Custom Operator Parameters

In this example, we can check the logs and see the following:

Generator Start 

Proxy started! 

Postgres connected successfully! 

(datetime.datetime(2023, 4, 24, 10, 10, 52, 593842, tzinfo=datetime.timezone.utc),)

The proxy registered successfully and was able to redirect the traffic coming to localhost:2222 to the on-premises PostgreSQL. As a result of the query, we get the current_timestamp in the python datetime object.

I hope this blog gave you a perspective of the possibilities you have even through the Cloud Connector to interact with any system. If you have any questions of scenarios for the SocketSwap package, feel free to leave a comment.

Surviving and Thriving with CAP: Destinations and the Transparent Proxy for Kubernetes

2023-05-04T09:07:02+02:00

In my last blog post, I've written about how I prepared an existing app that was already running on SAP BTP, Cloud Foundry to be deployed on SAP BTP, Kyma runtime. Along the way, I got to know and appreciate some new tools like helm/paketo and kubectl and the entire Kubernetes ecosphere which is quite eye-opening. I had some "touch barrier" for a long time, which may be why it took me a while to dive into the topic more deeply.

jan.schaffner's recent post on LinkedIn drew my attention to the Transparent Proxy for Kubernetes. As usual, I had to take a closer look at the topic and try it out myself to really understand what it is and how it works.

The Transparent Proxy is a software component designed for Kubernetes, which means it can also be used with SAP BTP, Kyma runtime. This tool simplifies the process of calling SAP BTP Destinations for developers. I was impressed with how the Transparent Proxy seamlessly integrates with SAP BTP's, Kyma runtime, and it definitely exceeded my expectations in terms of the benefits it offers.

Transparent Proxy for Kubernetes (help.sap.com)

Before diving into the steps for using the Transparent Proxy and its inner workings, I think it's important to first discuss SAP BTP Destinations in general. While these destinations are incredibly useful, they can also cause challenges if you're not using the correct libraries and SDKs. Luckily, SAP provides various SDKs and frameworks (spoiler: but not for all languages) that make using Destinations on SAP BTP a lot easier. Throughout this process, we'll explore the existing SDKs and frameworks by SAP and how they help us in using Destinations.

It's about CAP because it's not about CAP

You're right, it may seem like this blog post isn't directly related to CAP but once you've chosen a blog series title (#CAPTricks), you have to stick to it, right? 😉 However, that's only half of the truth. In this post, I intentionally set aside CAP code to focus on the real problem that the Transparent Proxy solves. Additionally, by the end of the post, readers will hopefully see (again) how CAP simplifies the use of SAP BTP Destination service as a whole.

To provide some context, CAP aims to simplify many essential and repetitive processes. According to the CAP documentation, it can significantly accelerate development, minimize boilerplate code, and improve overall quality by providing single points to fix and optimize. One such process is connecting to remote services or making API calls, which is precisely what this blog post will cover.

In SAP Business Technology Platform (SAP BTP), there's a service called Destination service that stores information about remote services.

Before I'll explain the concept of the Destination service, how about asking the most prominent assistant these days?

ChatGPT, what's the purpose of SAP BTP Destinations?

The Destination service in SAP BTP provides a centralized way to store information about remote services, including the URL, port number, HTTP headers, and authentication information. This approach offers a significant advantage by allowing updates to the Destination information on SAP BTP, rather than modifying the service information in every single application that wants to connect. This saves time and effort and ensures that all applications are updated at once, resulting in increased quality and minimized errors.

As the SAP BTP Destination service is part of your SAP BTP sub-account, not everyone can connect to it. To use the information about a particular Destination in your application, you first need to go through OAuth2 authentication to retrieve the information from the SAP BTP service. While constructing the subsequent call to the actual remote service is not difficult, it can still be challenging, particularly if the actual API is also secured by OAuth2 or if the API is an on-premise system behind an SAP Cloud Connector or SAP Private Link service. Additionally, token handling, such as refreshing them, is a common task that needs to be tackled.

very high level flow of calling a remote service/API using the SAP BTP Destination service

To handle these challenges, several SDKs and frameworks offer support for Destination handling, including the SAP Cloud SDK and the SAP Cloud Application Programming Model (CAP). CAP probably handles Destinations in the most convenient way.

We'll have a look at different scenarios peu a peu with more help of SDKs or services, but the common ground for all the following paragraphs is the Destination service on SAP BTP. We'll not going to move this out of this scenario. I'll only provide the configuration & code examples for Node.js for the sake of not bloating up this blog post. If you closely look at the endpoint we are using for the example code: Yes, it's basically an OData endpoint that I'm treating as a plain REST endpoint. Also, the examples below do not contain any kind of authentication or SAP Cloud Connector which makes the scenario by far more complex (and realistic, to be honest but less readable in this blog post).

This is the destination we'll be using throughout this blog post:

https://services.odata.org/v4/northwind/northwind.svc/

Destination usage with plain Node.js with @Sap/xsenv

When connecting to a remote service through the SAP BTP Destination service, you need to provide your service instance information, including service credentials, to your application. This can be achieved through various methods:

By providing environment variables using .env / default-env.json files for local development

On SAP BTP, Cloud Foundry runtime: Either via the SAP BTP Cockpit, the CF CLI or in the MTA descriptor

on SAP BTP, Kyma runtime: Via ServiceInstance and a ServiceBinding

This injects the service information (where and how to reach the SAP BTP Destination service and the SAP Connectivity service) either as an environment variable (in case of local development and SAP BTP, Cloud Foundry runtime) or as an mounted Volume for the respective runtime. CAP itself knows how to grab that information and uses it to connect to the SAP BTP Destination service make the subsequent call to the actual remote service.

In the following code, the only SAP-provided helper is the @Sap/xsenv package. This package serves as a "utility for easily reading application configurations for bound services and certificates in the SAP BTP, Cloud Foundry environment, SAP XS advanced model, and Kubernetes (K8S)". It is used to read environment variables based on the three options mentioned earlier in the text, enabling the application to retrieve the service instance information necessary to connect to the SAP BTP Destination service and call the Destination itself. The package reads environments variables based on the three above provided options.

const request = require("request-promise");

const env = require("@sap/xsenv");

env.loadEnv();

const destinationService = env.getServices({

  dest: { tag: "destination" },

}).dest;

const xsuaaService = env.getServices({ uaa: { tag: "xsuaa" } }).uaa;

const xsuaaCreds = `${destinationService.clientid}:${destinationService.clientsecret}`;

const destinationName = "northwind";



async function getNorthwindExampleData() {

  try {

    const tokenResponse = await request({

      uri: `${xsuaaService.url}/oauth/token`,

      method: "POST",

      headers: {

        Authorization: `Basic ${Buffer.from(xsuaaCreds).toString("base64")}`,

        "Content-type": "application/x-www-form-urlencoded",

      },

      form: {

        client_id: destinationService.clientid,

        grant_type: "client_credentials",

      },

    });

    const token = JSON.parse(tokenResponse).access_token;



    const destinationResponse = await request({

      uri: `${destinationService.uri}/destination-configuration/v1/destinations/${destinationName}`,

      headers: {

        Authorization: `Bearer ${token}`,

      },

    });

    const oDestination = JSON.parse(destinationResponse);



    const result = await request({

      method: "GET",

      uri: `${oDestination.destinationConfiguration.URL}Employees?$top=10&$select=FirstName`,

    });

As you may have noticed, calling the SAP BTP Destination service yourself, including authentication and token handling, and then making the call to the actual remote service can be quite challenging. In the example shown above, the remote service is not secured, making it relatively simple. However, handling token refresh and managing authentication makes the flow more complex. As you can see and most likely know or can even imagine: quite some effort and headache.

If you want to know more about it, carlos.roggan has written a blog post about the details of calling an on-premise API using plain Node.js How to call – onPrem System – from Node.js app – via Cloud Connector.

Destination usage with the SAP Cloud SDK for JavaScript

If we want to avoid the headaches and problems associated with the manual handling of SAP BTP Destination service authentication and remote service calls, one option is to use the SAP Cloud SDK for JavaScript. According to the SDK's landing page on https://sap.github.io/cloud-sdk/: "The SAP Cloud SDK for JavaScript helps you build cloud-based apps and extensions to SAP solutions using the power and flexibility of Node.js and its ecosystem."

While the application still requires service credentials for the SAP BTP Destination service, the SDK makes connecting to the service and calling the remote service much more convenient and robust, with all the necessary handling taken care of.

Below is an example of how to use the SAP Cloud SDK for JavaScript to connect to a remote service.

import { executeHttpRequest } from '@sap-cloud-sdk/http-client';

import xsenv from '@sap/xsenv';

xsenv.loadEnv();

const destinationName = 'northwind';

async function getNorthwindExampleData() {

    const result = await executeHttpRequest(

      {

        destinationName: destinationName,

      },

      {

        url: 'Employees?$top=10&$select=FirstName',

      },

      {

        fetchCsrfToken: false,

      }

    

}

Destination usage with SAP Cloud Application Programming Model (CAP) - Node.js & Java

Next, let's look at how to use SAP Cloud Application Programming Model (CAP) to call a remote service using Destinations. In essence, calling the remote service with CAP is not much different from using the SAP Cloud SDK, especially when it comes to the example request we are trying to accomplish here. You can read more about the similarities and differences between the two in milton's blog post: SAP Cloud SDK and its relationship with SAP Cloud Application Programming Model.

One advantage of using CAP is that the CDS cli provides additional convenience, such as the ability to mock remote services or a protocol agnostic query language like CQL. Another benefit is that you can provide the necessary service binding for local development using cds bind, which eliminates the need for manual configuration.

When it comes to configuring Destinations with CAP, there isn't much that needs to be done. You simply need to pass the Destination name to cds and call the remote service in your custom handlers.

cds configuration in package.json or cdsrc.json:

{

  "cds": {

    "requires": {

      "MyRemoteService": {

        "kind": "rest",

        "credentials": {

          "destination": "northwind"

        }

      }

    }

  }

}

With that you can simply connect to the remote service using the following lines of code:

const myRemoteServiceObject = await cds.connect.to('MyRemoteService');

const employees = await myRemoteServiceObject.get('/Employees?$top=10&$select=FirstName');

That's all. CAP takes care of the details about how to connect to that service. May it be the specifics of an on-premise system with the detour via the SAP Connectivity Service, or Principal Propagation and so on. CAP handles all of these scenarios for the developer, making it easier to focus on building the application logic instead of worrying about the details of connecting to the remote service.

So, Max, what's the Transparent Proxy for Kubernetes and why should I use it?

Let's talk about the Transparent Proxy for Kubernetes and why it's beneficial to use. I apologize for the detour in explaining how the SAP BTP Destination service is typically used in Node.js or Java (even though I haven't provided the corresponding samples) projects, but it's important to understand the sweet spot of the Transparent Proxy.

Consider a scenario where you don't have access to SDKs or frameworks like CAP for whatever reason. Perhaps you're using a programming language that doesn't have such tools available, like Python, Go, Rust, or others. In such a case, wouldn't it be great if there was a mechanism that could take away the responsibility of handling destination specifics from the developer, using an infrastructure mechanism to "intercept" calls and adjust them based on the configuration of the actual destination on SAP BTP?

The Transparent Proxy for Kubernetes does exactly that. Once the Transparent Proxy is installed in SAP BTP, Kyma runtime, or any Kubernetes cluster, developers only need to consider two key aspects:

A custom Kubernetes resource of kind destination.connectivity.api.sap that you need to apply:

---

apiVersion: destination.connectivity.api.sap/v1

kind: Destination

metadata:

  name: tproxydestination

spec:

  destinationRef:

    name: "northwind"

With that, one can easily make a call to the actual remote service by using the name of the destination resource defined above - TProxyDestination. To better understand it, a curl example beautifully demonstrates it (though escaping might make it a bit less pretty 😉).

curl tproxydestination/Employees?$top=10&$select=FirstName

Impressive, right? However, this was just a basic example using an unsecured public endpoint. The true power of the Transparent Proxy lies in its ability to handle secured remote services protected by OAuth and likely hidden behind a SAP Cloud Connector. By taking over the responsibility of handling destination specifics, the Transparent Proxy saves developers from dealing with cumbersome configurations and adjustments.

How can I use it and how does it work?

If you're interested in using the Transparent Proxy and want to know how it works, here's a brief overview. To get started, you'll need to configure the SAP-provided Helm Chart with information specific to your environment, such as the service details for the SAP BTP Destination service and the location of the Transparent Proxy Docker image. The instructions for setting up the Transparent Proxy with the Repository-Based Shipment Channel (RBSC) can be found on help.sap.com, but let's take a closer look at the step-by-step setup process.

1. Create a technical user for SAP Repositories Management

2. Create a secret for that technical user in your namespace

kubectl create secret docker-registry docker-secret --docker-server=73554900100900006891.dockersrv.cdn.repositories.cloud.sap --docker-username=<rbsc-user> --docker-password=<password>

3. Create a ServiceInstance and ServiceBinding for the SAP BTP Destination service, either using the SAP BTP, Kyma runtime Dashboard or kubectl:

apiVersion: services.cloud.sap.com/v1

kind: ServiceInstance

metadata:

  name: transparent-proxy-dest

spec:

  serviceOfferingName: destination

  servicePlanName: lite

---

apiVersion: services.cloud.sap.com/v1

kind: ServiceBinding

metadata:

  name: transparent-proxy-dest-binding

spec:

  serviceInstanceName: transparent-proxy-dest

4. In case you want to use the SAP Cloud Connector, please enable the Connectivity Proxy in your cluster.

5. Configure the necessary values in the values.yaml file. An example configuration that I have used is shown below, while a comprehensive list of all available properties, including the required ones, can be found on help.sap.com.

config:

  integration:

    destinationService:

      serviceCredentials:

        secretName: transparent-proxy-dest-binding

  platform:

    region: us20

  security:

    communication:

      internal:

        encryptionEnabled: false

    accessControl:

      destinations:

        defaultScope: namespaced



deployment:

  image:

    registry: 73554900100900006891.dockersrv.cdn.repositories.cloud.sap

    repository: com.sap.core.connectivity.transparent-proxy

    pullSecret: docker-secret

config.integration.destinationService.serviceCredentials.secretName: refers to the secret that has been created by creating the ServiceBinding resource.

config.platform.region: refers to the region of your SAP BTP subaccount (regions on help.sap.com)

deployment.image.registry: refers to the image registry. remains the same for all installations since this is the registry SAP uses to distribute the Transparent Proxy

deployment.image.pullSecret: refers to the secret name you have created in one of the earlier steps using kubectl

6. Once you've completed the previous configuration steps, you're ready to install the Transparent Proxy in your cluster using Helm. Simply reference the values.yaml file you configured in the earlier steps, and the installation process will use your customized settings.

helm install transparent-proxy transparent-proxy-helm/transparent-proxy --version=1.2.0 --namespace <yournamespace> -f values.yaml

What you get as a result is a set of pods (as part of a Deployment that you can scale using ReplicaSets) that seamlessly manage the entire lifecycle of Destinations as a crucial component of the Transparent Proxy solution.

7. Create a Destination as a custom Kubernetes resource in the SAP BTP, Kyma runtime cluster:

---

apiVersion: destination.connectivity.api.sap/v1

kind: Destination

metadata:

  name: tproxydestination

spec:

  destinationRef:

    name: "northwind"

The Transparent Proxy's reconciler will detect the Destination resource and generate a Kubernetes Service with the name specified in metadata.name. Once a Kubernetes Service is established, it is allocated a DNS name in the cluster's DNS service. This DNS name is derived from the Service name and its associated namespace, allowing it to be accessed from within the cluster.

By using the name of a Kubernetes Service to call it, Kubernetes will automatically resolve the DNS name to the Service's virtual IP address. This results in the request being directed to the appropriate Pod through its load balancing mechanism, and this Pod is one of the Pods created by the Helm chart installation as you can see below.

Takeaway:

Sometimes it's easy to forget just how complex it can be to call remote services and APIs, especially if you don't have the right tools at your disposal. While the SAP Cloud SDK and CAP do an excellent job for Node.js and Java, the Transparent Proxy is a great solution for taking the responsibility of low-level API calls away from developers in other languages.

To be very clear: This blog post is not specifically about CAP and the Transparent Proxy is only available for Kubernetes.

If you're interested in learning more about the goal and features of the Transparent Proxy, I recommend reading iliyanvidenov9's blog post: Transparent Consumption of Connectivity.

Slightly off-topic: In my recent blog post, I explored SAP BTP's, Kyma runtime and its various capabilities, benefits, and challenges. I realized that the world of Kubernetes is extensive and offers far more opportunities for software development than traditional deployment methods like "cf push/cf deploy" in Cloud Foundry. As a result, it can be challenging to compare the two runtimes. However, factors like the need for a service mesh, control over application distribution, and flexibility in tool selection often lead developers to choose SAP BTP's, Kyma runtime. It's important to remember that this discussion is not about comparing the two runtimes, but rather explaining the technical topic of the Transparent Proxy for Kubernetes.

If you're interested in learning more about the similarities and differences between SAP BTP's, Cloud Foundry and Kyma Runtimes, I highly recommend reading alperdedeoglu's blog post: Developing an Application on SAP Cloud Foundry Runtime and SAP Kyma Runtime: A Comparative Analysis.

Consuming a REST service from a CAP application

2023-06-06T10:50:16+02:00

In this blog post, we will deep dive into the process of constructing a comprehensive use case using the powerful capabilities of the SAP Cloud Application Programming Model (CAP).

Our focus will be on consuming a REST service within CAP, and I draw inspiration from a highly informative blog by Robert Witt that greatly helped me in developing this application.

Let's provide a summary of the exciting journey that lies ahead of us. “I can’t wait 😁”

Use Case

In our application, I have defined a data model consisting of two main entities: Customers and Locations. A single customer can operate in multiple locations, and each location has an attribute called "temperature." Our goal is to ensure that this attribute is regularly updated with the current temperature for each respective city.

To achieve this, we require a service capable of providing us with the latest temperature data. Fortunately, there are several services available for this purpose, and for our use case, we will be utilizing the OpenWeatherMap service.

Also, we will explore how to develop an action within CAP that will interact with the OpenWeatherMap REST service. This action will retrieve the current temperature information and subsequently update our data model with the latest temperature values.

Now, let's dive into the process of building this application, starting with the definition of our data model.

Data Model

type Coordinates : {

    longitude : Decimal(9,6);

    latitude : Decimal(9,6);

};



entity Customer {



    key name :      String(20);

    

}



entity Location {



    key city :          String(20);

        coordinates :   Coordinates;

        temperature :   Decimal;

        temp_unit   :   String(2);  

}



entity OperatedIn {



    key customer : Association to one Customer not null;

        location : Association to one Location not null;

}

We have a straightforward data model that revolves around two primary entities: Customer and Location. Additionally, we have a local type known as "Coordinates," which is declared within the same service.

The Location entity has an attribute called "temperature." This attribute holds the current temperature specifically associated with each location. It allows us to store and track the temperature data effectively.

Service

using { OperatedIn as OperatedIn } from '../db/schema.cds';



service sensorData {



    @readonly entity CustomerLocations as select *, location.temperature from OperatedIn;



//  Take the City as input and call the weather API to update the current temperature

    action updateTemperature(city: String(20)) returns Decimal;



}

Within our service, we have an unbound action called "updateTemperature". This action is responsible for handling the process of calling the OpenWeatherMap service to update the temperature data.

The updateTemperature action accepts the location name as input. It then reads the database to retrieve the geo-coordinates associated with that specific location. With the acquired coordinates, the action proceeds to invoke the OpenWeatherMap REST service.

Consuming the REST Service

In the package.json you have mentioned the service details.

"cds": {

    "requires": {

      "weatherservice": {

        "kind": "rest",

        "credentials": {

          "url": "https://api.openweathermap.org",

          "requestTimeout": 30000

        }

      }

    }

  }

Custom Handler

//  Take the City as input and call the weather API to update the current temperature 

    srv.on('updateTemperature', async (req) => {



        const {city} = req.data;



        const db = await cds.connect.to ('db');

        let {Location} = db.entities;



        let results = await db.read(Location,["city","coordinates_longitude as longitude","coordinates_latitude as latitude"]).where({city:city});



        const weatherAPI = await cds.connect.to("weatherservice");

        // Using template literals

        let res = weatherAPI.tx(req).get(`/data/2.5/weather?units=metric&appid=<ApiID>&lat=${results[0].latitude}&lon=${results[0].longitude}`);



        let final_data =

        res.then( async (data) => {

            let results1 = await db.update(Location).set({temperature: data.main.temp}).where({city:city});

            return data.main.temp;

         });

        

        return final_data;



    });

When the action is invoked, this custom handler comes into play. It performs a sequence of operations to ensure the successful retrieval and update of temperature data.

Firstly, it reads the coordinates associated with the input location from the database. Then it calls the designated service for temperature data retrieval. Once the temperature is obtained, it is both updated in the database and returned as the current temperature.

It is important to note that, you need to obtain an API ID from the OpenWeatherMap website. Kindly replace "<ApiID>" with your unique API ID provided by OpenWeatherMap.

If you are wondering how to use the OpenWeatherMap services then please refer to its documentation API doc.😊

Summary

Key Takeaways from this Use Case:

Understanding the process of consuming a REST service within CAP.

Learning how to effectively read and update the database.

Implementing an action in CAP to perform specific tasks.

Note:📝 The codes in this example are purely for illustrative purposes. No performance testing has been done. There may be alternative approaches for optimization.

Thank you. Hope you enjoyed this blog! Please feel free to leave a comment or question below.

Easy way to access destinations in SAP BTP cloud foundry via Node.js

2023-06-28T20:34:00+02:00

Introduction

I've checked blog-posts for accessing the destinations having authentication OAuth2ClientCredentials in Cloud Foundry via Node.js . But I didn't find any code working properly.

Then I came across iobert's blog post about Using the Destination service in SAP BTP, Cloud Foundry Environment. The blog-post is really very nice and helpful. But there was one problem. The library: request used is deprecated.

So, thought of rewriting the code using the axios library. The sample code snippet is provided below for reference.

Pre-requisites: Please bind your application with the instances of Destination Service & Authorization and Trust Management Service respectively.

Solution

// imports

const express = require("express");

const cfenv = require('cfenv');

const axios = require("axios");



const app = express();

const PORT = process.env.PORT || 4000;



const getProjectDef = async (req, res) => {



    const MAX_ROWS = "1000";



    // Get the UAA and destination services

    const UAA_SERVICE = cfenv.getAppEnv().getService('UAA_INSTANCE_NAME');

    const DESTINATION_SERVICE = cfenv.getAppEnv().getService('DESTINATION_SERVICE_NAME');



    // Combine the client ID and secret for the UAA service into a single string

    const UAA_CREDENTIALS = DESTINATION_SERVICE.credentials.clientid + ':' + DESTINATION_SERVICE.credentials.clientsecret;



    // Set the name of the destination to retrieve and the endpoint to call

    const DESTINATION_NAME = 'DESTINATION_NAME';

    const END_POINT = 'END_POINT_URL';



    // Set the payload for the POST request to the UAA to get a token

    const post_payload = {

        'client_id': DESTINATION_SERVICE.credentials.clientid,

        'grant_type': 'client_credentials'

    };



    // Set the configuration options for the POST request to the UAA

    const post_config = {

        method: 'POST',

        url: UAA_SERVICE.credentials.url + '/oauth/token',

        headers: {

            'Authorization': 'Basic ' + Buffer.from(UAA_CREDENTIALS).toString('base64'), // Encode the client ID and secret as base64

            'Content-type': 'application/x-www-form-urlencoded'

        },

        data: new URLSearchParams(post_payload).toString() // Encode the payload as x-www-form-urlencoded

    };



    // Make the POST request to the UAA to get a token

    axios(post_config)

        .then((response) => {

            if (response.status === 200) {

                const token = response.data.access_token; // Get the token from the response data

                // Set the configuration options for the GET request to the destination service

                const get_config = {

                    method: 'GET',

                    url: DESTINATION_SERVICE.credentials.uri + '/destination-configuration/v1/destinations/' + DESTINATION_NAME,

                    headers: {

                        'Authorization': 'Bearer ' + token // Include the token in the authorization header of the GET request

                    }

                };



                // Make the GET request to the destination service to retrieve the destination configuration

                axios(get_config)

                    .then((response) => {

                        const DESTINATION = response.data; // Get the destination configuration from the response data



                        // Get the auth token from the destination configuration

                        const token = DESTINATION.authTokens[0];



                        // Set the configuration options for the POST request to the endpoint of the destination service

                        // a CPI is built over the BAPI: BAPI_PROJECTDEF_GETLIST with OAuth2ClientCredentials authentication

                        const options = {

                            method: 'POST',

                            url: DESTINATION.destinationConfiguration.URL + END_POINT,

                            headers: {

                                'Authorization': `${token.type} ${token.value}` // Include the auth token in the authorization header of the GET request

                            },

                            data: {

                                MaxRows: MAX_ROWS,

                                ProjectDefinitionRange: {

                                    item: [

                                        {

                                            Sign: req.data.sign,

                                            Option: req.data.option,

                                            Low: req.data.low,

                                            High: req.data.high,

                                        },

                                    ],

                                },

                            }

                        };



                        // Make the GET request to the endpoint of the destination service

                        axios(options)

                            .then((response) => {

                                res.send(response.data); // Return data

                            })

                            .catch((error) => {

                                res.send(error); // Return error

                            });

                    })

                    .catch((error) => {

                        res.send(error); // Return error

                    });

            }

        })

        .catch((error) => {

            res.send(error); // Return error

        });

    console.log('Exit :------->');

};



// endpoint

app.get("/getProjectDef", getProjectDef);



app.listen(PORT),

  () => {

    console.log(`Listnening to the PORT: ${PORT}`);

  };

N.B: This code snippet can be easily used in the CAPM(Node.js) applications. There is another library named sap-cf-axios which can be used if you want to code less.

Happy Coding!

Integration Architecture For Enterprise Agility Powered By SAP BTP

2023-07-05T04:43:33+02:00

Introduction

While working with customers on different solutioning and implementation projects, one of the most important area zeroes down to - Integration.

As the enterprises, grow in size and scale, many custom applications and processes are created which are required to be connected to maintain consistency and data integrity.

In this blog, I have come up with a integration architecture leveraging SAP BTP services which will help to make an enterprise connect the dots and function intelligently and seamlessly. In this article, I am focusing on a use case where the customer's ERP is either on premise or on private cloud. Ofcourse, this is based on my experience and can differ based on scenarios, but I think it at least will give you a starting point.

I will break down this blog into 2 parts

Design thinking - Integration Architecture

Explore BTP services used in the Integration Architecture

Let's dive into the details.

1. Design thinking - Integration Architecture

Integration Architecture

2. Explore BTP services used in the Integration Architecture

In this section, I will go through some of the important cloud services available on SAP BTP which are shown in the above integration architecture.

SAP Integration Suite
- Cloud Integration
- API Management
- Open Connectors

Event Driven Architecture
- Event Mesh
- Advanced Event Mesh

Connectivity
- Cloud Connector

The intent here is to give a starting point and then help you navigate to the details of each topic.

SAP Integration Suite is an iPaaS – Integration Platform as a Service. It offers a suite of cloud services enabling organizations and development teams to create production grade integration flows. This helps to connect heterogenous systems, interfaces, services, cloud based or on premise applications and data. SAP Integration Suite provides lot of great capabilities but in this blog, I will be covering few of them as mentioned below
1. Cloud Integration (CI) - Cloud integration capability provides a working environment and a development studio where you can discover existing integration flows provided by SAP and also design new integration flows based on business requirements. It also the place where you can monitor the integrations and manage the tenant.
2. API Management - API management gives you ability to wrap your integration flows, ODATA services or open connectors instances to the external world. You can also configure certificates, API keys. Using API management you can monetize your APIs.
3. Open Connectors - SAP Open Connectors is a suite of 150+ connectors for integration with non SAP applications. For example – TWILIO, STRIPE etc.

Event Driven Architecture - As number of software applications grow, organization must develop a communication strategy without overhead on the systems and the underlying infrastructure. As systems evolve, point to point communication can become bottleneck in many scenarios. Decoupled/Asynchronous communication helps to improve the performance and scale the software landscape. SAP BTP provides services which will help you start with your journey of Event Driven Architecture.
1. Event Mesh - SAP Event Mesh capability has been available for quite some time on SAP BTP and suffices to notably lower performance thresholds than SAP Advanced Event Mesh. The maximum message size is 1 MB for all messaging protocols.The maximum storage space for all messages in all the queues per subaccount is 10 GB. SAP Event mesh is deployed centrally and does not offer distributed deployment option.
2. Advanced Event Mesh - It provides a complete event streaming, management and monitoring platform designed with higher performance threshold in mind. The maximum message size is 30 MB for all messaging protocols.The maximum storage space for all messages in all the queues per subaccount is 1TB. This is highly attractive solution for high volume real time business processes. On top of it, it has a ability to deploy event brokers at the edge of the business network which is close to the source of events. Both the event meshes may coexist based on use case. SAP Event mesh can subscribe to events published from SAP Advanced event mesh.

Connectivity - SAP BTP provides a connectivity service through cloud connector. Cloud connector is an application that can be installed on customer's operating system which creates a secure tunnel between SAP BTP and customers on premise or private cloud landscape.

Conclusion

As I always say Enterprise Architecture is full of constraints. Every customer landscape is different, processes are different.

But with the help of SAP BTP Integration capabilities you can navigate through the constraints effectively.

This is just a glimpse of wide set of capabilities offered by SAP Integration Suite. Stay curious! Keep learning!

To get more updates about this topic, please follow the below pages

SAP Integration Suite

Blogs

Feel free to “like“, “Share“, “Add a Comment” and to get more updates about my next blogs follow me – avinash.vaidya

Next in the SAP Garage – Build Events-to-Business Actions Apps with SAP BTP and MS Azure/AWS.

2023-08-01T06:53:14+02:00

In the August 2023 edition of SAP Garage we feature this use case that highlights event-to-action framework and SAP BTP services for developing, deploying and monitoring the solution like the CAP framework, SAP Event Mesh, SAP Connectivity Service, SAP Private Link Service, SAP Business Rules.

The demo will focus on consuming events from the Microsoft Azure IoT Platform and AWS IoT and based on business rules in SAP BTP, actions are triggered in the SAP S/4HANA system. Then the app can be customized to any event integrating from any source system to any SAP LoB systems.

Date of the session – August 2nd, 2023, Wednesday

SAP Discovery Center mission to be featured : Build Events-to-Business Actions Apps with SAP BTP and MS Azure/AWS

Meet the Experts:

praveenpadegal, working as a Development Expert in SAP BTP Platform Adoption & Advisory team to increase the adoption of SAP Business Technology Platform by building, enabling and validating use-cases that demonstrates value to customers and partners. He has 12+ Years of IT industry experience in different phases of software development, especially in building Industry 4.0 cloud products, publishing reference architectures, technical blogs and enablement sessions.

ajitkpanda, working as an Architect in SAP BTP Platform Adoption & Advisory team to increase the adoption of SAP Business Technology Platform by building, enabling and validating use-cases that demonstrates value to customers and partners. With 10+ years of experience in design and development of application and integration of SAP systems he has lead discovery conversations, architect, and build end to end applications, tools using different programming technologies like SAPUI5, NodeJS, CAPM, SAP Build and BTP services in conjunction with hyperscaler services.

Missed our last call?

No worries you can revisit the session with details below –

Topic : Extend SAP S/4HANA with SAP Build Process Automation

Speakers : Syed Ejazuddin, Senior BTP Product Specialist

Link to Recording

Join us.

Thank you for all of the positive feedback we continue to receive about the SAP Garage series. We are thrilled to bring you top SAP BTP use cases and demos directly from the experts. If you haven’t already, subscribe to the series, bring your curiosity, your questions, and join us as we cover key SAP Business Technology Platform topics and use cases!

Next release of the Cloud Connector is available: 2.16.0

2023-08-07T12:43:38+02:00

We again have released a new version of the cloud connector with interesting new features. A lot happened under the hood to ensure the highest security levels but that is not all.

Let me start with the installation (which of course can be found here) where we've added two more platforms to give you more flexibility for selecting the best platform:

Oracle Linux 9

macOS aarch64 (native support of the M1/M2 processors)

If you are using an ABAP System in the cloud and need more security for classic RFC endpoints, you should check the new feature to use SNC:

More about that can be found in the documentation.

Not for all users but at least for some it might be helpful that you're able to change now the login procedure from user and password to a client certificate. This works also for the REST APIs. But here we have to say that (in the version 2.16) we unfortunately have the restriction to setups without a shadow instance. This will of course change in the future. If you still want to use it or at least try it out find more here.

On the logging and monitoring we’ve worked on two features:

getting more insights for TCP and TCP SSL connections as you now see them also in the Back-end connections and in the usage statistics.

adding a csv export to the Audit Log to give you more transparency which modifications on the cloud connector setup had been done:

Additionally, under the hood we've updated lots of libraries to the most recent versions, cleaned up features which are no longer needed and improved the performance of the login procedure for some installations.

Therefore: don't wait and update your Cloud Connector as soon as possible (and have a look at the official release notes)!

SAP BTP client certificate destination for an IBP communication scenario: side-by-side extensibility

2023-08-07T14:29:10+02:00

Hello community,

after several years, I am returning to write a blog. For almost a year now, I start a new professional adventure with my fellow adventures 177ce313868444f78e8b2470e82a23d6 , sergio_ferrari and other friends. I feel a renewed enthusiasm in writing and sharing my experiences, especially regarding the BTP platform .

The journey from 2010s Netweaver blogs to the new BTP era. So, I hope that this will be the first blog of a new long series.

In this blog, I want to describe how to set up a secure connection based on X.509 certificate from BTP to SAP IBP (SAP Integrated Business Planning for Supply Chain) within the context of side-by-side extensibility.

You can find detailed explanations about this process in the official documentation at the following link: https://help.sap.com/docs/SAP_INTEGRATED_BUSINESS_PLANNING/feae3cea3cc549aaa9d9de7d363a83e6/21f637e22dab41599b2a5334ff9908f7.html?locale=en-US.

Important: Of course, this approach is applicable to any context of an ABAP environment and a communication scenario where we want to establish a connection via a certificate. In the blog, I will describe it focused on a specific communication scenario of SAP IBP.

Introduction

I am not an IBP consultant, and I won't go too deep into product-specific details.

The side-by-side extensibility scenario, as described in the link, involves a custom algorithm implemented in BTP / CAP, using the OData services exposed by SAP IBP for both reading and writing Master Data and Key Figures.

The communication arrangement used in this context is SAP_COM_0720, and you can find a detailed explanation of the OData services in this blog: https://blogs.sap.com/2022/06/20/integrate-sap-ibp-key-figure-delta-%CE%B4-data-with-external-systems/

I will describe how to configure a connection between BTP and SAP IBP using X.509 certificates, allowing a seamless integration of your custom algorithm with SAP IBP's OData services.

As we can see in the image below, OAuth 2.0 is not supported as an Inbound Authentication Method for the Communication Arrangement Managed by SAP

IBP Communication Scenario

We could generate a password for the user defined in IBP and use the simpler basic authentication. However, as we already know, and as emphasized in the IBP documentation "Secure Communication for Inbound Integration" https://help.sap.com/docs/SAP_INTEGRATED_BUSINESS_PLANNING/685fbd2d5f8f4ca2aacfc35f1938d1c1/37bcbb7616734a22ada7122e06654e90.html?locale=en-US the "Certificate-based authentication" option is highly recommended.

By opting for certificate-based authentication, we ensure a more robust and secure connection between BTP and SAP IBP.

Generate and download the Certificate via Connectivity service

With reference to the Connectivity service documentation (https://help.sap.com/docs/connectivity/sap-btp-connectivity-cf/use-destination-certificates), navigate to the "Destinations" section of your subaccount and select the "Certificates" option. From here, you can generate a new SAP Cloud PKI Infrastructure Certificate. In this example, I am generating a PEM certificate, setting a password, and configuring it to be valid for one year.

Once the certificate has been generated, it will be listed among the available certificates. However, as mentioned in the highlighted message, downloading the certificate is only possible through the REST API.

To identify the API for accessing certificates within the subaccount, we can leverage the SAP Business Accelerator HUB. In our case, we want to access the certificates associated with the subaccount using the relevant API https://api.sap.com/api/SAP_CP_CF_Connectivity_Destination/resource/Certificates_on_Subaccount_Level

Let's proceed to use Postman. If you haven't done so already, you can create a new instance of the "Destination Service" and generate a key to obtain the client ID / client secret pair required to invoke the API from Postman.

In Postman, we have to use the Grant Type “Client Credentials” where:

Access Token Url is the url of the key plus /oauth/token

Client ID and Client Secret the corresponding values from your key.

The endpoint we are going to use to access the certificates of the subaccount is https://destination-configuration.cfapps.eu20.hana.ondemand.com/destination-configuration/v1/subaccountCertificates

Once we obtain the OAuth token and send the request, we receive the list of certificates along with their corresponding content in Base64 encoding.

I’m a windows user , so to decode the Content from base64 to a file, I used Powershell but you can use the tool (or online tool) you prefer

$base64String = "pasteYourBase64Content"

$decodedString = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($base64String))

$decodedString | Set-Content -Path "C:\your\path\your_certificate.pem"

Now you have your .pem certificate containing both public cert and key.

pem can be opened with a text editor and splitted in 2 new files:

content between

----BEGIN ENCRYPTED PRIVATE KEY---—

----END ENCRYPTED PRIVATE KEY---—

in a .key and the fullchain starting from ----BEGIN CERTIFICATE---— in a .crt file

Upload the Certificate in Target System (SAP IBP in our scenario)

The certificate (.crt) must be uploaded in the “Communication User” Fiori App. This is the communication user will be used in the Communication Arrangement.

Now you can test the certificate with a request in Postman. In Settings, it’s possible to add key,crt , the host of your target system and the password you specified during the creation of PEM.

Additional details can be found here https://learning.postman.com/docs/sending-requests/certificates/#adding-client-certificates

Now we can create the destination that will use the certificate to authenticate towards IBP. However, before that, we need to upload the key that will be referenced during the definition phase. We can generate the key in .p12 format using OpenSSL (I'm not an OpenSSL expert). The command that can be used to generate it is:

openssl pkcs12 -export -in IBP_EXT.crt -inkey IBP_EXT.key -out IBP_EXT.p12

Go again in the “Certificates” of the subaccount and upload the .p12 file

Create the Destination

Now we are ready to create the destination with Authentication: ClientCertificateAuthentication

where URL is your IBP OData Endpoint of the communication arrangement https://youribphostname-api.scmibp.ondemand.com/sap/opu/odata/IBP

Key Store Location your .p12 and Key Store Password is your password

Side By Side: Test the destination in CAP / Application router

There are several ways to test the functionality of the destination to ensure it works correctly. We could create a CAP project or a project using the Productivity Tools within the space.

However, I believe the simplest and fastest way to test the routing and proper functioning of a destination is to define a custom app router that forwards inbound requests to the destination.

This tech byte perfectly explains how to define a custom app router for this purpose. The only adjustments I had to make compared to the template were to update the Node version, OAuth redirect-uris in xs-security.json and add a basic auth in order to easily test the application router's endpoint from Postman

Tutorial - Testing SAP BTP Destinations https://www.youtube.com/watch?v=0zzFCfuUIbs

my repository can be found here https://github.com/alespad/ibptestdestination

Once the Node app is deployed, we can open the link of the application router and append the relevant OData path for the master data (or key figure) we want to test.

At this point, we can verify that the OData service is correctly consumed using the routing destination.

Appendix

The list of SAP IBP trusted root CAs is available in SAP Note 2801396 (SAP Global Trust List). SAP is not in the list but I suppose it’s “trusted” by default (also used for Cloud Connector). If you have some useful additional info about this, please add your comment

Next in the SAP Garage – Integrate SAP S/4HANA with Microsoft Teams via SAP Business Technology Platform.

2023-09-05T14:40:09+02:00

In the September 2023 edition of SAP Garage we feature this use case that highlights how to extend the business processes in SAP S/4HANA to the Microsoft Teams app, by using the SAP BTP Bridge Framework.

The demo will focus on how to use the Bridge Framework to view the SAP S/4HANA purchase order data, update purchase order data if needed, and perform the release purchase order process in the Microsoft Teams.

Date of the session – September 6th, 2023, Wednesday

SAP Discovery Center mission to be featured : Integrate SAP S/4HANA with Microsoft Teams via SAP Business Technology Platform

Meet the Experts:

viancu, is a Developer for the Strategic Customer Engagements team of SAP Platform Adoption and Advisory. He works alongside fellow developers, designers, and data scientists to harness the power of SAP BTP to deliver innovative solutions to customers. He began his SAP journey during college as a frontend development intern for SAP Ariba’s Guided Buying team. Since then, his fulltime role has expanded to include full-stack development centered around SAP BTP and the broader SAP Ecosystem. Victor’s most recent work has focused on bringing two worlds—SAP and Microsoft Teams—closer together through an integration solution on BTP.”

Alex Bishka, a Developer in the Strategic Customer Engagements team of SAP Platform Adoption and Advisory. Alex started at SAP under the SAP Next Talent program. Since joining the team, Alex has focused on full stack development and automation around SAP BTP technologies: specifically on his current project which aims to bridge the gap between SAP and Microsoft Teams by implementing an integration solution via BTP.

Weikun Liu, a Software Developer from SAP T&I Strategic Customer Engagements team. He is an experienced software development engineer with 4+ years 'experience on crafting efficient and scalable back-end distributed systems within cloud environments. As a Software Developer at SAP Labs, Palo Alto, he collaborated closely with customers, orchestrating the development and delivery of systems for combining SAP Business Technology Platform (BTP) and other hyperscaler services.

Missed our last call?

No worries you can revisit the session with details below –

Topic : Build Events-to-Business Actions Apps with SAP BTP and MS Azure/AWS

Speakers : Praveen Kumar Padegal, Development Expert & Ajit Kumar Panda, Architect, Platform Adoption & Advisory team

Link to Recording

Easy way to provide schema access to a BTP HANA Cloud Standard User

2023-10-05T22:04:35+02:00

Problem

While working with the CAP applications, I have came across a situation that a standard HANA cloud user used in CPI cannot perform direct CRUD operations on the tables via the JDBC driver. The CPI will throw the error: insufficient privileges. So, I am writing the blogpost for reference purpose.

Solution

Please follow the below mentioned steps:

Open the SQL Console of the BTP HANA Cloud by selecting the DBADMIN node.

Create the User group:
```
CREATE USERGROUP USERGROUP_NAME
```

Creating the User & assigning it to the group: USER_01

CREATE USER USER_01 PASSWORD Password1111 NO FORCE_FIRST_PASSWORD_CHANGE SET USERGROUP USERGROUP_NAME

Grant the User: USER_01 operator of the group: USERGROUP_NAME

GRANT USERGROUP OPERATOR ON USERGROUP USERGROUP_NAME TO USER_01

If there is already a user present then please ignore the above steps.

Now, its time to check the SCHEMA_OWNER.
1. Get the SCHEMA name. Add the instance by selecting Instance Type: HDI Container.
2. Expand & click on the Tables. Select any table & then double click to open. You will get the SCHEMA name.

Go to the SAP HANA Cockpit.

Click on the drop down list & select Security and User Management.

Under the User & Role Management tile, select the Role Assignment option.

Select Assign roles to a user. Enter the user.

Click on the EDIT button & add the roles by providing the SCHEMA name.

Select & add the role you want. Then, save it.

Click on the added role to check more details.
1. Object Privilege
2. Assigned Users

Now, the user is having access to the SCHEMA.

That's it.

Now you can use the same user in the CPI for CRUD operations.

If you want to learn more about the privileges then please check this link.

If I have missed something, please feel free to add it in the comment section so that, this post can be useful to others.

References:

SAP HANA Cloud User Management

SAP HANA One Administration Guide

SAP BTP Destinations in a nutshell Part 3 - OAuth 2.0 Client Credentials

2023-10-08T09:15:07+02:00

Introduction

The goal of this series is to introduce the different destination types of the SAP BTP, to show how they work and also to make them tangible with basic and easy to execute code examples. In parts one and two of this series, I looked at the basics, as well as the simplest destination types, NoAuthentication and BasicAuthentication. In this part, I'll dive into a more complex topic for the first time: I'll show you how to implement an OAuth 2.0 client credentials flow using Destination.

OAuth 2.0

Before I turn to the concrete scenario and implementation, I will first take a closer look at the topic of OAuth. Most people in the IT environment have come across this term before, and a majority is still able to provide information when it comes to explaining at a high-level what it is and what it is used for. But as soon as it comes to the technical details, the exact workings of the protocols, the different authentication flows, etc., many (including me) quickly get tripped up. But fortunately I'm not alone in providing the most important information about OAuth, because there are also a large number of wonderful people who have a very deep understanding of the subject and are willing to document this knowledge and make it digitally available to their fellow world in an easy to understand way. One resource I came across a couple of years ago that has helped sharpen my understanding of this topic is this talk by Nate Barbettini: https://www.youtube.com/watch?v=996OiexHze0 I can highly recommend this video to anyone who would like to learn more about OAuth 2.0 (and for good measure, OIDC) or is looking for a simple yet solid introduction to the topic. The investment of this one hour is definitely worth it.

For those who are in a hurry, here is the most important information in a nutshell:

OAuth 2.0 is an open protocol that allows standardized, secure API authorization for desktop, web and mobile applications. This also covers the scenario of access delegation, i.e. that an application gets access to certain data or functionalities of another application with the permission of the user. Classic example: an application is granted permission by a user to send posts on Facebook or read contact data on their behalf.

4 Roles

Resource Owner - The owner of a resource (data). As a rule, this is simply a user, i.e. the person sitting in front of the computer / smartphone / tablet / ...

Resource Server - This is the application that stores the resources (data) of the resource owner (user)

Client - The client is the third-party application that wants to access resources (data) of the resource server on behalf of the resource owner (user) and needs a token for authentication that contains the corresponding authorization grant (so to speak, the proof that the client has received permission from the resource owner to access the resource server)

Authorization Server - The Authorization Server is the instance that manages Authorization grants and issues the tokens (these are usually JWT (JSON Web Tokens), i.e. JSON objects that contain the relevant token information) that authorize the client to access the Resource Server on behalf of the Resource Owner. In some cases, Resource Server and Authorization Server are identical, but often they are separate instances

Client Credentials Flow

Depending on the specific use case, there are different flows in the OAuth 2.0 standard. In this part of the series, we will look at the client credentials flow. In order for a client to access a resource server, it must be registered with the Authorization Server, for each registered client there are client credentials, these are ultimately username and password. In the client credentials flow, the client also acts as the resource owner, because it does not obtain delegated access to the resource server, but instead requests a token for itself from the authentication server (using basic authentication with the client credentials) in order to then authenticate itself to the resource server with its own (technical) identity outside of a user context.

XSUAA

The XSUAA service, inside of the SAP BTP, handles the authorization flow between users, identity providers, and the applications or services. The XSUAA service is an internal development from SAP dedicated for the SAP BTP. In the Cloud Foundry project, there is an open-source component called UAA. UAA is an OAuth provider which takes care of authentication and authorization. SAP used the base of UAA and extended it with SAP specific features to be used in SAP BTP.

https://learning.sap.com/learning-journey/discover-sap-business-technology-platform/illustrating-sap-authorization-and-trust-management-service-xsuaa-_b9fde282-4cff-4dca-b146-7c8f8dde9955

In the context of OAuth 2.0, XSUAA is acting as the authentication server, i.e. it manages authorization grants and issues and validates JWT tokens. Important to know is, that there is always one XSUAA service instance per subaccount. As you will also later see in the concrete example, it is best practice to bind a dedicated XSUAA "service" resource to every application service (in our example: Client and Server). It took me a while to understand that those XSUAA services are not real running services in terms of a deployed container or something similar, but actually this are "only" pairs of Client ID and Client Secret (plus some additional meta information) that are injected into the application environment variables (VCAP_SERVICES). I.e. this binding represents a client in the sense of OAuth 2.0 terminology.

Cloud 2 Cloud - OAuth 2.0 Client Credentials

https://github.com/jfranken/sap-btp-destinations/tree/cloud-2-cloud-oauth2-client-credentials

This all sounds rather abstract so far, so at this point I will fall back on our well-known example from the previous parts of this series to illustrate the use case:

In this scenario, the BTP Destinations Enthusiast acts as the resource owner, the server app is the resource server, and XSUAA represents the authorization server. When the user calls the client/helloWorldClientPlain(), the client app subsequently attempts to access the /server/helloWorldServer() endpoint upon execution, but not on behalf of the user, but in this scenarios by means of technical client credentials. This endpoint is secured using XSUAA, which means that the server app expects a JWT token in the request header, which is issued and validated by XSUAA. To obtain this token, the client makes use of an according OAuth2ClientCredentials destination in the destination service.

Server Implementation

At first, I will show you the basics of the implementation of the server app.

To be able to secure any endpoint of this service via XSUAA, we need to establish a binding between server app and XSUAA. This is done in the mta.yaml file that we use for the server app deployment:

...

resources:

...    

    - name: xsuaa-server-service

      type: org.cloudfoundry.managed-service

      parameters:

          service: xsuaa

          service-plan: application

          service-name: xsuaa-server-service

In addition, it is required to enable JWT authentication. With SAP CAP, this is pretty staightforward within the cds configuration, which is located in the .cdsrc.js:

{

    "requires": {

        "auth": {

            "kind": "jwt-auth"

        },

        "uaa": {

            "kind": "xsuaa"

        }

    }

}

Authentication checks can be added by means of annotations which can be added in the service CDS files (the example below shows the server-service.cds file), either on service level or for single entities / functions / actions. In the given example, I only add a very rudimentary check that requires nothing else than the caller to be authenticated at all (i.e. sending a valid JWT token, authenticated-user is an according pseudo role). It is also possible to check for concrete role assignments within the annotation, which is in real world scenarios normally the case, however, since this series aims at keeping the focus on the pure communication flow between the different instances, I try to keep it as simple as possible:

service ServerService @(requires: 'authenticated-user') {

    function helloWorldServer() returns String;

}

The service implementation within the file server-service.js isn't very spectacular and should be familiar to the attentive readers of the first two parts of the series:

module.exports = async (srv) => {

    srv.on('helloWorldServer', async (req) => {

        return 'Hello World from Server'

    })

}

Destination

As described above, the client app needs to use a destination to obtain a valid JWT token that allows access to the endpoint of the server app, which is secured via XSUAA. The most important aspect here is that I store the client credentials from the XSUAA binding of the server app in the destination, because these represent the OAuth 2.0 client that is authorized to access the server app. The destination configuration in detail is as follows:

Name: Server

Type: HTTP

URL: https://${SERVER_APP_ROUTE}/server/helloWorldServer()

ProxyType: Internet

Authentication: OAuth2ClientCredentials

Client ID: Server XSUAA Client ID from Service Binding

Client Secret: Server XSUAA Client Secret from Service Binding

Token Service URL: https://????????trial.authentication.????.hana.ondemand.com/oauth/token

Please check the README.md file in the github repo. There you find a detailed description how you can find the concrete URLs and credentials via the BTP Cockpit and the Cloud Foundry CLI.

Client Implementation

As always, the client includes two alternative implementations. On the one hand, the server call via Cloud SDK, and on the other hand, a plain variant that illustrates the communication processes step by step.

Cloud SDK

srv.on('helloWorldClientCloudSDK', async () => {

    return executeHttpRequest({ 

        destinationName: 'Server' 

    }).then((response) => response.data)

})

Plain

First, the client authenticates against XSUAA with the client credentials from the destination service binding. I.e. here we already have the first client credentials flow to obtain a token that allows the client app to access the destination service. Afterwards, the client app requests the destination "Server" from the destination service, which is described in the chapter above. During this call, the destination service executes another client credentials flow and adds the token into the response which is then sent back to the client app. This could already be used to access the endpoint of the server app. However, for demonstration purposes, the client app implementation triggers another client credentials flow against XSUAA with the client credentials of the destination configuration.

srv.on('helloWorldClientPlain', async () => {

    // get all the necessary destination service parameters from the  

    // service binding in the VCAP_SERVICES env variable

    const vcapServices = JSON.parse(process.env.VCAP_SERVICES)

    const destinationServiceUrl =

        vcapServices.destination[0].credentials.uri + 

        '/destination-configuration/v1/destinations/'

    const destinationServiceClientId = 

        vcapServices.destination[0].credentials.clientid

    const destinationServiceClientSecret = 

        vcapServices.destination[0].credentials.clientsecret

    const destinationServiceTokenUrl = 

        vcapServices.destination[0].credentials.url + 

        '/oauth/token?grant_type=client_credentials'



    // before we can fetch the destination from the destination 

    // service, we need to retrieve an auth token

    const token = await axios.post(

        destinationServiceTokenUrl, 

        null, 

        {

            headers: {

                authorization: 'Basic ' + 

                    Buffer.from(

                        destinationServiceClientId + 

                        ':' +            

                        destinationServiceClientSecret'

                    ).toString('base64'),

            },

        }

    )

    const destinationServiceToken = token.data.access_token



    // with this token, we can now request the "Server" destination 

    // from the destination service

    const headers = {

        authorization: 'Bearer ' + destinationServiceToken,

    }

    const destinationResult = await axios.get(

        destinationServiceUrl + 'Server', 

        { headers }

    )

    const destination = destinationResult.data



    // now, we use the retrieved the destination information to send 

    // a HTTP request to the token service endpoint;

    // to authenticate, we take the Client ID attribute and the 

    // Client Secret attribute from the destination,

    // encode ClientId:ClientSecret to Base64 and send the resulting 

    // string prefixed with "Basic " as Authorization

    // header of the request;

    // as a response, we receive a JWT token that we can then use to 

    // authenticate against the server

    // alternatively, the JWT token could directly be fetched from 

    // destination.authTokens[0].value

    const jwtTokenResponse = await axios.get(

        destination.destinationConfiguration.tokenServiceURL + 

        '?grant_type=client_credentials', 

        {

            headers: {

                Authorization: 'Basic ' +

                btoa(destination.destinationConfiguration.clientId + 

                ':' + 

                destination.destinationConfiguration.clientSecret),

            }

        }

    )

    const jwtToken = jwtTokenResponse.data.access_token



    // here we call the server instance with the bearer token we 

    // received from the token service endpoint

    const destinationResponse = await axios.get(

        destination.destinationConfiguration.URL, 

        {

            headers: {

                Authorization: 'Bearer ' + jwtToken,

            },

        }

    )

    return destinationResponse.data

})

Execution

Once the client and server are deployed and the destination is configured correctly, calls to both client endpoints should return the following result:

That's it! In the next part of this series, I will go one step further. Instead of using technical client credentials, I will show you how you can use OAuth2TokenExchange destinations with authorization delegation. The client app will call the server app by propagating the identity of the user who calls the client endpoint in the beginning, i.e. we will cover one of the core scenarios for which OAuth 2.0 has been invented.

Disclaimer

The code examples provided are not suitable to be transferred 1:1 into productive implementations, but they only serve to illustrate the functioning of destinations and Destination Service. In some cases, the interaction with the Destination Service is deliberately implemented in a “cumbersome” manner in order to illustrate the communication processes as explicitly and in detail as possible. SAP CAP and the SAP Cloud SDK provide various functionality to simplify and abstract the use of destinations. However, this would be more of a hindrance than a benefit to the purpose of the illustration.

Tell us about Your Experience with SAP BTP Connectivity

2023-10-18T23:22:32+02:00

Have you ever been in touch with the Connectivity service or the Destination service for SAP Business Technology Platform (SAP BTP)?

For example, creating or configuring a destination in the BTP cockpit, or doing some connectivity configuration in your app code?

If so, let us know about your experience.

There's a really short survey (5 minutes or less) that will help us improve SAP BTP Connectivity components according to your feedback.

You can find the survey here.

Thank you!

It has never been easier! Connecting cloud apps to Internet and on-premises systems using SAP BTP Connectivity

2023-11-14T21:29:59+01:00

It is a common scenario that cloud applications need to connect to remote systems to fulfil the business goals of their creators and those of their end users. This is essential for enterprise applications, which are generally complex and consume data from and/or push data to a variety of sources or destinations – systems that are directly accessible, systems hosted in public or private cloud, or such that are hosted in the customer premises. This use case is called hybrid connectivity.

Sounds complex, right?! With this blog post, I show you that it has never been easier to solve this problem. Let’s get started and see how SAP BTP Connectivity can help with this challenge, more specifically, in SAP BTP Kyma environment.

Prerequisites

Well, complex things cannot be made simple without proper preparation work. Therefore, I need to setup the environment. For the purposes of this blog post, I don’t get into details on how each step is done, if interested, you can follow the links:

Setup the cloud environment:
1. Create a SAP BTP subaccount - the PaaS context in the domain of SAP BTP, i.e., an account enabled to instantiate cloud application development environments, create and manage service instances, etc.
2. Enable Kyma environment - the cloud-native application hosting environment
  1. Enable Connectivity Proxy for cloud to premise technical connectivity, an integrated component in Kyma environment
  2. Enable Transparent Proxy for unified, virtually transparent technical connectivity to any destination or data source, an integrated module in Kyma environment

Setup the local environment
1. Install Kubectl - the command line interface for connecting to and interacting with the Kyma instance
2. Install Cloud Connector for controlled and secure exposure concrete systems or resources, hosted in a VPC on Hyperscalers or on-premises - in my case, on my PC.

For each scenario use case, create the relevant destinations in SAP Destination service using SAP BTP cockpit.

Overview of the scenario

Image: Scenario Schematic Overview

This solution diagram depicts the high-level architecture and layout of the SAP BTP tools, software components, services listed above in the Prerequisites section. The focus on the scenario is on the Application side. The rest is depicted for completeness and better end-to-end understanding.

In this blog, I showcase how I can connect my cloud application to the following target systems. I start with the trivial and then continue with the more advanced use cases:

Google - direct connectivity without using any of the SAP BTP Connectivity software and services

Google via destination - direct connectivity with usage of SAP BTP Connectivity software and services

Google via destination and Cloud Connector - indirect cloud to premise connectivity - in my setup Google is directly accessible via my local Cloud Connector
- Note: this use case is presented only for the purpose of showcase and ease the perception of the reader, it is not expected to be done in production

SAP Authorization and Trust Management Service (XSUAA) via destination - OAuth based REST API

An HTTP system hosted on-premises via destination and Cloud Connector - indirect cloud to premise connectivity

An HTTPS system hosted on-premises via destination and Cloud Connector - indirect cloud to premise connectivity with Principal Propagation enabled, i.e., end-to-end secure user context propagation, a.k.a Single Sign On (SSO).

I pick those systems to showcase what I claimed in the begging of this blog - It has never been easier!

Note: For the creation of the destination pointing to XSUAA in step 4 of the scenario, I followed these two simple steps in BTP cockpit:

Create a service instance of service "xsuaa", plan "apiaccess"

Create a destination pointing to the service instance via destinations UI - just a few clicks job!

Configure the scenario

To get the scenario in action, at first I need to configure the target systems as technical connection configurations, a.k.a. destinations. In this way I control to which systems the application has access to and can switch the used technical authentication and authorisation mechanisms on the fly – changing the destination attributes without affecting the experience of the end user, and without affecting the lifecycle of the application.

Expose the on-premises system to the cloud

For the destinations pointing to systems hosted in the customer premises (points 5 and 6 of the scenario overview), I need to securely expose those to the cloud.

You guessed it, for this I configure the respective Access Controls in my SAP Cloud Connector:

Image: Access Control entries in Cloud Connector

In this example, the two systems hosted in the on-premises are simple HTTP and HTTPS servers:

System #1: localhost:8000 - serves HTTP, returns status code: 200 with the received HTTP request line as a message.

System #2: localhost:9000 - serves HTTPS, returns status code: 200 with the received HTTP request line as a message, and the subject common name (CN) of the received X.509 client certificate as part of the HTTP request - the user context propagated from the cloud, achieving Single Sign-On (SSO).

Manage the technical connection configurations

One of the best practices for cloud-native applications is to externalise any configuration and avoid coupling it with the lifecycle of the application, e.g. via hard-coding it. I use Destination service for managing the technical connection configurations (a.k.a. destinations), as guided by the Golden Path defined in Cloud Application Programming model of SAP BTP.

In the context of my SAP BTP subaccount, I create the following destinations, pointing to the variety of systems I’ll connect my cloud app workload running in my Kyma instance.

Image: Destinations in BTP cockpit

Expose the destinations in the Kyma instance

To allow an application to consume the defined destinations, I declaratively expose only those I'm interested in, and are specific for this particular use case. For this, I create specific Destination Custom Resources, a common practice for cloud-native applications based on Kubernetes. In Kyma environment, I can do this either via Kyma Dashboard, or via command-line using Kubectl.

Image: Creation of a Destination CR in Kyma Dashboard

Shortly after the Destination CR is created, Transparent Proxy process it and updates the status of the Destination CR with a message that the technical connectivity is successfully configured, and this destination is ready to be consumed.

Image: Status of a Destination CR in Kyma Dashboard

Using the same approach, I expose all the destinations required specifically for this use case.

Image: Destination CRs in Kyma Dashboard

It's all set now, let's play with the application.

Scenario in action: Connect the application to the remote systems

As described in the status message of the Destination CR, the Transparent Proxy exposed the referenced destination via the specified name in the form of locally accessible host, leveraging the concept of Kubernetes Service.

As a result, it’s trivial for the application to connect to those local hosts, and this is the only task needed to be performed. It’s that easy!

For simplicity and versatility reasons, my application is represented by a local terminal attached to a container running in a Kubernetes Pod in the Kyma Instance. Then I use cURL command-line tool for executing HTTP requests towards the target systems.

How it’s done? Once connected to the Kyma instance via Kubectl, I run a sample cURL image as a Kuberenetes Pod and open a terminal session via the following command:

kubectl run mycurlpod -n sap-transp-proxy-system --image=curlimages/curl -i --tty -- sh

Executing request to Google:

This is a trivial direct invocation of the public web page of Google:

~ $ curl www.google.com -v

*   Trying 142.250.179.164:80...

* Connected to www.google.com (142.250.179.164) port 80

> GET / HTTP/1.1

> Host: www.google.com

> User-Agent: curl/8.4.0

> Accept: */*

> 

< HTTP/1.1 200 OK

...

< 

<!doctype html><html...<title>Google</title><script...

Executing request to Google via destination:

This is an example reaching the same public web page of Google, but this time via destination, locally exposed and served by Transparent Proxy, and centrally managed via Destination service:

~ $ curl google -v

*   Trying 10.111.255.220:80...

* Connected to google (10.111.255.220) port 80

> GET / HTTP/1.1

> Host: google

> User-Agent: curl/8.4.0

> Accept: */*

> 

< HTTP/1.1 200 OK

...

< 

<!doctype html><html...<title>Google</title><script...

Executing request to Google via destination via Cloud Connector:

This is an example of reaching the same public web page of Google via destination, locally exposed and served by Transparent Proxy, Connectivity Proxy, and centrally managed via Destination service. The destination is configured to point to an on-premises system, exposed to the cloud via Cloud Connector:

~ $ curl mypremisegoogle -v

*   Trying 10.111.159.5:80...

* Connected to mypremisegoogle (10.111.159.5) port 80

> GET / HTTP/1.1

> Host: mypremisegoogle

> User-Agent: curl/8.4.0

> Accept: */*

> 

< HTTP/1.1 200 OK

...

< 

<!doctype html><html...<title>Google</title><script...

Executing request to XSUAA API via destination:

This is an example for getting the registered service instances of the current subaccount via destination, locally exposed and served by Transparent Proxy, and centrally managed via Destination service.

~ $ curl xsuaa-api/sap/rest/authorization/v2/apps -v

*   Trying 10.106.192.162:80...

* Connected to xsuaa-api (10.106.192.162) port 80

> GET /sap/rest/authorization/v2/apps HTTP/1.1

> Host: xsuaa-api

> User-Agent: curl/8.4.0

> Accept: */*

> 

< HTTP/1.1 200 OK

< cache-control: no-cache, no-store, max-age=0, must-revalidate

< content-length: 10621

...

< 

[{"appid":"app123!b13","serviceinstanceid":"15442f82-7d82-11ee-b26d-ab53f17d39a5","planId":"HWEgt9/213f90b0+/7d82-11ee=","planName":"broker"...

Executing request to an on-premises HTTP server via destination via Cloud Connector:

This is an example of connecting to a simple on-premises HTTP server via destination, locally exposed and served by Transparent Proxy, Connectivity Proxy, and centrally managed via Destination service. The destination is configured to point to an on-premises system, exposed to the cloud via Cloud Connector:

~ $ curl mypremserver/test-path -v

*   Trying 10.110.23.19:80...

* Connected to mypremserver (10.110.23.19) port 80

> GET /test-path HTTP/1.1

> Host: mypremserver

> User-Agent: curl/8.4.0

> Accept: */*

> 

< HTTP/1.1 200 OK

...

< 

Response generated by HTTP server: 

GET /test-path HTTP/1.1

Executing request to an on-premises HTTPS server via destination via Cloud Connector with Single Sign-On enabled:

This is an example of connecting to an on-premises HTTPS server via destination, locally exposed and served by Transparent Proxy, Connectivity Proxy, and centrally managed via Destination service. The HTTPS server requires the cloud user identity to be propagated. The destination is configured with PrincipalPropagation as authentication type, and to point to an on-premises system, exposed to the cloud via Cloud Connector:

~ $ curl myppserver -H 'Authorization: Bearer eyJhbGciOiJSUzI1Ni...bwKMpAGKbhECqvkyibC7Q' -v

*   Trying 10.105.101.106:80...

* Connected to myppserver (10.105.101.106) port 80

> GET / HTTP/1.1

> Host: myppserver

> User-Agent: curl/8.4.0

> Accept: */*

> Authorization: Bearer yJhbGciOiJSUzI1Ni...bwKMpAGKbhECqvkyibC7Q

> 

< HTTP/1.1 200 OK

< server: envoy

< date: Tue, 07 Nov 2023 15:17:47 GMT

< content-type: text/plain; charset=utf-8

< content-length: 1956

< x-envoy-upstream-service-time: 657

< 

>>>>>>>>>

Response generated by HTTPS server: 

GET / HTTP/1.1

=========

Received X.509 client certificate with Subject: <Name(CN=manol.valchev@sap.com)>

Summary

As you can see, it’s that simple! No matter of the type and hosting location of the target system, from the application development perspective the user experience is the same – simple, unified, virtually transparent, and the technical complexity is handled by the usage of software components and services part of SAP BTP Connectivity product portfolio:

Destination service for managing the technical connection configurations (a.k.a. destinations)

Transparent Proxy for unified, virtually transparent technical connectivity to any destination or data source, an integrated module in Kyma environment, also available in Docker Hub.

Connectivity Proxy for cloud to premise technical connectivity, an integrated component in Kyma environment, also available in Docker Hub.

Cloud Connector for controlled and secure exposure concrete systems or resources, hosted in a VPC on Hyperscalers or on-premises.

Connectivity service as the backbone for the multitude of Connectivity Proxy and Cloud Connector instances serving thousands of SAP BTP customers

Тhis simple yet powerful approach enables application developers to focus more on their business goals and delegate the technical complexity to SAP BTP Connectivity, and at the same time the application administrators can manage the outbound technical connections (via destinations) without affecting the lifecycle and availability of the application itself.

Stay tuned and subscribe to What's New for SAP BTP Connectivity page.

Hey ABAP Cloud please let me save my data export to Azure Storage please🥺🙏- part 4

2023-11-21T13:21:26+01:00

👉🏿back to blog series or jump to GitHub repos🧑🏽‍💻

<<part 3

Hello and welcome back to your ABAP Cloud with Microsoft integration journey. Part 3 of this series got you covered with modern GraphQL API definition on top of your ABAP Cloud RAP APIs to expose a single API endpoint that may consume many different OData, OpenAPI, or REST endpoints at the same time.

Today will be different. Sparked by a SAP community conversation with dpanzer and lars.hvam including a community question by rammel on working with files with ABAP Cloud, I got inspired to propose a solution for the question below:

Before we dive into my proposal see here a list of alternative options that I came across as food for thought for your own research.

Mount a file system to a Cloud Foundry app	Create custom API hosted by your CF app and call via http client from ABAP Cloud
Connect to SFTP server via SAP Cloud Integration	Design iFlow and call via http client from ABAP Cloud
Integrate with SAP Document Management Service	Call SAP BTP REST APIs from ABAP Cloud directly
Integrate with SAP BTP Object Store exposing hyperscaler storage services using SDKs	Create custom API hosted by your CF or Kyma app and call via http client from ABAP Cloud
Serve directly from ABAP Code via XCO	Base64-encode your file content, wrap into ABAP code, and serve as XCO class. Lars likes it at least 😜. There were sarcastic smiles involved and some more “oh please”, so take it not too seriously.
Raise an influencing request at SAP to release something like the former NetWeaver MIME repos	Live the dream

A common theme among all the options is the need to interact with them from ABAP Cloud via the built-in http client. On the downside some options require an additional app on CF or Kyma to orchestrate the storage interactions.

Ideally ABAP Cloud integrates directly with the storage account to reduce complexity and maintenance.

You guessed rightly my own proposal focusses on direct integration with Azure Blob

To get started with this sample I ran through the SAP developer tutorial “Create Your First ABAP Cloud Console Application” and steps 1-6 of “Call an External API and Parse the Response in SAP BTP ABAP Environment. This way you can easily reproduce from an official reference.

Got your hello world on Eclipse? Great, onwards, and upwards in the stack we go then 🪜. Or down to the engine room – that depends on your perspective.

All the blob storage providers offer various options to authenticate with the service. See the current coverage for Azure here.

Fig.1 Screenshot of supported authentication methods for Azure Storage

The Microsoft Entra ID option offers superior security capabilities compared to access keys – which can be leaked or lost for example – and is therefore recommended by Microsoft.

For developer ease, I left the code using the simpler to configure “Shared-Access-Signature (SAS) tokens” commented on the shared GitHub repos. SAS tokens can be created from the Azure portal with two clicks.

The shared key approach requires a bit of hashing and marshaling on ABAP. Use the ABAP SDK for Azure to accelerate that part of your implementation. Check the “get_sas_token” method for reference.

Anonymous read access would only be ok for less sensitive content like static image files or the likes because anyone can access them once they have the URL.

For an enterprise-grade solution however, you will need to use a more secure protocol like OAuth2 with Microsoft Entra ID

Technically you could do the OAuth2 token fetching with plain http-client requests from ABAP Cloud. See this blog by jacek.wozniczak for instance. However, it is recommended to use the steampunk “Communication Management” to abstract away the configuration from your code. Think “external configuration store”. Also, it reduces the complexity of your ABAP code, because Communication Management handles the OAuth2 flow for you.

📢Note: SAP will release the needed capability to maintain OAuth2 scopes in communication arrangements as part of your ABAP Cloud requests with the upcoming SAP BTP, ABAP environment 2402.

So, till then you will need to use the BTP Destination service. Target destinations living on subaccount level by calling them like so (omitting the i_service_instance_name, thank you thwiegan for calling that out here😞

destination = cl_http_destination_provider=>create_by_cloud_destination(

        i_name = |azure-blob|

        i_authn_mode = if_a4c_cp_service=>service_specific

    ).

Or call destinations living on Cloud Foundry spaces like so:

destination = cl_http_destination_provider=>create_by_cloud_destination(

        i_name = |azure-blob|

        i_service_instance_name = |SAP_BTP_DESTINATION|

        i_authn_mode = if_a4c_cp_service=>service_specific

    ).

For above Cloud Foundry variation you need to deploy the “standard” communication scenario SAP_COM_0276. My generated arrangement id in this case was “SAP_BTP_DESTINATION”.

Be aware, SAP marked the approach with BTP destinations as deprecated for BTP ABAP. And we can now see why. It will be much nicer doing it from the single initial communication arrangement only, rather than having the overhead with additional services and arrangements. Looking forward to that in February 😎

Not everything is “bad” about using BTP destinations with ABAP Cloud though. They have management APIs, which the communication arrangements don’t have yet. Also, re-use of APIs across your BTP estate beyond the boundary of your ABAP Environment tenant would be useful.

A fully automated solution deployment with the BTP and Azure terraform providers is only possible with the destination service approach as of today.

See this TechEd 2023 session and watch this new sample repos (still in development) for reference.

The application flow is quite simple once the authentication part is figured out

Access your communication management config from your ABAP web Ui:

https://your-steampunk-domain.abap-web.eu20.hana.ondemand.com/ui#Shell-home

Steampunk supports the typical set of authentication flows for outbound communication users using http that you are used to from BTP. I chose the OAuth2 Client Credentials grant because that is most widely referenced in the BTP world and reasonably secure.

Fig.2 ABAP Cloud API flow including OAuth2 token request from Microsoft Entra ID

Since I am integrating with an Azure Storage account, I will need to authenticate via Microsoft Entra ID (formerly known as Azure Active Directory).

Yes, Microsoft likes renaming stuff from time to time, too 😉.

Using the Azure Storage REST API I can create, update, delete, and list files as I please.

The Entra ID setup takes a couple of clicks

Create a new App registration from Microsoft Entra ID service on your Azure portal and generate a new secret. Beware of the expiry date!

Below preferred option will start working once SAP adds the scope parameter for OAuth2 Client Credentials grant as described before.

Fig.3 Screenshot of attribute and secret mapping for ABAP Cloud Outbound user

For now, let’s have a look at a destination on subaccount level instead. Be aware the scope parameter needs to be “https://storage.azure.com/.default” (see fig.4 below, additional properties section called “scope” on the bottom right). That is also the setting that we are missing for the preferred approach mentioned above.

The standard login URLs for OAuth token endpoints on Microsoft Entra ID are the following:

https://login.microsoftonline.com/your-tenantId/oauth2/v2.0/token

https://login.microsoftonline.com/your-tenantId/oauth2/v2.0/authorize

Fig.4 Screenshot of attribute mapping from Entra ID to SAP BTP Destination

So far so good. Let’s roll the integration test from our ABAP console application on Eclipse (ADT).

Fig.5 Screenshot of file interaction from ABAP Cloud and data container view on Azure

Excellent, there is our booking request: Safe and sound stored as Azure Blob, posted from ABAP, and read again seamlessly.

See the shared Postman collection to help with your integration testing.

Thoughts on production readiness

The biggest caveat is the regularly required OAuth2 client credential secret rotation. Unfortunately, credential-free options with Azure Managed Identities are not possible, because BTP is hyperscaler-agnostic and does not expose the underlying Azure components to you.

Some of you might say next: let’s use client certificates with “veeery long validity time frames like 2038” to push out the problem beyond so far out someone else will have to deal with it. Well, certificate lifetimes get reduced more and more (TLS certs for instance have a maximum of 13 months at DigiCert since 2020) and you have to rotate them eventually, too 😉. With shorter certificate lifetimes more secure hashing algorithms come into effect much quicker for instance.

I will dedicate a separate post on client certificates (mTLS) with steampunk to consume Azure services.

What about federated identities? You could configure trust between your SAP Cloud Identity Service (or Steampunk auth service) and Microsoft Entra ID to allow requests from ABAP Cloud to authorize Azure services. However, that would be a more complex configuration with implications for your overall setup causing larger integration test needs. And we embarked on this journey to discover a simple solution not too far away from AL11 and the likes, right? 😅

See a working implementation of federated identities with SAP Cloud Identity service consuming Microsoft Graph published by my colleagure mraepple in his blog series here.

Ok, then let’s compromise and see how we can automatically rotate secrets. Azure Key Vault exposes events for secrets, keys, and certificates to inform downstream services about due expiry. With that a small low code app can be provided to perform the secret update. See below sample that went the extra mile asking the admins via Microsoft Teams if they wanted to perform the change or not:

Fig.6 Architecture of secret rotation with Azure Key Vault and secret refresh approval

A new secret for the app registration on Entra can be generated with the Microsoft Graph API like so. See this post for details on the Azure Key Vault aspects of the mix.

To apply that flow and propagate the new secret to steampunk, we need to call BTP APIs to save the new secret. See the BTP REST API for Destinations here to learn about the secret update method.

Have a look at my earlier blog post for specifics on how to do the same with certificates.

Estimated cost for such a secret rotation solution for 1000 rotations per month is around 2$ per month. With simpler configurations and less rotations, it can be covered by free tiers even.

Once you have applied the means of automation as discussed above you may incorporate this into your DevOps process and live happily ever after with no manual secret handling 😊.

Final Words

That’s a wrap 🌯you saw today how – in the absence of an application server file system and NetWeaver MIME repository (good old days) – you can use Azure Storage Account as your external data store from BTP ABAP Environment (steampunk) using ABAP Cloud. In addition to that, you gained insights into the proper setup for authentication and what flavors are supported by steampunk now. You got a glimpse into automated deployment of the solution with the BTP and Azure terraform provider.

To top it up you learnt what else is needed to operationalize the approach at scale with regular secret/certificate rotation.

Check SAP’s docs for external APIs with steampunk for further official materials.

What do you think dpanzer and lars.hvam? Not too bad, is it? 😉

Find all the resources to replicate this setup on this GitHub repos. Stay tuned for the remaining parts of the steampunk series with Microsoft Integration Scenarios from my overview post.

Cheers

Martin

Child's play: Install SAP BTP transparent proxy using Helm

2024-01-15T16:14:28+01:00

It is inevitable that cloud solutions have to communicate with other remote solutions. The latter can be situated on public or private clouds, or set up on client sites. Of course, it would be easier to have levers to facilitate this in the simplest possible way. This is where the SAP BTP Connectivity services and components come to the rescue! In this blog post, you will understand how to install one of these components using Helm: the SAP BTP transparent proxy.

SAP BTP transparent proxy simplifies the connection between Kubernetes workloads and target systems defined as destinations in the SAP Destination service. To understand more about some of the features of the Transparent Proxy, you could check this blog.

Prerequisites

Before you start, you should have the following:

A Kubernetes cluster

Kubectl installed and configured on your local machine

Helm installed on your local machine

SAP BTP subaccount with a Destination service instance and a Connectivity Proxy instance (optional for on-premise connectivity)

Istio or cert-manager running in your Kubernetes cluster as a foundation for traffic encryption between the micro-components of the Transparent Proxy

Connectivity proxy installed in your cluster (optional for on-premise connectivity)

Installation steps

To install the Transparent Proxy using Helm, follow these steps:

Create a namespace for the Transparent Proxy in your Kubernetes cluster. For example:
```
kubectl create namespace transparent-proxy
```

Create a Kubernetes secret with the credentials of your Destination service instance
- You can obtain the credentials from the SAP BTP cockpit. Navigate to Services -> Instances and Subscriptions -> Click on the service instance row -> Service Keys -> View -> Copy JSON
- Use the service key data to create the Kubernetes secret. For example:
```
kubectl create secret generic dest-svc-key -n transparent-proxy --from-literal=secret='<credentials>'
```

Create values.yaml according to your needs. You can find all available parameters here. For example:

deployment:

  autoscaling:

    http:

      horizontal:

        # Enables or disables the Horizontal Pod Autoscaler mechanism.

        enabled: true

        # Upper limit for the number of HTTP Transparent Proxy replicas to which the autoscaler can scale up.

        maxReplicaCount: 3

        metrics:

          # Target value of the average CPU metric across all Transparent HTTP Proxy pods, represented as a percentage of the requested value of the CPU for the pods.

          cpuAverageUtilization: 80

          # Target value of the average memory metric across all Transparent HTTP Proxy pods, represented as a percentage of the requested value of the memory for the pods.

          memoryAverageUtilization: 80

    tcp:

      horizontal:

        # Enables or disables the Horizontal Pod Autoscaler mechanism.

        enabled: true

        # Upper limit for the number of TCP Transparent Proxy replicas to which the autoscaler can scale up.

        maxReplicaCount: 3

        metrics:

          # Target value of the average CPU metric across all Transparent TCP Proxy pods represented as a percentage of the requested value of the CPU for the pods.

          cpuAverageUtilization: 80

          # Target value of the average memory metric across all Transparent TCP Proxy pods represented as a percentage of the requested value of the memory for the pods.

          memoryAverageUtilization: 80

config:

  # Defines the tenant mode in which Transparent Proxy is working in. The option "dedicated" shows that the proxy works in single-tenant mode.

  tenantMode: "dedicated"

  security:

    accessControl:

      destinations:

         ## Defines the scope of Destination CRs.

        defaultScope: "clusterWide"

    communication:

      internal:

        # Enables/Disables mTLS communication between the Transparent Proxy micro-components.

        # It may be disabled only in test environments or if you want to integrate with a Service mesh like Istio.

        encryptionEnabled: true

        certManager:

          issuerRef:

            name: <cert-manager issuer name>

            kind: ClusterIssuer

          # Certificate properties used by cert-manager's Certificate controller.

          certificate:

            privateKey:

              algorithm: ECDSA

              encoding: PKCS8

              size: 256

            duration: 720h

            renewBefore: 120h

  manager:

    # The interval on which the Transparent Proxy will check for updates in the Destination service instance. 

    executionIntervalMinutes: 3

  integration:

    destinationService:

      instances:

        # The local cluster name of the Destination service instance, which can later be used as a reference in the Destination CR

      - name: dest-service-instance

        serviceCredentials:

          # The key in the Destination service secret resource, which holds the value of the destination service key.

          secretKey: secret

          # The name of the existing secret, which holds the credentials for the Destination service.

          secretName: dest-svc-key

          # The namespace of the secret to be used, which holds the credentials for the Destination service.

          secretNamespace: transparent-proxy

    connectivityProxy:

      # The Kubernetes service name + namespace that are associated with the Connectivity Proxy workload.

      serviceName: <connectivity proxy service name>.<connectivity proxy namespace>

      # The port on which the HTTP interface of the Connectivity Proxy is started.

      httpPort: 20003

      # The port on which the TCP interface of the Connectivity Proxy is started.

      tcpPort: 20004

Install the Transparent Proxy using the Helm values from step 3:

helm install transparent-proxy oci://registry-1.docker.io/sapse/transparent-proxy --version <version of helm chart> --namespace transparent-proxy -f <path-to-values.yaml>

You should receive a similar to this response:

Successful installation of Transparent Proxy with Helm

Verify that the Transparent Proxy is running by checking the status of the pods and the health check:
```
kubectl get pods -n transparent-proxy
```
There should be two pods running:

Transparent Proxy components after installation

As you can see, the Transparent Proxy has a health check pod which constantly checks the status of all Transparent Proxy components. You can look at what capabilities the health check has in the Verification and Testing page in the Help portal. Here's how you can execute a component check:
```
kubectl run perform-hc --image=curlimages/curl -it --rm --restart=Never -- curl -w "\n" 'sap-transp-proxy-int-healthcheck.transparent-proxy/status'
```
And the result should be the following:

This means that the sap-transp-proxy-manager, the heart of the Transparent Proxy, is running smoothly and you are ready to consume your first target system through the Transparent Proxy!

Try it out

To use the Transparent Proxy, you should create a Destination Custom Resource (CR). Let's create a dynamic one. "Dynamic" means a Destination CR will serve all destinations for a Destination service instance or its tenants. Follow these steps:

Create a Destination CR file with name dynamic-destination.yaml:

apiVersion: destination.connectivity.api.sap/v1

kind: Destination

metadata:

  name: dynamic-destination

  namespace: transparent-proxy

spec:

  destinationRef:

    name: "*"

  destinationServiceInstanceName: dest-service-instance

Create the resource from step 1 in your cluster:
```
kubectl create -f dynamic-destination.yaml
```

Wait for a successful status of the Destination CR. To check it execute:

kubectl get dst dynamic-destination -n transparent-proxy -o yaml

You should observe a status similar to this one:

status:

  conditions:

  - lastUpdateTime: "2024-01-11T11:56:33.605473101Z"

    message: Technical connectivity is configured. Kubernetes service with name

      dynamic-destination is created.

    reason: ConfigurationSuccessful

    status: "True"

    type: Available

Create a curl pod, from where you can test the consumption of the target system through the Transparent Proxy:
```
kubectl run curlpod -n transparent-proxy --image=curlimages/curl -n transparent-proxy -i --tty -- sh
```

Consume a target system defined as a destination in your Destination service instance:
```
curl dynamic-destination -H "X-Destination-Name: <destination-name>"
```

Examples

In the context of my SAP BTP subaccount, I have created two destinations: one pointing to the SAP XSUAA API, and another pointing to a server on my local machine, exposed to the cloud via the SAP Cloud Connector.

Configured destinations in the SAP BTP Cockpit

Executing a request to the SAP XSUAA API. This is an example for getting the registered service instances of the current subaccount via destination, locally exposed and served by Transparent Proxy, and centrally managed via the SAP Destination service:

~ $ curl dynamic-destination/sap/rest/authorization/v2/apps -H "X-Destination-Name: xsuaa-api" -v

* Host dynamic-destination:80 was resolved.

...

> GET /sap/rest/authorization/v2/apps HTTP/1.1

> Host: dynamic-destination

> User-Agent: curl/8.5.0

> Accept: */*

> X-Destination-Name: xsuaa-api

> 

< HTTP/1.1 200 OK

...

[{"appid":"auditlog!b3718","serviceinstanceid":"0889a7e7-61d8-41...

Executing a request to an on-premise system using principal propagation. That system is a simple server that maps the user certificate to a concrete user. The current response greets the requestor.

~ $ curl dynamic-destination/principal-propagation -H "X-Destination-Name: my-on-premise-system" -H "Authorization: Bearer $TOKEN" -v

* Host dynamic-destination:80 was resolved.

...

* Connected to dynamic-destination (10.104.69.106) port 80

> GET /principal-propagation HTTP/1.1

> Host: dynamic-destination

> User-Agent: curl/8.5.0

> Accept: */*

> X-Destination-Name: my-on-premise-system

> Authorization: Bearer eyJhbGciOiJSUzI1NiIsImprdS...

...

< HTTP/1.1 200 OK

...

Hello Iliyan Videnov!

In this blog post, you have learned how to install SAP BTP transparent proxy using Helm, and how to easily set up it for system consumption. I hope you find it useful and enjoy it. Ideas, suggestions, and comments are welcome. Thank you for reading!

New Release Available: SAP Cloud Connector 2.17.0

2024-05-10T14:33:54.505000+02:00

We are happy to announce that the fresh version of the SAP Cloud Connector is now available for download. It is (as usual) packed with a host of new features and improvements. From bug fixes to enhancements, we've worked diligently to deliver an updated connector that addresses critical issues while also enhancing usability and functionality, which you can find more detailed in the release notes.
Moving onto enhancements, we've made changes to the underlying architecture and features of the Cloud Connector. One of the major changes is the switch from JavaWeb 3.x runtime on Tomcat 8.5 to JavaWeb 4.x, which operates on Tomcat 9. This new runtime container facilitates better performance and stability.
In addition, Cloud Connector 2.17 now supports the use of SapMachine 21 as Java runtime. This change can provide increased efficiency and flexibility for your operations.
One of the significant enhancements in this release is the addition of support for up to 3 LDAP servers for authentication.

This feature is essential for setups where the user base is spread across multiple LDAP user stores or multiple user bases in a single LDAP user store (find more in the documentation).
We've also introduced the option to configure a separate port for the HA-related communication between the master and shadow instances.

This new feature enables you to use HA in conjunction with certificate-based authentication to overcome the limitation from 2.16 (find more here)
Additional hardware monitoring REST APIs have been provided for disk and CPU status, allowing for more comprehensive system insights. Plus, you can now use the hardware monitor on the shadow instance as well.
To improve your operations, we’ve introduced for the access control settings a creation timestamp, providing more detailed and useful information for your operations:

Finally, we've improved the Cloud Connector UI by adding a session expiration progress bar. This new addition helps you keep track of your active session and alerts you when you need to log in again.

In summary, with this release, we've not only ensured the highest security levels but also worked on improving the overall user experience and functionality. Don't hesitate to upgrade your Cloud Connector to version 2.17.0 today (by downloading it from here) and explore these new features and improvements. For more detailed information, make sure to check out the official release notes. Enjoy the enhanced performance and functionality of the new SAP Cloud Connector!Happy Connecting!

Configuration as code (CaC) with destinations.

2024-05-13T12:54:57.123000+02:00

Configuration as code (CaC) with destinations.

Destinations are very handy and powerful mechanism to facilitate access to target systems and devices.

When it comes to SAP BTP destinations, the idea is to manage both subaccount and instance level destinations (and/or their certificates) as shared configuration resources on a provider subaccount level.

That way, the destinations configurations can be stored as versioned assets in a source repository and need to be maintained only once per provider, thus, without incurring application runtime tie-in.

Last but not least, BTP destination service is used as a self-configuration tool.

Configuration as code with SAP BTP destination service

Table of Contents

Configuration as code with SAP BTP destination service.

create shared destination service instance and binding.

Provision bootstrap destinations.

Configure destination resources.

Documentation.

PS.

Bootstrap destinations definitions.

Even, if there is no intrinsic BTP CLI command to assist in creation of destinations from service bindings, this can be achieved quite easily with a bit of jq gimmick by applying service binding credentials to a json payload template, for instance:

{
    "init_data": {
        "subaccount": {
            "destinations": [
                  {
                    "Description": "dest-httpbin",
                    "Type": "HTTP",
                    "clientId": "sb-clone12847c4c89544b4f9234b26ede429f62!b282590|destination-xsappname!b62",
                    "HTML5.DynamicDestination": "true",
                    "HTML5.Timeout": "60000",
                    "Authentication": "OAuth2ClientCredentials",
                    "Name": "dest-httpbin",
                    "tokenServiceURL": "https://<subdomain>.authentication.us10.hana.ondemand.com/oauth/token",
                    "ProxyType": "Internet",
                    "URL": "https://httpbin.org",
                    "tokenServiceURLType": "Dedicated",
                    "clientSecret": "<clientSecret>"
                  },
                  {
                    "Description": "SAP Destination Service APIs",
                    "Type": "HTTP",
                    "clientId": "sb-clone12847c4c89544b4f9234b26ede429f62!b282590|destination-xsappname!b62",
                    "HTML5.DynamicDestination": "true",
                    "HTML5.Timeout": "60000",
                    "Authentication": "OAuth2ClientCredentials",
                    "Name": "destination-service",
                    "tokenServiceURL": "https://<subdomain>.authentication.us10.hana.ondemand.com/oauth/token",
                    "ProxyType": "Internet",
                    "URL": "https://destination-configuration.cfapps.us10.hana.ondemand.com/destination-configuration/v1",
                    "tokenServiceURLType": "Dedicated",
                    "clientSecret": "<clientSecret>"
                  }
            ],
           "certificates": [
           ],

            "existing_certificates_policy": "update",
            "existing_destinations_policy": "update"           
       }
   }
}

Alternatively, one could resort to using SAP Cloud SDK built-in service binding destinations.

The below nodejs code snippet demonstrates how to leverage SAP Cloud SDK with its service binding destinations with the likes of service manager and destinations services.

apiVersion: serverless.kyma-project.io/v1alpha2
kind: Function
metadata:
  name: {{ .Values.services.srv.name }}
  labels:
    {{- include "app.labels" . | nindent 4 }}
    app: {{ .Values.services.srv.name }}
spec:
  runtime: {{ .Values.services.srv.runtime }}
#  runtimeImageOverride: {{ .Values.services.srv.runtimeImageOverride }}
  source:
    inline:
      dependencies: |
        {
          "name": "{{ .Values.services.srv.name }}",
          "version": "0.0.1",
          "dependencies": {
            "axios":"latest"
            ,"debug": "latest"
            ,"@sap/xsenv": "latest"
            ,"@sap-cloud-sdk/http-client": "latest"
            ,"@sap-cloud-sdk/connectivity": "latest"
            ,"@sap-cloud-sdk/resilience": "latest"
            ,"async-retry": "latest"
          }
        }
      source: |
        const debug = require('debug')('{{ .Values.services.srv.name }}:function');
        const NOT_FOUND = 'Not Found';
        const xsenv = require('@sap/xsenv');

        const services = xsenv.getServices({
          sm: { label: 'service-manager', name: 'saas-sm' }
          ,
          dest: { label: 'destination' }

        });
        console.log('saas-sm: ', services.sm);

        const readServices = xsenv.readServices();
        console.log('readServices: ', readServices);

        const httpClient = require('@sap-cloud-sdk/http-client');

        const cloudSdkConnectivity = require('@sap-cloud-sdk/connectivity');
        const { retrieveJwt, decodeJwt, Destination } = require('@sap-cloud-sdk/connectivity');
        const { setGlobalLogLevel, createLogger } = require('@sap-cloud-sdk/util');
        const { retry } = require ('@sap-cloud-sdk/resilience');
        const { resilience } = require ('@sap-cloud-sdk/resilience');
        const ResilienceOptions = {
          retry: 10,
          circuitBreaker: false,
          timeout: 300*1000 // 5 minutes in milliseconds
        };          

        const retryme = require('async-retry');

        setGlobalLogLevel('debug');
        const logger = createLogger('http-logs');

        module.exports = {
          main: async function (event, context) {
            const req = event.extensions.request;

            const message = `Hello World`
              + ` from the Kyma Function ${context['function-name']}`
              + ` running on ${context.runtime}!`
              + ` with the request headers ${JSON.stringify(req.headers,0,2)}`;
            console.log(message);
            
            if (typeof req.path !== undefined) {
              console.log('path: ', JSON.stringify(req.path,0,2))
            }
            if (typeof req.params !== undefined) {
              console.log('params: ', JSON.stringify(req.params,0,2))
            }
            if (typeof req.url !== undefined) {
              console.log('url: ', JSON.stringify(req.url,0,2))
            }
            if (typeof req.authInfo !== undefined) {
              console.log('authInfo: ', JSON.stringify(req.authInfo,0,2))
            }

            const { pathname } = new URL(req.url || '', `https://${req.headers.host}`)
            console.log('pathname: ', pathname)

            const url = require("url");
            var url_parts = url.parse(req.url);
            console.log(url_parts);
            console.log(url_parts.pathname);

            // returns an array with paths
            let path_array = req.url.match('^[^?]*')[0].split('/').slice(1);
            console.log(path_array)

            console.log(req.url.match('^[^?]*')[0])

            if (!path_array?.length) return 'Please use an API verb';  
            const actions = [ 
               { name: 'offerings', verb: 'service_offerings', dest:  'saas-sm', url: '/v1/' },
               { name: 'plans', verb: 'service_plans', dest:  'saas-sm', url: '/v1/'  },
               { name: 'instances', verb: 'service_instances', dest:  'saas-sm', url: '/v1/'  },
               { name: 'bindings', verb: 'service_bindings', dest:  'saas-sm', url: '/v1/'  },
               { name: 'instanceDestinations', verb: 'instanceDestinations', dest:  'faas-dest-x509', url: '/destination-configuration/v1/'  },
               { name: 'subaccountDestinations', verb: 'subaccountDestinations', dest:  'faas-dest-x509' , url: '/destination-configuration/v1/' }
            ];
            
            const action = actions.find( ({ name }) => name === path_array[1] )
            console.log('action found: ', action)

            if (path_array[0] == 'srv' &&  action !== undefined) {

              path_array = req.url.match('^[^?]*')[0].split('/').slice(2);

              console.log('path_array: ', path_array)

              const queryString = req.query;
              console.log('queryString: ', queryString)
              const urlParams = new URLSearchParams(queryString);

              const params = req.params;
              console.log('params: ', params) 

              try {
                  // https://sap.github.io/cloud-sdk/docs/js/features/connectivity/destinations#service-binding-environment-variables
                  const endpoint =  path_array[1] !== undefined ? '/' + path_array[1] : '';
                  console.log(endpoint)
                  let res = await httpClient.executeHttpRequest({ destinationName: action.dest }, {
                      method: 'GET',
                      url: action.url + action.verb + endpoint
                  });
                  return res.data;
              } catch (err) {
                  console.log(err.stack);
                  return err.message;
              }
              
            }            
          }
        }
   scaleConfig:
    maxReplicas: 5
    minReplicas: 3
  resourceConfiguration:
    function:
      profile: S
  env: ## https://kyma-project.io/docs/kyma/latest/05-technical-reference/00-configuration-parameters/svls-02-environment-variables/#node-js-runtime-specific-environment-variables
    - name: FUNC_TIMEOUT ## Specifies the number of seconds in which a runtime must execute the code.
      value: '1800'
    - name: REQ_MB_LIMIT ## payload body size limit in megabytes.
      value: "10"

    - name: DEBUG
      value: '{{ .Values.services.srv.name }}:*'
    - name: SERVICE_BINDING_ROOT
      value: /bindings
    

  secretMounts: 
    - secretName: {{ .Values.services.sm.bindingSecretName }}
      mountPath: "/bindings/saas-sm"
    - secretName: {{ .Values.services.dest.bindingSecretNamex509 }}
      mountPath: "/bindings/faas-dest-x509"

SAP Community - SAP Connectivity service

Surviving and Thriving with the SAP Cloud Application Programming Model: Deployment to SAP BTP, Kyma runtime

Common Ground: SAP BTP, Cloud Foundry runtime and SAP BTP, Kyma runtime

Deployment differences: SAP BTP, Cloud Foundry runtime and SAP BTP, Kyma runtime

What's Paketo?

What's Helm?

How do I deploy my application now?

Can I use the Cloud Connector with SAP BTP, Kyma Runtime?

How does CAP know where to get the service binding information from?

SAP Event Mesh and SAP S/4HANA on-premise - little detour needed

Mind your DB Pool configuration!

SAP Data Intelligence Python Operators and Cloud Connector – TCP

Access TCP-based resources:

Prerequisite:

1. Configuration:

2. Creating a Data Intelligence Connection:

3. Developing a Custom Operator:

4. Testing the Custom Operator:

SFTP via Cloud Connector Python Operator in SAP Data Intelligence

Connect Python Operator to SFTP via Cloud Connector:

Prerequisite:

1. Configuration:

2. Creating a Data Intelligence Connection:

3. Developing a Custom Operator:

4. Testing the Custom Operator:

Proxy Third-Party Python Library Traffic - Using SAP Cloud Connector in SAP Data Intelligence Python Operators

Prerequisite:

1. Configuration:

2. Creating a Data Intelligence Connection:

3. Developing a Custom Operator:

4. Test the custom operator:

Surviving and Thriving with CAP: Destinations and the Transparent Proxy for Kubernetes

It's about CAP because it's not about CAP

Destination usage with plain Node.js with @Sap/xsenv

Destination usage with the SAP Cloud SDK for JavaScript

Destination usage with SAP Cloud Application Programming Model (CAP) - Node.js & Java

So, Max, what's the Transparent Proxy for Kubernetes and why should I use it?

How can I use it and how does it work?

Takeaway:

Consuming a REST service from a CAP application

Use Case

Data Model

Service

Consuming the REST Service

Custom Handler

Summary

Easy way to access destinations in SAP BTP cloud foundry via Node.js

Introduction

Solution

Integration Architecture For Enterprise Agility Powered By SAP BTP

Introduction

1. Design thinking - Integration Architecture

2. Explore BTP services used in the Integration Architecture

SAP Integration Suite

Event Driven Architecture

Connectivity

Conclusion

Next in the SAP Garage – Build Events-to-Business Actions Apps with SAP BTP and MS Azure/AWS.

Next release of the Cloud Connector is available: 2.16.0

SAP BTP client certificate destination for an IBP communication scenario: side-by-side extensibility

Introduction

Generate and download the Certificate via Connectivity service

Upload the Certificate in Target System (SAP IBP in our scenario)

Create the Destination

Side By Side: Test the destination in CAP / Application router

Appendix

Next in the SAP Garage – Integrate SAP S/4HANA with Microsoft Teams via SAP Business Technology Platform.

Easy way to provide schema access to a BTP HANA Cloud Standard User

Problem

Solution

References:

SAP BTP Destinations in a nutshell Part 3 - OAuth 2.0 Client Credentials

Introduction

OAuth 2.0

4 Roles

Client Credentials Flow

XSUAA

Cloud 2 Cloud - OAuth 2.0 Client Credentials

Server Implementation

Destination