--- description: The template will log API calls from Azure API Management to Moesif API analytics and monetization platform page_type: sample products: - azure - azure-resource-manager urlFragment: api-management-logs-to-moesif-using-eventhub-webapp languages: - bicep - json --- # Moesif API Analytics and Monetization ![Azure Public Test Date](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/PublicLastTestDate.svg) ![Azure Public Test Result](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/PublicDeployment.svg) ![Azure US Gov Last Test Date](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/FairfaxLastTestDate.svg) ![Azure US Gov Last Test Result](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/FairfaxDeployment.svg) ![Best Practice Check](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/BestPracticeResult.svg) ![Cred Scan Check](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/CredScanResult.svg) ![Bicep Version](https://azurequickstartsservice.blob.core.windows.net/badges/quickstarts/microsoft.apimanagement/api-management-logs-to-moesif-using-eventhub-webapp/BicepVersion.svg) [![Deploy To Azure](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazure.svg?sanitize=true)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.apimanagement%2Fapi-management-logs-to-moesif-using-eventhub-webapp%2Fazuredeploy.json) [![Deploy To Azure US Gov](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazuregov.svg?sanitize=true)](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.apimanagement%2Fapi-management-logs-to-moesif-using-eventhub-webapp%2Fazuredeploy.json) [![Visualize](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/visualizebutton.svg?sanitize=true)](http://armviz.io/#/?load=https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.apimanagement%2Fapi-management-logs-to-moesif-using-eventhub-webapp%2Fazuredeploy.json) With Moesif, grow and monetize your API products with a powerful API analytics and billing platform. This template will log API traffic from Azure API Management using [Moesif API Analytics & Monetization](https://www.moesif.com/?language=azure-api-management) in a few minutes without any code changes or restarts. The integration also provides visibility into rejected requests that never reach your underlying service. ## How it works This solution is deployed using an [Azure Resource Manager Template](https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/api-management-logs-to-moesif-using-eventhub-webapp/). An XML Policy configures an APIM logger to send API logs to an Azure EventHub. An Azure WebJob reads from the EventHub and sends to Moesif for data processing. [More info on this integration](https://www.moesif.com/implementation/log-http-calls-from-azure-api-management?platform=azure-management). ![Architecture Diagram Logging API Calls from Azure API Management](images/azure-api-management-logging-architecture-diagram.png) ## How to install ### 1. Start Azure Resource Deployment Click the below button to start a Custom deployment with the Moesif Azure Resource Template. [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.apimanagement%2Fapi-management-logs-to-moesif-using-eventhub-webapp%2Fazuredeploy.json) ### 2. Configure Parameters Within the Azure Template Deployment panel, set the following properties: * Set _Resource group_ to the same resource group that contains your existing Azure APIM instance. This ensures the APIM logger, `moesif-log-to-event-hub`, is automatically created for you. * Set _Moesif Application Id_ to the one displayed after logging into your Moesif account. You can create a free one on [Moesif's website](https://www.moesif.com/?language=azure-api-management) * Set _Existing Api Mgmt Name_ to the name of your Azure APIM instance. If blank, you will need to manually create the [APIM logger](https://docs.microsoft.com/azure/api-management/api-management-log-to-eventhub-sample#policy-declaration). Once done, click the _Review+create_ button at the bottom and finish the template creation wizard. > Occasionally, Azure reports a failed deployment due to slow propagation of new DNS settings even though everything was deployed successfully. We recommend proceeding with rest of process. If you still have issues after last step, [view troubleshooting](https://www.moesif.com/docs/server-integration/azure-api-management/#troubleshooting). ### 3. Add XML Policy Within the Azure portal, navigate to your existing Azure API Management instance. Select either a single API you want to add Moesif logging to, or select _All APIs_ to add Moesif globally. > It's recommended to add the XML policy globally for all APIs to reduce code replication. Then, use Moesif [dynamic sampling](https://www.moesif.com/docs/platform/dynamic-sampling/) to control what data is collected based on user behavior or regex rules. Under the _Inbound Processing_ section, click the _Add Policy_ button. Select _Other Policies_ which will open the XML editor. Paste in the XML code you see below. More info on editing APIM policies is available on the [Azure docs](https://docs.microsoft.com/azure/api-management/set-edit-policies) ```xml @{ var body = context.Request.Body?.As(true); var MAX_BODY_EH = 145000; var origBodyLen = (null != body) ? body.Length : 0; if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH); } var headers = context.Request.Headers .Where(h => h.Key != "Ocp-Apim-Subscription-Key") .Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray(); var jwtToken = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt(); var userId = (context.User != null && context.User.Id != null) ? context.User.Id : (jwtToken != null && jwtToken.Subject != null ? jwtToken.Subject : string.Empty); var companyId = (context.Subscription != null && context.Subscription.Id != null) ? context.Subscription.Id : string.Empty; var cru = new JObject(); if (context.User != null) { cru.Add("Email", context.User.Email); cru.Add("Id", context.User.Id); cru.Add("FirstName", context.User.FirstName); cru.Add("LastName", context.User.LastName);} var crus = System.Convert.ToBase64String(Encoding.UTF8.GetBytes(cru.ToString())); var requestBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty); return new JObject( new JProperty("event_type", "request"), new JProperty("message-id", context.Variables["moesif-message-id"]), new JProperty("method", context.Request.Method), new JProperty("ip_address", context.Request.IpAddress), new JProperty("uri", context.Request.OriginalUrl.ToString()), new JProperty("user_id", userId), new JProperty("contextRequestUser", crus), new JProperty("company_id", companyId), new JProperty("request_headers", string.Join(";;", headers)), new JProperty("request_body", requestBody), new JProperty("contextTimestamp", context.Timestamp.ToString("o")), new JProperty("metadata", $@"") ).ToString();} @{ var body = context.Response.Body?.As(true); var MAX_BODY_EH = 145000; var origBodyLen = (null != body) ? body.Length : 0; if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);} var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray(); var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty); return new JObject( new JProperty("event_type", "response"), new JProperty("orig_body_len", origBodyLen), new JProperty("message-id", context.Variables["moesif-message-id"]), new JProperty("status_code", context.Response.StatusCode), new JProperty("response_headers", string.Join(";;", headers)), new JProperty("contextTimestamp", context.Timestamp.Add(context.Elapsed).ToString("o")), new JProperty("response_body", responseBody) ).ToString();} @{ var body = context.Response.Body?.As(true); var MAX_BODY_EH = 145000; var origBodyLen = (null != body) ? body.Length : 0; if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);} var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray(); var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty); return new JObject( new JProperty("event_type", "response"), new JProperty("orig_body_len", origBodyLen), new JProperty("message-id", context.Variables["moesif-message-id"]), new JProperty("status_code", context.Response.StatusCode), new JProperty("response_headers", string.Join(";;", headers)), new JProperty("contextTimestamp", context.Timestamp.Add(context.Elapsed).ToString("o")), new JProperty("response_body", responseBody) ).ToString();} ``` That's it. Once the XML is added to your APIs, the logs should start showing up in Moesif. ## Configuration Options The below fields in the XML policy can be modified by you to meet your requirements. XML Policies support a number of [context variables](https://docs.microsoft.com/azure/api-management/api-management-policy-expressions#ContextVariables) which you can also leverage. ### User Id _String_, The field `user_id` identifies who the user is making the API and enables Moesif to associate API calls to user profiles. The default XML policy extracts the user id from the `context.User.Id` or the Subject of the JWT Token. If you are a B2B company, this can be used simultaneously with company id to track API Usage both at the individual user-level and also account-level. See [identifying customers in Moesif](https://www.moesif.com/docs/getting-started/identify-customers/) for more info. ### User Metadata _String_, The field `contextRequestUser` allows you to store additional user metadata as part of the [user's profile in Moesif](https://www.moesif.com/docs/getting-started/users/). By default, the XML policy also saves Email, FirstName, and LastName from Azure's `context.User` object. Any fields can be stored. Keep in mind `contextRequestUser` is expecting a base64 encoded string. ### Company Id _String_, The field `company_id` identifies which company is making the API and enables Moesif to associate API calls to company profiles. The default XML policy does not set this field. See [identifying customers in Moesif](https://www.moesif.com/docs/getting-started/identify-customers/) for more info. ### Event Metadata _String_, A JSON string that allows you to add custom metadata that will be associated with the API call. For example, you may want to store the `context.Api.Name` or `context.Api.Version` with the API calls by reading from the [context variables](https://docs.microsoft.com/azure/api-management/api-management-policy-expressions#ContextVariables). The `metadata` field must be a JSON encoded string. ## Manual deployment The individual components can be deployed directly if needed. ### WebJob The WebJob is deployed as part of the overall deployment. To re-deploy the WebJob: 1. Download the [run.bat](https://raw.githubusercontent.com/Moesif/ApimEventProcessor/v1/azure-app-service-webjobs/run.bat) script to your computer. 2. Within the Azure Portal, go to your WebApp and select the WebJobs panel. -- If there is an existing WebJob, stop it and remove it. 3. Click the _+Add_ button to create a new job. Give it a name, set type to _continuous_ and upload the `run.bat` you previously downloaded. Once created, the script will clone the [ApimEventProcessor repo 'v1' branch](https://github.com/Moesif/Apimeventprocessor/tree/v1), run `dotnet build`, and starts the worker. ### APIM Logger If the name of an existing Azure API Management is not specified during deployment, you will need to add the `log-to-eventhub` logger to your Azure API Management service manually. To do so, utilize the [`nested/microsoft.apimanagement/service/loggers.bicep` ARM template](nested/microsoft.apimanagement/service/loggers.bicep) or view [Microsoft docs](https://docs.microsoft.com/azure/api-management/api-management-howto-log-event-hubs) More info on configuring Moesif is available on [Microsoft's documentation](https://docs.microsoft.com/azure/api-management/api-management-log-to-eventhub-sample). ## Updating the integration To run the latest version of webjob, simply stop, and then restart the WebJob (Within the Azure Portal, go to your WebApp and select the WebJobs panel). ## Steps performed by the Azure Resource Template This template performs the following tasks - Create Azure Eventhub and policies for Send and Listen. - If the name of an existing Azure API Management instance is provided, the template creates a new [log-to-eventhub](https://docs.microsoft.com/azure/api-management/api-management-howto-log-event-hubs) with the name `moesif-log-to-event-hub`. - Create an Azure Storage Account to periodically checkpoint the EventHub read location. - Create an Azure WebApp and configures the environment variables required by [ApimEventProcessor](https://github.com/Moesif/Apimeventprocessor/tree/v1). - Deploys [ApimEventProcessor](https://github.com/Moesif/Apimeventprocessor/tree/v1) as a Webjob on the WebApp. ## Troubleshooting - It is possible that the final step of deployment `app-service-webjob-msdeploy` reports a failed deployment with error such as `conflict` or `BadRequest` or `GatewayTimeout`. Despite these initial errors reported by deployment, it is possible that the WebJob deployment within Azure App Service may succeed automatically after 5-10 minutes without further user action. At the end of this template deployment, you may view detailed logs in your App Service/Activity log. - Ensure the `log-to-eventhub` logger is created - Ensure the `policy` is set on Api Management Apis - Ensure App Service configuration contains correct environment variables. View your App Service/Settings/Configuration/Application settings - Review the logs of App Service Webjob named `azure-api-mgmt-logs-2-moesif` and ensure it is running. View your App Service/Settings/WebJobs `Tags: Azure API Management, API Management, API Monetization, EventHub, Event Hub, API Gateway, Monitoring, Analytics, Observability, Logs, Logging, API Monitoring, API Analytics, API Logs, API Logging, Moesif, Kong, Tyk, Envoy, Gloo, Solo, Billing, Usage-based, usage-based, billing, WebApp, WebJob, App, Microsoft.Resources/deployments, Microsoft.ApiManagement/service/apis/policies, Microsoft.ApiManagement/service/loggers, Microsoft.EventHub/namespaces, Microsoft.EventHub/namespaces/eventhubs, Microsoft.EventHub/namespaces/eventhubs/authorizationRules, Microsoft.EventHub/namespaces/authorizationRules, Microsoft.Storage/storageAccounts, Microsoft.Web/sites/extensions, Microsoft.Web/serverfarms, Microsoft.Web/sites, SystemAssigned, Microsoft.Web/sites/config, Microsoft.Web/sites/hostNameBindings`