- 14 minutes to read

Blob Storage Policy

Info
This guide teaches applying a Nodinite-specific policy to enable logging from the Azure API Management platform to Nodinite.

graph LR subgraph "Payload (Any size)" AA[Azure API Management
with Policy 2] --> |1. Log Event,
Any size | roBS[Blob Storage] end subgraph "Nodinite" roBS --> C{Pickup Service} C --> F[Log API] end

When the Blob Storage Policy is in place and active, the code running in the Azure API Management platform creates a Nodinite-specific JSON Log Event and sends it to the Blob Storage Container.

Tip
You can use the Nodinite Azure Monitoring Agent to get alerts if you stockpile Log Events. The Nodinite Pickup Log Events Service Logging Agent should consume the Log Events. You can have a backout container for malformed messages, which should also be included in the Monitoring.

The Nodinite Pickup Log Events Service Logging Agent consumes the Nodinite Log Events from the Blob Storage Container. This pattern is Asynchronous.

Tip
You can use the Nodinite Non-Events Monitoring Agent to get alerts if you have too many or too few events within a period.

Configuration Steps

Follow the steps below to enable Logging to Nodinite in your Azure API Management Service APIs:

Next, you must configure the Nodinite Pickup Log Events Service Logging Agent.

Consume Log Events from the Blob Storage Container.
Malformed Log Events should be sent to a backout container.

Next, configure the Nodinite Azure Monitoring Agent to ensure you are not stockpiling Log Events.

Destination container
Backout container

Next, create and use Nodinite Log Views with Search Fields to share insights and access to the logged data with your business and key stakeholders.

Next, you can configure the Nodinite Non-Events Monitoring Agent to validate you get the expected number of Log Events during a period.

Managed Identity

In the Blob Storage Policy, you are using a user-assigned Managed Identity.

Create (or reuse) a Managed Identity in the Azure Portal.
Note the Client ID; you need it in the Named Values section.
Add the Storage Blob Contributor role to the Managed Identity. Apply on the Storage Account pointed to by the logging-blob-url named value.
Add the Managed Identity to the API Management Service where you intend to apply and use the Blob Storage Policy.

Named Values

The Blob Storage Policy uses the variables to avoid hard-coding values. You can expand the sample with additional properties as your business demands and policy require.

In the Azure Portal, navigate to the API Management Service and click Named Values in the sidebar.

Create the following, and set the values:

logging-blob-mid-client-id - Enter the Client ID of the Managed Identity from the previous step.
logging-blob-url - Enter the URL for the Blob Storage Container. Ensure the value ends with a trailing slash (/). You can omit the trailing slash if you also change the code in the sample.

Policy Fragments

The Blob Storage Policy has three Policy Fragments.

Inbound Policy Fragment - Includes the code to create a Nodinite JSON Log Event for Inbound Processing.
Outbound Policy Fragment- Includes the code to create a Nodinite JSON Log Event for Outbound Processing.
On Error Policy Fragment - Includes the code to create a Nodinite JSON Log Event when there is an error in Inbound and Outbound Processing.

Tip
All events in the sample make use of the x-ms-client-tracking-id header value. This ID facilitates end-to-end logging and is vital to all Nodinite Log Events. You can easily group all Log Events for an interchange in a Nodinite Log View by the built-in Search Field Application interchange id. If you call an Azure Function using the Nodinite Serilog sinks, add this header to the context to retain the value through the whole chain of events. If you invoke a Logic App, you do not need to do anything, the system has this covered for you.

Inbound Policy Fragment

In the Azure Portal, navigate to the API Management Service, then, in the sidebar, click Policy Fragments.
Create a new Policy Fragment and name it, for example, InboundLoggingPolicyFragment.

Property	Value	Comment
LogAgentValueId	3	Once set, you should NOT change this value. Ensure it is not already in use before you start to log.
EndPointDirection	10	Two-way Receive
EndPointTypeId	71	Microsoft Azure API Management
EventDirection	21	ExternalIncomingRequest

The fields are all part of a Nodinite Log Event.

<fragment>
	<!-- Start Logging -->
	<set-header name="x-ms-client-tracking-id" exists-action="skip">
		<value>@(Guid.NewGuid().ToString())</value>
	</set-header>
	<set-variable name="correlationId" value="@{
            return context.Request.Headers.GetValueOrDefault("x-ms-client-tracking-id");
        }" />
	<send-request mode="new" timeout="20" response-variable-name="blobdata" ignore-error="true">
		<set-url>@($"{{logging-blob-url}}{Guid.NewGuid().ToString()}.json")</set-url>
		<set-method>PUT</set-method>
		<set-header name="x-ms-version" exists-action="override">
			<value>2019-07-07</value>
		</set-header>
		<set-header name="x-ms-blob-type" exists-action="override">
			<value>BlockBlob</value>
		</set-header>
		<set-body>@{
            var body = context.Request?.Body?.As<string>(preserveContent: true) ?? "";
            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers.Where(h => !h.Key.Equals("authorization", StringComparison.InvariantCultureIgnoreCase) && !h.Key.Equals("ocp-apim-subscription-key", StringComparison.InvariantCultureIgnoreCase));
            var correlationId = context.Variables.GetValueOrDefault<string>("correlationId");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", correlationId);
        
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 3,
                EndPointName = context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection = 21,
                OriginalMessageTypeName = context.Variables.GetValueOrDefault<string>("loggingSchemaRoot", "UndeclaredSchemaRoot") + "#" + context.Variables.GetValueOrDefault<string>("loggingSchemaName", "UndeclaredSchemaName"),
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = context.Variables.GetValueOrDefault<string>("logtext", ""),
                Body = bodyToLog
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
		<authentication-managed-identity client-id="{{logging-blob-mid-client-id}}" resource="https://storage.azure.com" />
	</send-request>
	<!-- End Logging -->
</fragment>

Outbound Policy Fragment

Property	Value	Comment
LogAgentValueId	3	Once set, you should NOT change this value. Ensure it is not already in use before you start to log.
EndPointDirection	10	Two-way Receive
EndPointTypeId	71	Microsoft Azure API Management
EventDirection	25	ExternalIncomingResponse

<fragment>
	<!-- Start Logging -->
	<set-header name="x-ms-client-tracking-id" exists-action="skip">
		<value>@(Guid.NewGuid().ToString())</value>
	</set-header>
	<set-variable name="correlationId" value="@{
            return context.Request.Headers.GetValueOrDefault("x-ms-client-tracking-id");
        }" />
	<send-request mode="new" timeout="20" response-variable-name="blobdata" ignore-error="true">
		<set-url>@($"{{logging-blob-url}}{Guid.NewGuid().ToString()}.json")</set-url>
		<set-method>PUT</set-method>
		<set-header name="x-ms-version" exists-action="override">
			<value>2019-07-07</value>
		</set-header>
		<set-header name="x-ms-blob-type" exists-action="override">
			<value>BlockBlob</value>
		</set-header>
		<set-body>@{
            var body = context.Request?.Body?.As<string>(preserveContent: true) ?? "";
            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers.Where(h => !h.Key.Equals("authorization", StringComparison.InvariantCultureIgnoreCase) && !h.Key.Equals("ocp-apim-subscription-key", StringComparison.InvariantCultureIgnoreCase));
            var correlationId = context.Variables.GetValueOrDefault<string>("correlationId");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", correlationId);
        
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 3,
                EndPointName = context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection = 25,
                OriginalMessageTypeName = context.Variables.GetValueOrDefault<string>("loggingSchemaRoot", "UndeclaredSchemaRoot") + "#" + context.Variables.GetValueOrDefault<string>("loggingSchemaName", "UndeclaredSchemaName"),
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = context.Variables.GetValueOrDefault<string>("logtext", ""),
                Body = bodyToLog
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
		<authentication-managed-identity client-id="{{logging-blob-mid-client-id}}" resource="https://storage.azure.com" />
	</send-request>
	<!-- End Logging -->
</fragment>

On-Error Policy Fragment

Note
The LogStatus is set to -1 to indicate an Error condition.

<fragment>
	<!-- Start Logging -->
	<set-header name="x-ms-client-tracking-id" exists-action="skip">
		<value>@(Guid.NewGuid().ToString())</value>
	</set-header>
	<set-variable name="correlationId" value="@{
            return context.Request.Headers.GetValueOrDefault("x-ms-client-tracking-id");
        }" />
	<send-request mode="new" timeout="20" response-variable-name="blobdata" ignore-error="true">
		<set-url>@($"{{logging-blob-url}}{Guid.NewGuid().ToString()}.json")</set-url>
		<set-method>PUT</set-method>
		<set-header name="x-ms-version" exists-action="override">
			<value>2019-07-07</value>
		</set-header>
		<set-header name="x-ms-blob-type" exists-action="override">
			<value>BlockBlob</value>
		</set-header>
		<set-body>@{
            var body = context.Request?.Body?.As<string>(preserveContent: true) ?? "";
            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers.Where(h => !h.Key.Equals("authorization", StringComparison.InvariantCultureIgnoreCase) && !h.Key.Equals("ocp-apim-subscription-key", StringComparison.InvariantCultureIgnoreCase));
            var correlationId = context.Variables.GetValueOrDefault<string>("correlationId");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", correlationId);
        
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 3,
                EndPointName = context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection = 25,
                OriginalMessageTypeName = context.Variables.GetValueOrDefault<string>("loggingSchemaRoot", "UndeclaredSchemaRoot") + "#" + context.Variables.GetValueOrDefault<string>("loggingSchemaName", "UndeclaredSchemaName"),
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = context.Variables.GetValueOrDefault<string>("logtext", ""),
                Body = bodyToLog,
                LogStatus = -1
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
		<authentication-managed-identity client-id="{{logging-blob-mid-client-id}}" resource="https://storage.azure.com" />
	</send-request>
	<!-- End Logging -->
</fragment>

Add a reference to Policy Fragments in All APIs

In the Azure Portal, navigate to the API Management Service, then, in the sidebar, click APIs, then All APIs.

Inbound Processing

Use the Inbound Policy Fragment to create Nodinite Log Events for the Request.

Inbound Policy Fragment (Inbound Processing)

Click on the Add policy button.
Add a reference to the Inbound Policy Fragment.

 <inbound>
    <include-fragment fragment-id="InboundLoggingPolicyFragment" />
</inbound>

Outbound Processing

Use the Outbound Policy Fragment to create Nodinite Log Events for the Response.

Outbound Policy Fragment (Outbound Processing)

Click on the Add policy button.
Add a reference to the Outbound Policy Fragment.

<outbound>
    <include-fragment fragment-id="outboundLoggingPolicyFragment" />
</outbound>

On Error

Use the On-Error Policy Fragment to create Nodinite Log Events when there is an error.

<on-error>
    <include-fragment fragment-id="onerrorLoggingPolicyFragment" />
</on-error>

Complete example with all Policy Fragments

<policies>
    <inbound>
        <include-fragment fragment-id="InboundLoggingPolicyFragment" />
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <include-fragment fragment-id="outboundLoggingPolicyFragment" />
    </outbound>
    <on-error>
        <include-fragment fragment-id="onerrorLoggingPolicyFragment" />
    </on-error>
</policies>

Variables

In the final steps with the Blob Storage Policy, you need to set a few values. These are commonly used to attribute the Nodinite JSON Log Event. You can develop additional dynamics based on the provided examples.

The following variables are part of the default Nodinite Blob Storage Policy:

Variable	Default Value	Example	Comment
`loggingSchemaRoot`	`UndeclaredSchemaRoot`	`Nodinite.Logging.Schemas`	The prefix part of the Message Type name
`loggingSchemaName`	`UndeclaredSchemaName`	`Orders`	The suffix part of the Message Type name. Should be set on individual operation level
`logtext`	``	`Hello World`	Empty by default; Can be set on individual operation level

Info
Using the example values, the name of the Message Type in the resulting Nodinite JSON Log Event is Nodinite.Logging.Schemas#Orders. If possible, please use a versioning scheme in real-world scenarios, i.e. Nodinite.Logging.Schemas/1.0#Orders

In the On-Error, the LogStatus is set to -1 (indicating an Error), this is a candidate you may be interested in setting as a variable. Feel free to modify the default Policy Fragments to fit your use case.

Log Status Codes

Set variable on All operations

In the Azure Portal, navigate to the API Management Service. In the sidebar, click APIs, and then select one of the APIs, then All operations.

Complete example

Below is an example of setting the value for loggingSchemaRoot for all operations (inbound, outbound, and on-error). You can override the value if you want to on the operation level.

<policies>
    <!-- Throttle, authorize, validate, cache, or transform the requests -->
    <inbound>
        <set-variable name="loggingSchemaRoot" value="Nodinite.Logging.Schemas" />
        <base />
    </inbound>
    <!-- Control if and how the requests are forwarded to services  -->
    <backend>
        <base />
    </backend>
    <!-- Customize the responses -->
    <outbound>
        <set-variable name="loggingSchemaRoot" value="Nodinite.Logging.Schemas" />
        <base />
    </outbound>
    <!-- Handle exceptions and customize error responses  -->
    <on-error>
        <set-variable name="loggingSchemaRoot" value="Nodinite.Logging.Schemas" />
        <base />
    </on-error>
</policies>

Set variables on the API level (inbound)

For a specific operation, for example, a POST, in the inbound processing step, you can set the suffix part of the Message Type name as described in the Variables heading. Also, you can set a Log Text (a description field for the Nodinite Log Event)

 <inbound>
    <set-variable name="loggingSchemaName" value="Orders" />
    <set-variable name="logtext" value="Post OK" />
    <base />
    <set-backend-service id="apim-generated-policy" backend-id="nodinitefunctionapps" />
</inbound>

Set variables on the API level (outbound)

In the outbound processing step, we now make changes to the variables. You should set these values according to your business case and match the data in orbit.

 <outbound>
    <set-variable name="loggingSchemaName" value="OrderResponse" />
    <set-variable name="logtext" value="Response OK" />
    <base />
</outbound>

Set variables on the API level (on-error)

In the on-error processing step, we again make changes to the variables. You should set these values according to your business case and match the data in orbit.

 <on-error>
    <set-variable name="loggingSchemaName" value="Error" />
    <set-variable name="logtext" value="Error in processing" />
    <base />
</on-error>

Next Step

Search Fields
Log Views

Interested in logging from other Azure Related Services?

Explore Nodinite for 30 days – completely free

See how Nodinite resolves your technical issues, in a

LIVE Demo!