- 17 minutes to read

Configuration of Nodinite Logging Agent - Pickup Service

This guide teaches how to configure the Nodinite Pickup LogEvents Service Logging Agent so it can pick up JSON Log Events from any of the supported sources to achieve a true end-to-end Logging.

Configuration file overview

Below is a complete list of all fields/properties and their purpose

The configuration file (Settings.json), is in JSON format.

The default path is:

C:\Program Files\Nodinite\Logging Agent - Pickup Service\Settings.json

You must restart the Pickup LogEvents Service Logging Agent for any changes to take effect.

General

The general section holds some global information that affects mostly the internal behaviour of the Pickup LogEvents Service Logging Agent.

Property Description Value Example Comment
Environment The name for the type of environment this agent is running in Test, Prod, QA
Debug A flag that determines the amount of diagnostics information to write true or false (default) In a production environment, this value is ordinarily false
Version The internal version number for the configuration file 1.2.3.4 READ ONLY, this is internal information do NOT change this value manually!!!
CultureInfo format the output of date and so on SV-SE, EN-us Currently not implemented/honoured

Folders

The Folders section is where you manage file folders. These can either be local folders or SMB enabled shares.

Folders is an array of a Folder object. Each Folder points to one root (a file folder or SMB share) where the Log Agent should look for log files with JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
Folder The Folder where to look for log files for a network share, this means \\server\share will be \\\\server\\share in the JSON configuration file Use \\ to escape \ in the JSON notation.
BackoutFolder The Folder where to put invalid log files for a network share, this means \\server\theothershare will be \\\\server\\theothershare in the JSON configuration file Use \\ to escape \ in the JSON notation. CAUTION This should NOT be a child folder if the IncludeChildFolders option is enabled, unless, you have a proper exclude configuration for these
RemoveEmptyFiles When true, then empty files will be removed by the system true, false Empty files should not have been here anyway...
Filter A file filter to only look for file matching pattern. *.log or *.json, ... Can contain a combination of valid literal path and wildcard (* and ?)
IncludeChildFolders A flag used to search for JSON Log Event in subfolders true or false
ExcludeChildFolders An array of child folders to exclude if the IncludeChildFolders property is set to true Note: The full path must be provided, not only the name of the subfolder
ReplaceRules Review the shared Replace Rules section for additional information

ActiveMQs

The ActiveMQs section is where you manage Apache ActiveMQ related sources.

ActiveMQs is an array of an ActiveMQ queues. Each configuration is valid for one ActiveMQ queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ActiveConsumers Number of consumers that the agent will use to collect messages 1-n We recommend a positive integer between 1 and 4
ConnectionString The connection string used to the ActiveMQ server, can also be a fail-over address tcp://127.0.0.1:61616 or failover:(tcp://127.0.0.1:61616,tcp://host2:61616)?randomize=false&timeout=1000&startupMaxReconnectAttempts=2&|maxReconnectDelay=2000
Username The user name used to connect with ActiveMQ System
Password The password for the user Manager
Queue The name of the ActiveMQ Queue from where to pickup JSON Log Event from
DeadLetterQueue The name of the Dead letter queue where the pickup service puts invalid events and/or events that fails to log
ReplaceRules Review the shared Replace Rules section for additional information

MSMQs

The MSMQs section is where you manage Microsoft MSMQ related sources.

MSMQs is an array of MSMQ queues. Each configuration is valid for one MSMQ queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ActiveConsumers Number of consumers that the agent will use to collect messages 1-n Use any positive integer, between 1 and 4 is recommended
Server The name of the Windows server with MSMQ
Queue The name of the MSMQ Queue from where to pickup JSON Log Event from Note: Only the name of the queue
DeadLetterQueue The name of the specific Dead letter queue where the pickup service will put invalid events and/or events that can't be logged
ReplaceRules Review the shared Replace Rules section for additional information

EventHubs

The EventHubs section is where you manage Azure EventHub related sources. The JSON Log Events on the event hub may come from logging using a Policy from your API management entry points, please review the 'Logging the Request/Response from the Azure API Management platform' user guide.

  • An Azure Event Hub (target for Logging).
    • You can use one or more Event Hub entities as the target.
  • A storage container to keep track of the current checkpoint.
    • Each Event Hub Entity must have its unique container to store the bookmark (checkpoint).
Note: You must have one unique storage container per unique Event Hub entity! Otherwise, you will overwrite the checkpoint, preventing the Logging feature from being operational.
graph TD subgraph "Azure Cloud / Subscriptions" storage(fa:fa-boxes Storage Container) A[fal:fa-clouds Logic App
with diagnostic setting] --> |Diagnostics data| B(fa:fa-list Azure Event Hub) end subgraph "Nodinite Instance" B --- C[fal:fa-monitor-waveform Azure Logic Apps
Logging and Monitoring Agent] C--- db(fa:fa-database Agent Database) C --> |Log Event| F[fa:fa-cloud-download Nodinite's Log API] C --- |Checkpoint| storage end

EventHubs is an array of a EventHub object. Each EventHub object points to one EventHub where the Log Agent should look for log rows with JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false You don't have to remove the whole EventHub object to temporarily disable logging
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
EventHubConnectionString Connection string for EventHub Endpoint=sb://replaceme.servicebus.windows.net/;SharedAccessKeyName=[KeyName];SharedAccessKey=[Key];EntityPath=[EntityPathName]
EnableAmqpLinkRedirect A flag used to control the EnableAmqpLinkRedirect property true (default) - currently not honoured. Default is True Review more information here
EventHubName Name of the EventHub If EntityPath=[EntityPathName] is not provided in the connection string then you must provide the name using this entry
StorageAccountName Storage Acccount Name in use to store the checkpoint (syncpoint) Please review the Storage Account Name section below
StorageAccountKey The access key to the storage account Please review the Storage Account Keys section below
WebProxy Proxy 1.3.3.7:1337 Sets the EventProcessorOptions.WebProxy Property. When in use, also set the TransportType = EventHubsTransportType.AmqpWebSockets
StorageContainerName Name of the storage Checkpoints are being used and these are written to the storage
ReplaceRules Review the shared Replace Rules section for additional information

Note about EnableAmqpLinkRedirect

Clients that use AMQP connections over TCP require ports 5671 and 5672 to be opened in the local firewall. Along with these ports, it might be necessary to open additional ports if the EnableLinkRedirect feature is enabled. EnableLinkRedirect is a new messaging feature that helps skip one-hop while receiving messages, thus helping to boost throughput. The client would start communicating directly with the back-end service over port range 104XX.

A .NET client would fail with a SocketException ("An attempt was made to access a socket in a way forbidden by its access permissions") if these ports are blocked by the firewall. The feature can be disabled by setting EnableAmqpLinkRedirect=false in the connectiong string, which forces the clients to communicate with the remote service over port 5671.

Storage Account Name

The StorageAccountName can be found as pictured below in the Azure Portal:
Storage Account Name

Storage Account Keys

The StorageAccountKey can be found as pictured below in the Azure Portal:
Access Keys

NOTE: Use the value from a Key, Not the connection string.

ServiceBusQueues

The ServiceBusQueues section is where you manage Microsoft Azure Service Bus Queue related sources.

ServiceBusQueues is an array of a ServiceBus queue. Each configuration is valid for one Service Bus queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ConnectionString Service Bus Connection string Endpoint=sb://nodinite.com/servicebus.windows.net/;SharedAccessKeyName=yoursecretkey;SharedAccessKey=ib2+ofZaFyEhbwzgn1nAhw2XDY3UVc1JefdlGgLGIbb=" Copy from the Azure portal
Queue The name of the Service Bus Queue from where to pickup JSON Log Event from Note: Only the name of the queue
ReplaceRules Review the shared Replace Rules section for additional information

SQLServers

The SQLServers section is where you manage Microsoft SQL Server related sources.

SQLServers is an array of a SQLServer object. Each SQLServer object points to one Nodinite specific SQL Server table where the Log Agent should go look for log rows with the JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ConnectionString SQL Server Database Connection string Server=myServerAddress;Database=myDataBase;Integrated Security=SSPI;Connection Timeout=60
MaxRowsLimit The maximum number of rows to fetch on each loop 500 (default) Using larger values will impact the memory used
ReplaceRules Review the shared Replace Rules section for additional information

SQL Server table

If you are using a PostgreSQL database to store Log Events then your custom logging solution must have the following LogEvents table defined as:

Column Datatype Purpose
LogEvent json Your JSON encoded Log Event goes here
Id bigint Automatically created identity, the PickupService must know which row to flag as invalid if value in LogEvent column is not valid
ValidationFailed boolean Automatically set to false when inserted and might later be changed to true during processing if any errors are detected
ValidationFailedText text Column data is Updated if errors occur validating the LogEvent during processing
Created timestamp Automatically created during insert, might come in handy when troubleshooting logged but invalid log events

Invalid Log Events will get the ValidationFailed column set to True and you must rid these entries manually.

-- Table: [LogEvents], used by :Nodinite: Logging

-- DROP TABLE [dbo].[LogEvents];

CREATE TABLE [dbo].[LogEvents](
	[Id] [bigint] IDENTITY(1,1) NOT NULL,
	[LogEvent] [nvarchar](max) NOT NULL,
	[ValidationFailed] [bit] NOT NULL,
	[ValidationFailedText] [nvarchar](max) NULL,
	[Created] [datetimeoffset](7) NOT NULL,
 CONSTRAINT [PK_LogEvents] PRIMARY KEY NONCLUSTERED 
(
	[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 7) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

GO

ALTER TABLE [dbo].[LogEvents] ADD  CONSTRAINT [DF_LogEvents_ValidationFailed]  DEFAULT ((0)) FOR [ValidationFailed]
GO

ALTER TABLE [dbo].[LogEvents] ADD  CONSTRAINT [DF_LogEvents_Created]  DEFAULT (sysutcdatetime()) FOR [Created]
GO

-- Do not forget to assign user access rights

Remember to assign the service account for the Pickup Service the following rights:

  • db_datareader
  • db_datawriter

PostgreSQLs

The PostgreSQLs section is where you manage PostgreSQL related sources.

PostgreSQLs is an array of a PostgreSQL object. Each PostgreSQL object points to one Nodinite specific postgreSQL table where the Log Agent should go look for log rows with JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ConnectionString PostgreSQL Connection string Server=replaceme.databases.appdomain.cloud;Port=31902;Database=ibmclouddb;User Id=ibm_cloud_id;Password=yoursecretpassword;SSLMode=Prefer;Trust Server Certificate=true
MaxRowsLimit The maximum number of rows to fetch on each loop 500 (default) Using larger values will impact the memory used
ReplaceRules Review the shared Replace Rules section for additional information

PostgreSQL table

If you are using a PostgreSQL database to store Log Events then your custom logging solution must have the following LogEvents table defined as:

Column Datatype Purpose
LogEvent json Your JSON encoded Log Event goes here
Id bigint Automatically created identity, the PickupService must know which row to flag as invalid if value in LogEvent column is not valid
ValidationFailed boolean Automatically set to false when inserted and might later be changed to true during processing if any errors are detected
ValidationFailedText text Column data is Updated if errors occur validating the LogEvent during processing
Created timestamp Automatically created during insert, might come in handy when troubleshooting logged but invalid log events

Invalid Log Events will get the ValidationFailed column set to True and you must rid these entries manually.

-- Table: public."LogEvents"

-- DROP TABLE public."LogEvents";

CREATE TABLE public."LogEvents"
(
    "LogEvent" json NOT NULL,
    "Id" bigint NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 9223372036854775807 CACHE 1 ),
    "ValidationFailed" boolean NOT NULL DEFAULT false,
    "ValidationFailedText" text NULL,    
    "Created" timestamp with time zone NOT NULL DEFAULT now(),
    CONSTRAINT "LogEvents_pkey" PRIMARY KEY ("Id")
)
WITH (
    OIDS = FALSE
)
TABLESPACE pg_default;

ALTER TABLE public."LogEvents"
    OWNER to admin;

GRANT ALL ON TABLE public."LogEvents" TO usergoeshere;

-- Repeat the grants as needed

Make sure to change the value for the 'usergoeshere'

AnypointMQs

The AnypointMQs section is where you manage MuleSoft Cloudhub AnypointMQ related sources.

AnypointMQs is an array of an AnypointMQ object. Each AnypointMQ object points to one AnypointMQ queue where the Log Agent should look for utf-8 encoded JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ActiveConsumers Number of threads to run 1 Currently limited to 1 thread
Username The name of the user to logon with nodinite-reader
Password Password for user 1337
Environment The name of the environment Test
EnvironmentId Identifier for environment f7a16b7b-1337-4cb5-93ba-1e5289f707ec Guid
OrganizationId Identifier for organization 65ec82b6-f930-4ea2-1337-7c56ee24c9d3 Guid
RegionId Name of region North America Northern Virginia (us-east-1), Oregon (us-west-2), Central Canada (ca-central-1),
Europe Ireland (eu-west-1), London (eu-west-2)
APAC Singapore (ap-southeast-1), Sydney (ap-southeast-2)
According to your run-time, for example eu-west-1
UseConnectedApp Flag to use Connected App, or not true or false True when ClientId and ClientSecret is set, otherwise use Username/Password New 6.0.2.0
IsEuPanel True when using EU control plane true or false New 6.0.3.0
ClientId ClientId when using Connected App option 9186414a9dc4dfc94c4ccf670d25e4e New 6.0.2.0
ClientSecret ClientSecret when using Connected App option replaceme New 6.0.2.0
Queue Name of the queue with JSON Log Events Nodinite-LogEvents
DeadLetterQueue Name of queue for invalid Log Events Nodinite-LogEvents.DLQ
ReplaceRules Review the shared Replace Rules section for additional information

Replace Rules

This is a shared section for many of the configuration options above ReplaceRules is an array, of Rules to update provided Endpoints with a new Name or Uri if they are dynamic with for example id:s

New 6.0.1.0 Features named groups and RegEx without a group which supports additional advanced use cases.

Property Description Value Example
Name The name of this entry to document purpose
ReplaceName A flag to indicate whether to fix the Endpoint Name, or not true (default) or false
ReplaceUri A flag to indicate whether to fix the Endpoint URI, or not true (default) or false
Pattern A RegEx pattern to match the value to replace. Note: Only the value in the first group is replaced unless you use named groups /([0-9]{4,})$
Group1Value The new value to write instead of the match. Do not enter a value here if you are using named groups {customerId}

ReplaceRules Examples

Value Pattern Group1Value Result
https://www.nodinite.com/int042/getCustomer/12345 /([0-9]{4,})$ {customerId} https://www.nodinite.com/int042/getCustomer/{CustomerId}
INT(101) 101 100 INT100
INT101 New 6.0.1.0 INT101 INT100 INT100
https://www.nodinite.com/api/envelopes/H1ZUP5uDh/documents/rJxZ8P5OP2/files/CDRS1QQ1U7ul_o7n/url?asObject=true#GET https://www.nodinite.com/api/envelopes/(?<envId>\\w+)/documents/(?<docId>\\w+)/files/(?<fileId>\\w+) https://www.nodinite.com/api/envelopes/{envId}/documents/{docId}/files/{fileId}/url?asObject=true#GET

If you do not configure ReplaceRules before deploying your solution, you may end up with an awful lot of Endpoints, contact our support to get help tidying up.

ReplaceRules JSON Examples

Below is a set of different possible configs

"ReplaceRules": [
        {
          "Name": "Fix endpoint uri according to RegEx, supports named groups",
          "ReplaceName": false,
          "ReplaceUri": true,
          "Pattern": "https://wwww\\.nodinite\\.com/api/envelopes/(?<envId>\\w+)/documents/(?<docId>\\w+)/files/(?<fileId>\\w+)",
          "Group1Value": ""
        },
        {
          "Name": "Fix endpoint name according to RegEx, supports named groups",
          "ReplaceName": true,
          "ReplaceUri": false,
          "Pattern": "INT(101)",
          "Group1Value": "100"
        }
      ]

Example configuration file

In the Troubleshooting section, there is an example of a fully populated configuration file.

Next Step

JSON Log Event

Install Log Agent - Pickup Service
Log Agents
Logging Service