Release Notes
Configuration of Nodinite Logging Agent - Pickup Service
This guide teaches how to configure the Nodinite Pickup LogEvents Service Logging Agent so it can pick up JSON Log Events from any of the supported sources to achieve a true end-to-end Logging.
Configuration file overview
Below is a complete list of all fields/properties and their purpose
The configuration file (Settings.json
), is in JSON format.
The default path is:
C:\Program Files\Nodinite\Logging Agent - Pickup Service\Settings.json
You must restart the Pickup LogEvents Service Logging Agent for any changes to take effect.
- General
- ActiveMQs
- AnypointMQs
- EventHubs
- Service Bus Queues
- Folders
- MSMQs
- PostgreSQLs
- SQLServers
- Replace Rules
General
The general section holds some global information that affects mostly the internal behaviour of the Pickup LogEvents Service Logging Agent.
Property | Description | Value Example | Comment |
---|---|---|---|
Environment | The name for the type of environment this agent is running in | Test, Prod, QA | |
Debug | A flag that determines the amount of diagnostics information to write | true or false (default) | In a production environment, this value is ordinarily false |
Version | The internal version number for the configuration file | 1.2.3.4 | READ ONLY, this is internal information do NOT change this value manually!!! |
CultureInfo | format the output of date and so on | SV-SE, EN-us | Currently not implemented/honoured |
Folders
The Folders section is where you manage file folders. These can either be local folders or SMB enabled shares.
Folders is an array of a Folder object. Each Folder points to one root (a file folder or SMB share) where the Log Agent should look for log files with JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
Folder | The Folder where to look for log files | for a network share, this means \\server\share will be \\\\server\\share in the JSON configuration file |
Use \\ to escape \ in the JSON notation. |
BackoutFolder | The Folder where to put invalid log files | for a network share, this means \\server\theothershare will be \\\\server\\theothershare in the JSON configuration file |
Use \\ to escape \ in the JSON notation. CAUTION This should NOT be a child folder if the IncludeChildFolders option is enabled, unless, you have a proper exclude configuration for these |
RemoveEmptyFiles | When true , then empty files will be removed by the system |
true , false |
Empty files should not have been here anyway... |
Filter | A file filter to only look for file matching pattern. | *.log or *.json , ... |
Can contain a combination of valid literal path and wildcard (* and ?) |
IncludeChildFolders | A flag used to search for JSON Log Event in subfolders | true or false | |
ExcludeChildFolders | An array of child folders to exclude if the IncludeChildFolders property is set to true |
Note: The full path must be provided, not only the name of the subfolder | |
ReplaceRules | Review the shared Replace Rules section for additional information |
ActiveMQs
The ActiveMQs section is where you manage Apache ActiveMQ related sources.
ActiveMQs is an array of an ActiveMQ queues. Each configuration is valid for one ActiveMQ queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
ActiveConsumers | Number of consumers that the agent will use to collect messages | 1-n | We recommend a positive integer between 1 and 4 |
ConnectionString | The connection string used to the ActiveMQ server, can also be a fail-over address | tcp://127.0.0.1:61616 or failover:(tcp://127.0.0.1:61616,tcp://host2:61616)?randomize=false&timeout=1000&startupMaxReconnectAttempts=2&|maxReconnectDelay=2000 |
|
Username | The user name used to connect with ActiveMQ | System | |
Password | The password for the user | Manager | |
Queue | The name of the ActiveMQ Queue from where to pickup JSON Log Event from | ||
DeadLetterQueue | The name of the Dead letter queue where the pickup service puts invalid events and/or events that fails to log | ||
ReplaceRules | Review the shared Replace Rules section for additional information |
MSMQs
The MSMQs section is where you manage Microsoft MSMQ related sources.
MSMQs is an array of MSMQ queues. Each configuration is valid for one MSMQ queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
ActiveConsumers | Number of consumers that the agent will use to collect messages | 1-n | Use any positive integer, between 1 and 4 is recommended |
Server | The name of the Windows server with MSMQ | ||
Queue | The name of the MSMQ Queue from where to pickup JSON Log Event from | Note: Only the name of the queue | |
DeadLetterQueue | The name of the specific Dead letter queue where the pickup service will put invalid events and/or events that can't be logged | ||
ReplaceRules | Review the shared Replace Rules section for additional information |
EventHubs
The EventHubs section is where you manage Azure EventHub related sources. The JSON Log Events on the event hub may come from logging using a Policy from your API management entry points, please review the 'Logging the Request/Response from the Azure API Management platform' user guide.
- An Azure Event Hub (target for Logging).
- You can use one or more Event Hub entities as the target.
- A storage container to keep track of the current checkpoint.
- Each Event Hub Entity must have its unique container to store the bookmark (checkpoint).
with diagnostic setting] --> |Diagnostics data| B(fa:fa-list Azure Event Hub) end subgraph "Nodinite Instance" B --- C[fal:fa-monitor-waveform Azure Logic Apps
Logging and Monitoring Agent] C--- db(fa:fa-database Agent Database) C --> |Log Event| F[fa:fa-cloud-download Nodinite's Log API] C --- |Checkpoint| storage end
EventHubs is an array of a EventHub object. Each EventHub object points to one EventHub where the Log Agent should look for log rows with JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | You don't have to remove the whole EventHub object to temporarily disable logging |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
EventHubConnectionString | Connection string for EventHub | Endpoint=sb://replaceme.servicebus.windows.net/;SharedAccessKeyName=[KeyName];SharedAccessKey=[Key];EntityPath=[EntityPathName] |
|
EnableAmqpLinkRedirect | A flag used to control the EnableAmqpLinkRedirect property | true (default) - currently not honoured. Default is True | Review more information here |
EventHubName | Name of the EventHub | If EntityPath=[EntityPathName] is not provided in the connection string then you must provide the name using this entry |
|
StorageAccountName | Storage Acccount Name in use to store the checkpoint (syncpoint) | Please review the Storage Account Name section below | |
StorageAccountKey | The access key to the storage account | Please review the Storage Account Keys section below | |
WebProxy | Proxy | 1.3.3.7:1337 | Sets the EventProcessorOptions.WebProxy Property. When in use, also set the TransportType = EventHubsTransportType.AmqpWebSockets |
StorageContainerName | Name of the storage | Checkpoints are being used and these are written to the storage | |
ReplaceRules | Review the shared Replace Rules section for additional information |
Note about EnableAmqpLinkRedirect
Clients that use AMQP connections over TCP require ports 5671 and 5672 to be opened in the local firewall. Along with these ports, it might be necessary to open additional ports if the EnableLinkRedirect feature is enabled. EnableLinkRedirect is a new messaging feature that helps skip one-hop while receiving messages, thus helping to boost throughput. The client would start communicating directly with the back-end service over port range 104XX.
A .NET client would fail with a SocketException ("An attempt was made to access a socket in a way forbidden by its access permissions") if these ports are blocked by the firewall. The feature can be disabled by setting EnableAmqpLinkRedirect=false in the connectiong string, which forces the clients to communicate with the remote service over port 5671.
Storage Account Name
The StorageAccountName can be found as pictured below in the Azure Portal:
Storage Account Keys
The StorageAccountKey can be found as pictured below in the Azure Portal:
NOTE: Use the value from a Key, Not the connection string.
ServiceBusQueues
The ServiceBusQueues section is where you manage Microsoft Azure Service Bus Queue related sources.
ServiceBusQueues is an array of a ServiceBus queue. Each configuration is valid for one Service Bus queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
ConnectionString | Service Bus Connection string | Endpoint=sb://nodinite.com/servicebus.windows.net/;SharedAccessKeyName=yoursecretkey;SharedAccessKey=ib2+ofZaFyEhbwzgn1nAhw2XDY3UVc1JefdlGgLGIbb=" |
Copy from the Azure portal |
Queue | The name of the Service Bus Queue from where to pickup JSON Log Event from | Note: Only the name of the queue | |
ReplaceRules | Review the shared Replace Rules section for additional information |
SQLServers
The SQLServers section is where you manage Microsoft SQL Server related sources.
SQLServers is an array of a SQLServer object. Each SQLServer object points to one Nodinite specific SQL Server table where the Log Agent should go look for log rows with the JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
ConnectionString | SQL Server Database Connection string | Server=myServerAddress;Database=myDataBase;Integrated Security=SSPI;Connection Timeout=60 |
|
MaxRowsLimit | The maximum number of rows to fetch on each loop | 500 (default) | Using larger values will impact the memory used |
ReplaceRules | Review the shared Replace Rules section for additional information |
SQL Server table
If you are using a PostgreSQL database to store Log Events then your custom logging solution must have the following LogEvents table defined as:
Column | Datatype | Purpose |
---|---|---|
LogEvent | json | Your JSON encoded Log Event goes here |
Id | bigint | Automatically created identity, the PickupService must know which row to flag as invalid if value in LogEvent column is not valid |
ValidationFailed | boolean | Automatically set to false when inserted and might later be changed to true during processing if any errors are detected |
ValidationFailedText | text | Column data is Updated if errors occur validating the LogEvent during processing |
Created | timestamp | Automatically created during insert, might come in handy when troubleshooting logged but invalid log events |
Invalid Log Events will get the ValidationFailed column set to True and you must rid these entries manually.
-- Table: [LogEvents], used by :Nodinite: Logging
-- DROP TABLE [dbo].[LogEvents];
CREATE TABLE [dbo].[LogEvents](
[Id] [bigint] IDENTITY(1,1) NOT NULL,
[LogEvent] [nvarchar](max) NOT NULL,
[ValidationFailed] [bit] NOT NULL,
[ValidationFailedText] [nvarchar](max) NULL,
[Created] [datetimeoffset](7) NOT NULL,
CONSTRAINT [PK_LogEvents] PRIMARY KEY NONCLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 7) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[LogEvents] ADD CONSTRAINT [DF_LogEvents_ValidationFailed] DEFAULT ((0)) FOR [ValidationFailed]
GO
ALTER TABLE [dbo].[LogEvents] ADD CONSTRAINT [DF_LogEvents_Created] DEFAULT (sysutcdatetime()) FOR [Created]
GO
-- Do not forget to assign user access rights
Remember to assign the service account for the Pickup Service the following rights:
- db_datareader
- db_datawriter
PostgreSQLs
The PostgreSQLs section is where you manage PostgreSQL related sources.
PostgreSQLs is an array of a PostgreSQL object. Each PostgreSQL object points to one Nodinite specific postgreSQL table where the Log Agent should go look for log rows with JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
ConnectionString | PostgreSQL Connection string | Server=replaceme.databases.appdomain.cloud;Port=31902;Database=ibmclouddb;User Id=ibm_cloud_id;Password=yoursecretpassword;SSLMode=Prefer;Trust Server Certificate=true |
|
MaxRowsLimit | The maximum number of rows to fetch on each loop | 500 (default) | Using larger values will impact the memory used |
ReplaceRules | Review the shared Replace Rules section for additional information |
PostgreSQL table
If you are using a PostgreSQL database to store Log Events then your custom logging solution must have the following LogEvents table defined as:
Column | Datatype | Purpose |
---|---|---|
LogEvent | json | Your JSON encoded Log Event goes here |
Id | bigint | Automatically created identity, the PickupService must know which row to flag as invalid if value in LogEvent column is not valid |
ValidationFailed | boolean | Automatically set to false when inserted and might later be changed to true during processing if any errors are detected |
ValidationFailedText | text | Column data is Updated if errors occur validating the LogEvent during processing |
Created | timestamp | Automatically created during insert, might come in handy when troubleshooting logged but invalid log events |
Invalid Log Events will get the ValidationFailed column set to True and you must rid these entries manually.
-- Table: public."LogEvents"
-- DROP TABLE public."LogEvents";
CREATE TABLE public."LogEvents"
(
"LogEvent" json NOT NULL,
"Id" bigint NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 9223372036854775807 CACHE 1 ),
"ValidationFailed" boolean NOT NULL DEFAULT false,
"ValidationFailedText" text NULL,
"Created" timestamp with time zone NOT NULL DEFAULT now(),
CONSTRAINT "LogEvents_pkey" PRIMARY KEY ("Id")
)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;
ALTER TABLE public."LogEvents"
OWNER to admin;
GRANT ALL ON TABLE public."LogEvents" TO usergoeshere;
-- Repeat the grants as needed
Make sure to change the value for the '
usergoeshere
'
AnypointMQs
The AnypointMQs section is where you manage MuleSoft Cloudhub AnypointMQ related sources.
AnypointMQs is an array of an AnypointMQ object. Each AnypointMQ object points to one AnypointMQ queue where the Log Agent should look for utf-8 encoded JSON Log Events.
Property | Description | Value Example | Comment |
---|---|---|---|
Enabled | A flag used to disable or enable logging | true or false | This feature means that you don't have to remove the configuration to disable the pickup service temporarily |
LogAPI | The address to the Nodinite LogAPI | https://localhost/Nodinite/LogAPI/ | Change according to your setup |
UseConfigurationDatabase | A flag used to bypass the LogAPI | true or false | Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system |
ConfigurationDatabaseConnectionString | Connection string to Configuration database | Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60 |
|
ActiveConsumers | Number of threads to run | 1 | Currently limited to 1 thread |
Username | The name of the user to logon with | nodinite-reader | |
Password | Password for user | 1337 | |
Environment | The name of the environment | Test | |
EnvironmentId | Identifier for environment | f7a16b7b-1337-4cb5-93ba-1e5289f707ec | Guid |
OrganizationId | Identifier for organization | 65ec82b6-f930-4ea2-1337-7c56ee24c9d3 | Guid |
RegionId | Name of region | North America Northern Virginia (us-east-1), Oregon (us-west-2), Central Canada (ca-central-1), Europe Ireland (eu-west-1), London (eu-west-2) APAC Singapore (ap-southeast-1), Sydney (ap-southeast-2) |
According to your run-time, for example eu-west-1 |
UseConnectedApp | Flag to use Connected App, or not | true or false |
True when ClientId and ClientSecret is set, otherwise use Username/Password New 6.0.2.0 |
IsEuPanel | True when using EU control plane | true or false |
New 6.0.3.0 |
ClientId | ClientId when using Connected App option | 9186414a9dc4dfc94c4ccf670d25e4e | New 6.0.2.0 |
ClientSecret | ClientSecret when using Connected App option | replaceme | New 6.0.2.0 |
Queue | Name of the queue with JSON Log Events | Nodinite-LogEvents | |
DeadLetterQueue | Name of queue for invalid Log Events | Nodinite-LogEvents.DLQ | |
ReplaceRules | Review the shared Replace Rules section for additional information |
Replace Rules
This is a shared section for many of the configuration options above ReplaceRules is an array, of Rules to update provided Endpoints with a new Name or Uri if they are dynamic with for example id:s
New 6.0.1.0 Features named groups and RegEx without a group which supports additional advanced use cases.
Property | Description | Value Example |
---|---|---|
Name | The name of this entry to document purpose | |
ReplaceName | A flag to indicate whether to fix the Endpoint Name, or not | true (default) or false |
ReplaceUri | A flag to indicate whether to fix the Endpoint URI, or not | true (default) or false |
Pattern | A RegEx pattern to match the value to replace. Note: Only the value in the first group is replaced unless you use named groups | /([0-9]{4,})$ |
Group1Value | The new value to write instead of the match. Do not enter a value here if you are using named groups | {customerId} |
ReplaceRules Examples
Value | Pattern | Group1Value | Result |
---|---|---|---|
https://www.nodinite.com/int042/getCustomer/12345 |
/([0-9]{4,})$ |
{customerId} |
https://www.nodinite.com/int042/getCustomer/{CustomerId} |
INT(101) |
101 |
100 |
INT100 |
INT101 |
New 6.0.1.0 INT101 |
INT100 |
INT100 |
https://www.nodinite.com/api/envelopes/H1ZUP5uDh/documents/rJxZ8P5OP2/files/CDRS1QQ1U7ul_o7n/url?asObject=true#GET |
https://www.nodinite.com/api/envelopes/(?<envId>\\w+)/documents/(?<docId>\\w+)/files/(?<fileId>\\w+) |
https://www.nodinite.com/api/envelopes/{envId}/documents/{docId}/files/{fileId}/url?asObject=true#GET |
If you do not configure ReplaceRules before deploying your solution, you may end up with an awful lot of Endpoints, contact our support to get help tidying up.
ReplaceRules JSON Examples
Below is a set of different possible configs
"ReplaceRules": [
{
"Name": "Fix endpoint uri according to RegEx, supports named groups",
"ReplaceName": false,
"ReplaceUri": true,
"Pattern": "https://wwww\\.nodinite\\.com/api/envelopes/(?<envId>\\w+)/documents/(?<docId>\\w+)/files/(?<fileId>\\w+)",
"Group1Value": ""
},
{
"Name": "Fix endpoint name according to RegEx, supports named groups",
"ReplaceName": true,
"ReplaceUri": false,
"Pattern": "INT(101)",
"Group1Value": "100"
}
]
Example configuration file
In the Troubleshooting section, there is an example of a fully populated configuration file.
Next Step
Related
Install Log Agent - Pickup Service
Log Agents
Logging Service