- 7 minutes to read

Release Notes

Monitoring Correlated Events

Technical guide to ACK/NAK transaction correlation monitoring—configure start/end patterns, correlation identifiers, timeout thresholds, and transaction state tracking across text-based log files.

What is Correlated Event Monitoring?

Correlated event monitoring tracks business transaction sequences by matching start patterns with corresponding end patterns using extracted correlation identifiers. When a start event is detected but no matching end event occurs within the configured timeout, the Log File Parser Monitoring Agent triggers a warning or error alert.

Correlation Monitoring Flow

flowchart TD Start([Start Event Detected<br/>in Log File A]) --> Extract[Extract Correlation ID<br/>e.g., OrderId: 12345] Extract --> Track[Register Transaction<br/>with Timestamp] Track --> Monitor{Monitor for<br/>End Event} Monitor -->|Found in Log File A or B| EndA[End Event Matched<br/>OrderId: 12345] Monitor -->|Warning Timeout| Warn[Warning Alert<br/>Transaction Incomplete] Monitor -->|Error Timeout| Error[Error Alert<br/>Transaction Failed] EndA --> Complete([✓ Transaction Complete<br/>State: OK]) Warn --> WaitMore{End Event<br/>Arrives?} WaitMore -->|Yes| Complete WaitMore -->|No, Error Timeout| Error Error --> Failed([✗ Alert Triggered<br/>State: Error]) style Start fill:#e3f2fd style Extract fill:#fff9c4 style Track fill:#fff9c4 style EndA fill:#c8e6c9 style Complete fill:#4caf50,color:#fff style Warn fill:#fff3e0 style Error fill:#ffcdd2 style Failed fill:#f44336,color:#fff

Key mechanism: Events can span multiple log files (incoming.log → outgoing.log), with correlation identifiers linking start and end events regardless of file location.

When to Use This Monitoring Type

Use correlated event monitoring for transaction sequence tracking scenarios:

  • Business workflows - Order processing, payment confirmations, approval workflows requiring completion verification
  • Integration patterns - API request/response pairs, message queue acknowledgments, data synchronization completion
  • Process monitoring - Batch job start/finish, file processing pipelines, multi-step operations
  • SLA compliance - Track transaction processing times against service level agreements

For independent event detection without correlation, use Monitoring Content Files instead.

Note

For benefits and use cases, see Log File Parser Monitoring Agent. This page focuses on technical correlation mechanics and configuration options.

Correlation Mechanics

Transaction Lifecycle Tracking

  1. Start event detection - Agent matches line against start pattern RegEx
  2. Identifier extraction - Capture groups extract correlation ID (e.g., OrderId, TransactionId)
  3. Transaction registration - Store transaction with identifier, timestamp, source file position
  4. End event matching - Subsequent lines tested against end pattern with same correlation ID
  5. Timeout evaluation - Transactions without end events checked against warning/error thresholds
  6. State determination - Resource state updated based on incomplete transaction count
  7. Alert generation - State changes trigger notifications via Monitor Views and Alarm Plugins

Pattern Configuration Syntax

Start Pattern:

OrderReceived.*OrderId:(\d+)
  • Matches: 2025-01-29 10:15:00 OrderReceived for OrderId:12345
  • Extracts: 12345 (captured via parentheses)

End Pattern:

OrderShipped.*OrderId:$1
  • $1 references first capture group from start pattern
  • Matches: 2025-01-29 10:45:00 OrderShipped for OrderId:12345
  • Correlation: Same OrderId (12345) links start and end events

Multi-file correlation:

  • Start pattern in incoming.log
  • End pattern in outgoing.log or same file
  • Agent correlates by identifier across file boundaries

Timeout Threshold Configuration

  • Warning threshold - Time elapsed before transaction generates Warning state (e.g., 15 minutes)
  • Error threshold - Time elapsed before transaction generates Error state (e.g., 30 minutes)
  • Time calculation - Measured from start event timestamp to current time
  • Dynamic adjustment - Thresholds modifiable via Edit Thresholds without agent restart

Example SLA configuration:

E-commerce order processing:
  Warning: 2 hours
  Error: 24 hours

Payment gateway transactions:
  Warning: 30 seconds
  Error: 5 minutes

File Selection and Correlation Scope

  • Single-file correlation - Start and end patterns both in same log file
  • Multi-file correlation - Start pattern in File A, end pattern in File B
  • Wildcard file patterns - payment_*.log handles daily rotation automatically
  • Network paths - UNC paths (\\server\share\logs), SMB shares, local folders
  • Time-based filtering - Control which files are included in correlation monitoring based on file timestamps:
    • Modified/Created after Clear Date Time - Monitor only files modified or created after specified timestamp
    • Modified/Created after File time span - Monitor files within relative time window (e.g., last 24 hours)
    • Last Modified/Last Created, one file only - Monitor only the most recently modified or created file
    • Evaluate all - Monitor all matching files regardless of timestamp

Time option use cases:

  • Active transaction logs only - Use "Modified after Clear Date Time" to ignore archived transaction files
  • Rolling correlation window - Use "Modified after File time span" for 24-hour transaction monitoring
  • Current transaction file - Use "Last Modified, one file only" when correlation spans single active log file

Correlated Events Overview
Example: Correlated event configurations tracking business transactions in a Monitor View

Monitor Configuration Options

Nodinite presents each Correlated Events Configuration as a Resource, evaluated continuously for transaction completion:

  • Organize by workflow - Group configurations by Application (Order Processing, Payment Gateway, Customer Service)
  • Categorize by type - All correlation configurations appear under 'Log File Event' Category
  • Scale monitoring - Deploy multiple agents across transaction processing servers
  • Apply different SLAs - Use varied timeout thresholds for different transaction types

Correlated Events Categories
Example: Correlated Events configurations organized by Category

Configuration Elements

  • Display name comes from your Correlated Events Configuration settings
  • Application association links transaction monitoring to business systems
  • Start/End patterns define what constitutes transaction begin and completion events
  • Timeout thresholds control when warnings and errors are triggered for incomplete transactions

Resource State Evaluation

Each Correlated Events Configuration (Resource) maintains one of these states based on incomplete transaction evaluation:

State Condition Description
Unavailable File access failed Agent cannot read files (network issue, permissions, path not found)
Error Timeout exceeded One or more transactions exceed error threshold without end event
Warning Approaching timeout One or more transactions exceed warning threshold but not error threshold
OK Transactions complete All start events have matching end events within configured timeouts

Transaction States
Example: Correlated events resource showing transaction monitoring state

Override default state evaluation using the Expected State feature (e.g., expect Warning state during batch processing windows).

Remote Actions for Incomplete Transactions

When incomplete transactions are detected or timeout thresholds require adjustment, perform these Remote Actions from the Web Client:

Remote Actions
Example: Remote actions for incomplete transactions

  • Edit Thresholds - Adjust warning/error timeout values without agent restart to match changing SLA requirements
  • Error Report - View incomplete transaction details, correlation IDs, timeout status, and file context
  • Clear Errors - Reset resource state after resolving stuck transactions or post-maintenance cleanup
  • Show Log Files - Display monitored transaction log files, download for offline analysis

All actions require appropriate Role permissions and are logged to the Audit Trail.

Tip

See Managing Correlated Events for complete action details, including timeout adjustment scenarios, transaction investigation workflows, and troubleshooting.

Correlation Pattern Examples

Order Processing Workflow 

Start Pattern: OrderReceived.*OrderId:(\d+)
End Pattern:   OrderShipped.*OrderId:$1
Warning: 2 hours
Error: 24 hours

Detects orders that aren't shipped within SLA.

Payment Transaction Tracking 

Start Pattern: PaymentInitiated.*TransactionId:([A-Z0-9]+)
End Pattern:   Payment(Confirmed|Failed).*TransactionId:$1
Warning: 30 seconds
Error: 5 minutes

Detects stuck payment transactions, matches both success and failure end events.

API Request/Response Monitoring 

Start Pattern: APIRequest.*RequestId:([a-f0-9-]+).*POST /api/orders
End Pattern:   APIResponse.*RequestId:$1.*(200|201)
Warning: 10 seconds
Error: 30 seconds

Ensures API requests receive successful responses within SLA.

Batch Job Processing 

Start Pattern: BatchJobStarted.*JobId:([A-Z0-9]+).*daily-reconciliation
End Pattern:   BatchJob(Completed|Failed).*JobId:$1
Warning: 3 hours
Error: 4 hours

Monitors overnight batch job completion.

Supported Correlation Scenarios

File Organization Patterns

  • Single file - Start and end events in same log file (common for application logs)
  • Multi-file - Start event in incoming.log, end event in outgoing.log (integration scenarios)
  • Wildcard rotation - payment_*.log handles daily/hourly log rotation
  • Mixed paths - Start event on local file, end event on SMB share (distributed systems)

Identifier Patterns

  • Numeric IDs - Order numbers, transaction IDs: (\d+)
  • Alphanumeric - Customer references, session IDs: ([A-Z0-9]+)
  • GUIDs - Transaction GUIDs: ([a-f0-9-]{36})
  • Composite keys - Multiple capture groups: Customer:(\w+).*Order:(\d+)

End Event Variations

  • Success only - OrderCompleted.*OrderId:$1
  • Success or failure - Order(Completed | Failed).*OrderId:$1
  • Multiple outcomes - (Success | Warning | Error).*TransactionId:$1

Next Steps

Configuration and setup:

Configuration - Correlated Events - Complete setup guide with pattern syntax and timeout configuration
Prerequisites - File permissions, network access, agent requirements
Install Agent - Agent installation and Core Services registration

Management and operations:

Managing Correlated Events - Remote actions for incomplete transactions (Edit, Error Report, Clear, Show Files)
Monitoring Content Files - Alternative monitoring type for independent event detection

Implementation examples:

How to Monitor Correlated Events - Step-by-step configuration walkthrough
How to Monitor Content Files - Step-by-step configuration walkthrough

Managing Content Files - Content management and investigation actions
Monitoring Content Files - Pattern-based content detection and alerting
Configuration - Correlated Events - Complete setup and pattern configuration
Troubleshooting Correlation Monitoring - Common issues and resolution guides