- 6 minutes to read

RegEx with Capturing Groups Search Field Expression Plugin

The RegEx with capturing groups plugin runs a regular expression on the message body of a Log Event and returns only the text inside the capturing group you choose, not the whole match. Use it to pick a value out of its surroundings, such as 054 from CityId="054" or the 4th field of a delimited flat file record.

  • ✅ Returns one, several, or all capturing groups of each match
  • ✅ Global and Multi line options for record-by-record flat files
  • ✅ Works on any text format in the message body

RegEx with capturing groups returns the mail domain from the ORD-10009 order confirmation, animated walkthrough (7.12)
Animated walkthrough: the pattern [\w.]+@([\w.-]+) with group 1 on a logged Order-to-Cash order confirmation returns northwind.example.com, the group, not the whole address orders@northwind.example.com.

When to use it

Four RegEx plugins cover the combinations of where the text is and what part of the match you want. For every plugin, see Choose a plugin.

Your data You want Use
Message body The whole match RegEx
Message body Only the part inside a capturing group ( ) RegEx with capturing groups (this page)
One context value (you know the key) The whole match RegEx on Message Context
One context value (you know the key) Only the part inside a capturing group RegEx on Message Context with capturing groups
One context value (you know the key) The value as it is, no pattern Message Context Key
A group you need to transform or combine Nested functions Formula with regexGroup

Expression syntax

Select RegEx with capturing groups in the Type list. The plugin shows these fields:

Type list filtered on regex, with the RegEx plugins (7.12)
The RegEx with capturing groups plugin and its siblings in the Type list.

Field What goes in it
Expression The regular expression, with at least one capturing group ( ). The help text under the field ends with "The pattern ignores case and white space: write a space as \s or [ ] and # as \# (an unescaped # starts a comment)."
RegEx groups The group or groups to return: a single number (1), a list (1,3,4,5,6), a range (1,3-6), or group names (value1,value2, with no space after the comma). Group 0 is the whole match, and named groups are numbered after the unnamed ones. Left empty, the plugin returns the whole match. The help text under the field: "The groups to return, comma-separated: group numbers (1,3,4 or a range 1,3-6; 0 is the whole match, named groups are numbered after the unnamed ones) and/or group names (value1,value2). Leave empty to return the whole match."
Global Checked: return every match. Unchecked: stop after the first match.
Multi line Checked: ^ and $ match the start and end of each line, not only of the whole body.
  • Groups are numbered from 1 in the order their opening parentheses appear. (?: ) is a non-capturing group and gets no number.
  • A repeated group keeps the value of its last repetition. ^(?:(.*?)\|){4} therefore returns the 4th pipe-delimited field of a line, counting from 1.
  • Matching ignores case: the pattern cityId matches CityId in Example 1.
  • Spaces in the pattern are ignored, and # starts a comment: a b matches ab, not a b. Write a literal space as \s or [ ], and a literal # as \#.
  • Each value is trimmed of leading and trailing white space, and an empty group is dropped.
  • A pattern may run for 30 seconds per event at most.
  • The Global checkbox above belongs to the plugin. The dialog has a second Global under Apply to, which applies the expression to all Message Types (see Add or manage Search Field).
  • The plugin reads the whole message body; for large XML or JSON payloads, prefer XPath or JSON Path.
  • The Test Expression tab has its own Global and Multi line checkboxes; tick them there before you press Test.

Expressions tab with Type RegEx with capturing groups and the help texts under Expression and RegEx groups (7.12)
The plugin's fields with their help texts.

Test Expression tab of RegEx with capturing groups with its own Global checkbox ticked (7.12)
The plugin's Global checkbox, ticked on the Test Expression tab.

Configure and test the expression as described in Add or manage Search Field and Test Your Expression. When nothing matches, Test Expression reports Processed with warning: The Expression yields no result:

Test Expression result: Processed with warning, The Expression yields no result (7.12)
Test result when the pattern matches nothing in the message body.

Examples

Example 1: Extract city IDs from an order message

Input

<ns0:Orders xmlns:ns0="Common.Schemas/Nodinite/1.0">
    <Order>
        <Id>101</Id>
        <Amount>1337</Amount>
        <City CityId="054">Karlstad</City>
    </Order>
    <Order>
        <Id>102</Id>
        <Amount>10</Amount>
        <City CityId="08">Stockholm</City>
    </Order>
</ns0:Orders>

Expression

cityId="([0-9]+)"
  • RegEx groups: 1
  • Global: checked
  • Multi line: unchecked

Result

054
08

The same pattern in the RegEx plugin returns CityId="054" and CityId="08".

Test Expression for RegEx with capturing groups returning 054 and 08 from an Orders XML message (7.12)
Test Expression: group 1 of each match, two unique values. The first City line is blurred in the screenshot; it is the same as in the input above.

Example 2: Extract the 4th field from each pipe-delimited order record

Input

ORD|101|ExampleCompany1|456|Company Name ACME |Dieselgate valley 1|123 45|Flameburg|
ORD|102|ExampleCompany2|789|Company Name MECA |Radiator Spring 420|543 21|Apocalypsenburg|

Expression

^(?:(.*?)\|){4}
  • RegEx groups: 1
  • Global: checked
  • Multi line: checked

Result

456
789

The group repeats four times per line and keeps the last value: ORD, 101, ExampleCompany1, then 456. Multi line makes ^ match at the start of each record, and Global returns both records.

Test Expression for RegEx with capturing groups returning 456 and 789 from two pipe-delimited records (7.12)
Test Expression: the 4th field of each flat file record.

Example 3: Extract a field from the line that starts with a tag

Input

ST|123
KV|Z|AN|1337|ABC123
ID|1912121212|

Expression

^KV(?:(.*?)\|){4}
  • RegEx groups: 1
  • Global: checked
  • Multi line: checked

Result

1337

After KV, the first repetition captures the empty text before the first |, then Z, AN, and 1337. Counting the tag KV as field 1, {4} returns the 4th field. Only one line starts with KV, so Global makes no difference here.

Test Expression for RegEx with capturing groups returning 1337 from the KV line of a flat file (7.12)
Test Expression: the 4th field of the line tagged KV.

Example 4: Extract the 3rd field from a semicolon-delimited header line

Input

HDR;ORDER;42
KV;Z;AN;1337;ABC123
ID;1912121212;

Expression

^HDR(?:(.*?)[;\r\n]){3}
  • RegEx groups: 1
  • Global: unchecked
  • Multi line: unchecked

Result

42

Each repetition ends at a ; or a line break, so the last field of the line is also captured. Counting the tag HDR as field 1, {3} returns the 3rd field: the empty text before the first ;, then ORDER, then 42. Without Multi line, ^ matches only at the start of the body, where the HDR line is.

Test Expression for RegEx with capturing groups returning 42 from the HDR line of a flat file (7.12)
Test Expression: the 3rd field of the HDR line, with Global and Multi line unchecked.

Next Steps