Extract mode

Use regular expressions to extract text from fields in incoming Events.

Features​ 

• Specify one or more Ruby compatible regular expressions to match and extract text from incoming Events.

• Any matched text will be included in an array in a new Event. If there are no matches, no Event will be emitted.

• Specify the name of the key for each array of matched text.

Configuration Options​ 

• mode: 'extract'

• matchers: When using 'extract' mode, define an array of regular expression configuration blocks to match text in incoming Events:

• path: Specify the wrapped JSON path for the field containing text to extract.

• regexp: Specify the regular expression to be used to extract text.

• to: Specify the name of the field to contain the array of matches.

Emitted Events​ 

{
  "email_addresses": ["alice@example.com"],
  "ips": ["10.1.1.12"],
  "urls": ["http://example.com", "https://tines.com"]
}

Example Configuration Options​ 

Given the incoming Event below, extract all email addresses and store them in a field called 'email_addresses' in a new Event.

{
  "text": "You received to email to alice@example.com sent from bob@example.com"
}

{
  "mode": "extract",
  "matchers": [
    {
      "path": "<<text>>",
      "regexp": "\\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\b",
      "to": "email_addresses"
    }
  ]
}

Given the incoming Event below, extract all email addresses from 'text' store them in a field called 'email_addresses'; extract all URLs from 'referrers' and store them in a field called 'urls'; and extract all IP addresses from 'servers' and store them in a field called 'ips'.

{
  "text": "You received to email to alice@example.com sent from bob@example.com",
  "metadata": {
    "servers": "10.1.1.1, 10.2.3.4, 10.15.6.8",
    "referrers": "https://tines.com and https://www.example.com"
  }
}

{
  "mode": "extract",
  "matchers": [
    {
      "path": "<<text>>",
      "regexp": "\\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\b",
      "to": "email_addresses"
    },
    {
      "path": "<<metadata.referrers>>",
      "regexp": "https?:\\/\\/[\\S]+",
      "to": "urls"
    },
    {
      "path": "<<metadata.servers>>",
      "regexp": "\\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\b",
      "to": "ip_addresses"
    }
  ],
  "message": "Email addresses taken from text, Urls from referrers and IPs from servers."
}