Deduplicate mode

Specify a field in an incoming Events and only emit the Event if that field contains a unique value.

Features​ 

• Define how many historical events from previous story runs should be used to determine if value in field is unique.

• Specify a period of time to check for duplicates.

• Specify the field in an incoming Event to use as a measure of uniqueness.

• If an Event is deemed unique, it will be emitted. If it is deemed a duplicate, it will be ignored and a corresponding log entry will be created.

Configuration Options​ 

• mode: 'deduplicate'

• path: a JSON wrapped path the value of which should be used to determine the uniqueness of the event.

• lookback: (Optional: must use either lookback or period) Number of past events (maximum of 1000) to examine for uniqueness.

• period: (Optional: must use either lookback or period) Number of seconds to use for deduplication. When an event is received, subsequent events will not be emitted until this period has elapsed.

• emit_duplicates: (Optional) Set to true if duplicate events should be emitted. 'unique_event': false will be included in emitted event.

• tolerance: (Optional) The number of events with the same value to allow before deduplicating. Events of the same value will be treated as unique until the provided tolerance is reached.

Emitted Events​ 

{
  "unique_event": true
}

Example Configuration Options​ 

Check the last 100 received Events and only emit an Event if the value in the person.name field is unique.

{
  "mode": "deduplicate",
  "lookback": "100",
  "path": "<<person.name>>"
}