Deduplicate mode

Specify a field in an incoming Events and only emit the Event if that field contains a unique value.

Features​ 

  • Define how many historical events from previous story runs should be used to determine if value in field is unique.

  • Specify a period of time to check for duplicates.

  • Specify the field in an incoming Event to use as a measure of uniqueness.

  • If an Event is deemed unique, it will be emitted. If it is deemed a duplicate, it will be ignored and a corresponding log entry will be created.

Configuration Options​ 

  • mode: 'deduplicate'

  • path: a JSON wrapped path the value of which should be used to determine the uniqueness of the event.

  • lookback: (Optional: must use either lookback or period) Number of past events (maximum of 1000) to examine for uniqueness.

  • period: (Optional: must use either lookback or period) Number of seconds to use for deduplication. When an event is received, subsequent events will not be emitted until this period has elapsed.

  • emit_duplicates: (Optional) Set to true if duplicate events should be emitted. 'unique_event': false will be included in emitted event.

  • tolerance: (Optional) The number of events with the same value to allow before deduplicating. Events of the same value will be treated as unique until the provided tolerance is reached.

Emitted Events​ 

{
  "unique_event": true
}

Example Configuration Options​ 

Check the last 100 received Events and only emit an Event if the value in the person.name field is unique. To see this on your storyboard, copy the JSON below into an event transform action by selecting View code at the top right of the action and pasting the JSON in.

{
  "mode": "deduplicate",
  "lookback": "100",
  "path": "<<person.name>>"
}
Was this helpful?