Skip to main content

Drop Duplicates

Available in real-time analytics only.

The Drop Duplicates tool drops repeated features for the same track ID based on matching attribute values and geometry. Fields that change with each request, such as request timestamps, can be excluded from duplicate comparison. This tool is useful for preventing unchanged features from triggering repeated downstream processing or notifications.

Examples

The following are example use cases for the tool:

  • A feed polls a vehicle location API every 10 seconds, but the API updates each vehicle once a minute. The Drop Duplicates tool drops repeated features between updates, reducing the features sent to downstream tools and outputs.

  • A feed polls a weather API every minute, but station readings update every five minutes. The Drop Duplicates tool drops repeated readings for each station, reducing downstream processing. If the response includes a request timestamp that changes with every poll, you can ignore that field when comparing features for duplicates.

Usage notes

Note the following when working with the tool:

  • Identify a Track ID key field in the input dataset. You cannot include the Track ID key field in Fields to ignore.

  • The tool retains the most recent processed feature for each track ID, excluding features it rejects as out of order. The tool compares incoming features only with that retained feature, not with the track's complete history.

  • The first feature received with a valid track ID value passes downstream and becomes the retained feature for that track. Subsequent features replace it, including duplicates, unless the tool rejects them as out of order.

  • All attributes participate in duplicate comparison except fields selected in Fields to ignore.

  • Start time and feature ordering:

    • While the tool retains a feature for a track, it checks timestamp order if you configure a Start time key field and both features have valid start times.

    • If the incoming start time is later than or equal to the retained start time, the tool compares the attributes and geometry. The tool drops a matching feature and passes a different feature downstream.

    • If the incoming start time is earlier than the retained start time, the tool drops the feature without duplicate comparison. It does not replace the retained feature or restart the retention window.

    • The Start time key field participates in duplicate comparison unless you select it in Fields to ignore. Ignoring it does not disable timestamp ordering: the tool still drops features with an earlier start time, even if their other attributes or geometry differ.

    • A dropped duplicate can update the retained start time. For example, if the Start time key field is ignored, the tool drops a feature at 10:03 a.m. that otherwise matches the retained feature at 10:02 a.m., but that feature becomes the retained feature. A subsequent feature at 10:02 a.m. is then out of order.

    • If the input has no Start time key field, the tool compares features in the order it processes them, without checking timestamp order. Timestamp attributes still participate in duplicate comparison unless you ignore them.

  • Track retention:

    • The Track retention time window parameter determines how long the tool retains a feature for each track. The default is one hour. The tool measures the window using elapsed processing time, not the start times stored in the features.

    • Features that pass downstream and duplicates restart the window. Features rejected as out of order do not restart it. Continuous duplicates can therefore remain suppressed indefinitely. The window does not periodically forward unchanged features.

    • After the window expires, the next feature for that track passes without duplicate or timestamp-ordering comparison and becomes the new retained feature. It can pass even if it matches the previously retained feature or has an earlier start time.

How the tool works

A feed polls a vehicle location API every 10 seconds, but the source updates vehicle reports every 30 seconds. Between updates, the API returns the same report, including its original timestamp and point geometry.

Each feature contains a Vehicle ID (the Track ID key field), a Report time (the Start time key field), and point geometry. No fields are ignored, and the track retention time window is one hour.

The following features arrive in order on the same day. Location shows the point geometry as longitude and latitude in WGS 84. Arrival time is included only to explain the example and is not an input attribute.

Vehicle ID

Report time

Location (longitude, latitude)

Result

Reason

Arrival time

Vehicle_5

9:00:00 a.m.

(-117.195, 34.057)

Passed

First report for this vehicle.

9:00:00 a.m.

Vehicle_5

9:00:00 a.m.

(-117.195, 34.057)

Dropped

Same report time and geometry.

9:00:10 a.m.

Vehicle_5

9:00:00 a.m.

(-117.195, 34.057)

Dropped

Same report time and geometry.

9:00:20 a.m.

Vehicle_5

9:00:30 a.m.

(-117.194, 34.057)

Passed

Source updated the report time and location.

9:00:30 a.m.

Vehicle_5

9:00:30 a.m.

(-117.194, 34.057)

Dropped

Same report time and geometry as the updated report.

9:00:40 a.m.

Only two of the five features pass downstream. The tool drops the repeated polls because their attribute values and geometry match the retained report. Because the comparison includes Report time, a newer report time makes a feature different even if the vehicle has not moved.

Parameters

The following table describes the parameters for the tool:

Parameter

Description

Data type

Input layer

The features to evaluate. You must identify a Track ID key field.

Features

Fields to ignore

Fields to exclude when comparing features for duplicates. You cannot ignore the Track ID key field or feature geometry.

If the Start time key field is ignored, the tool still drops features with a start time earlier than the latest retained feature for the same track ID.

The tool does not select fields by default.

Fields

Track retention time window

How long the tool retains a feature for each track ID, measured using processing time rather than feature timestamps. Incoming features, including duplicates, restart this window.

When the Start time key field is configured, the tool drops features with a start time earlier than the retained feature, and those features do not restart the window. After the window expires, the next feature for that track passes without comparison.

The default is 1 hour.

String

Output layer

The output contains features that pass the tool's evaluation, with their original fields, attribute values, time values, and geometry.

Fields selected in Fields to ignore remain in the output. The tool does not add fields or provide a separate output for dropped features.

Features

Output layer

The output contains features that pass the tool's evaluation, with their original fields, attribute values, time values, and geometry. Fields selected in Fields to ignore remain in the output. The tool does not add fields or provide a separate output for dropped features.

Considerations and limitations

Consider the following when using the tool:

  • The tool does not provide a numeric or spatial tolerance for duplicate comparison. Small changes in compared values or geometry can cause features to pass.

  • If features are expected to arrive out of order, use Delay and Reorder Tracks to reorder features before duplicate evaluation. Reordering adds latency and is limited by that tool's time window and maximum feature count.

  • The tool does not reduce feed requests or the bandwidth and API quota they consume. To reduce requests, adjust the feed's polling interval.