Use third-party object tracking models with ArcGIS

Available with Image Analyst license.

The object tracking capability in motion imagery allows you to locate an object of interest, track its movement as the video plays, and manage tracked objects throughout the workflow. This capability uses deep learning models that are trained to track moving or stationary objects and re-identify objects after periods of obscuration across video frames to maintain consistent tracking as the video plays. The object tracking capability relies on trained deep learning models packaged as .dlpk files to perform tracking, and you can configure these models directly on the Object Tracking tab.

For details about on how object tracking in motion imagery works, see Object tracking in Motion Imagery

While the object tracking capability in ArcGIS Pro provides deep learning–based tracking, creating a new model within ArcGIS often requires substantial effort. High-quality tracking models depend on extensive labelled training data, and training deep learning models can be computationally demanding. In ArcGIS Pro, you can use third-party deep learning models to track objects. Python developers can prepare their tracking models, package them as a compatible .dlpk files, and load them directly through the object tracking interface for use in motion imagery workflows.

Custom Python object tracking function

You can create a custom object tracking function in Python to integrate third-party deep learning models into the motion imagery tracking workflow. These functions allow ArcGIS Pro to use external deep learning models for initializing, maintaining, and managing tracked objects in video data.

The object tracking function methods are listed in the following table and are described in detail in the subsections below.

Method	Description
`__init__`	Initializes instance variables such as the function name, description, model references, and other attributes required for the object tracking function.
`initialize`	Loads the object tracking model and performs any setup required before tracking begins. This is done once at the start of the function.
`getParameterInfo`	Defines the parameters that the object tracking function accepts. This includes any configuration settings required to load or connect to the model as well as parameters needed for tracking-related settings.
`getConfiguration`	Describes how the function will process video frames and produce tracking outputs. It includes details for any preprocessing or postprocessing steps necessary for the function.
`init_tracker`	Initializes the tracker using the initial video frame and the user-provided bounding box around the object to track. This method sets up the internal state needed to begin tracking.
`track`	Performs tracking on each new video frame. This method updates the object's position and returns the resulting bounding box or centroid for the tracked object.

Function methods

The object tracking function methods are described below.

Use the _init_ method

The __init__ method is the constructor of the custom object tracking class. It initializes instance variables such as the name, description, references to deep learning models, and other attributes required for tracking objects in video data. This method sets up the initial state of the tracker function and ensures all default settings and required properties are defined before tracking begins.

When creating an instance of the object tracking class, the constructor prepares everything necessary for tracking, such as model references, default tracking parameters, or other configuration options needed by the init_tracker and track methods.

class MyObjectTracker: 

    def __init__(self, **kwargs): 

        """ 

        Initializes the object tracker by setting up its core properties,  

        including name, description, and any additional configuration parameters. 

        """ 

        self.name = "Object Tracker" 

        self.description = ( 

            "The `MyObjectTracker` class is designed to perform object tracking " 

            "in video data using a pre-trained deep learning model." 

        ) 

        self.model = None  # Placeholder for the deep learning model (.dlpk) 

        # Additional initialization code here 

        ...

Use the initialize method

The initialize method is called at the start of the custom object tracking function. This method receives kwargs['model'], which is the path or URL to the Esri deep learning package file (.dlpk). Use this method to load the model weights and set up the tracker, ensuring a reference to the loaded model is available for subsequent operations such as init_tracker and track.

def initialize(self, **kwargs): 

    """ 

    Initialize object tracking model parameters, including loading the pretrained 

    deep learning model from a `.dlpk` file. 

    """ 

    dlpk_path = kwargs['model']  # Path or URL to the `.dlpk` file 

     

    # Load the deep learning package 

    self.model = load_deep_learning_model(dlpk_path) 

     

    # Additional initialization code here, e.g., setting default tracking parameters 

    ...

Use the getParameterInfo method

The getParameterInfo method is called after the initialize method and is where the parameters needed by the object tracking function are defined. This method returns a list of input parameters expected by the custom tracker. Each parameter is described using a dictionary containing the name, data type, display name, description, and whether the parameter is required, as shown below.

def getParameterInfo(self): 

    return [ 

        { 

            "name": "confidence_threshold", 

            "dataType": "numeric", 

            "required": False, 

            "displayName": "Confidence Threshold", 

            "description": "Minimum confidence score required to maintain a tracked object.", 

            "value": 0.1 

        }, 

        # Additional tracking parameters here 

        ... 

    ]

Key attributes of each dictionary include the following:

name—A string identifier for the parameter
dataType—The type of data the parameter accepts (string, numeric, Boolean, and so on)
value—The default value for the parameter
required—Boolean indicating whether the parameter is required
displayName—A user-friendly name shown in the UI
description—A detailed description of the parameter

The list of parameters is displayed through the custom model’s arguments in ArcGIS Pro. You can set these values interactively using the object tracking interface or programmatically pass them into the tracking function as keyword arguments.

Use the init_tracker method

The init_tracker method initializes the object tracker using the initial video frame and the user-provided bounding boxes around each object to track. It prepares the internal state required to begin tracking, including assigning unique object IDs, converting bounding boxes into the required format, and calling the underlying tracker’s initialization routine.

def init_tracker(self, frame, boxes): 

    """ 

    Prepare the tracker using the initial frame and bounding boxes. 

    """ 

    # Convert user-provided boxes to float arrays 

    boxes = ensure_numpy_array_of_floats(boxes) 

 

    for box in boxes: 

        # Store unique object ID 

        self.obj_id = int(box[0]) 

        self.labels = np.array([1], dtype=np.int32) 

 

        # Convert from [x_min, y_min, x_max, y_max] to [x, y, w, h] 

        bbox = box[1:] 

        bbox = [bbox[0], bbox[1], bbox[2] - bbox[0], bbox[3] - box[1]] 

 

        # Initialize the underlying tracker 

        self._tracker.initialize(frame, bbox)

Use the track method

The track method updates the object’s position for each new video frame. After the tracker has been initialized using init_tracker, this method is called repeatedly—once per frame—to maintain the object’s trajectory as the video plays.

The method takes the current video frame as input, applies the underlying deep learning–based tracker, and returns the updated bounding box or centroid for the tracked object. It may also produce additional metadata such as tracking confidence scores or object identifiers, depending on the model.

This method is central to the tracking workflow, as it enables continuous object localization across frames while handling challenges such as motion, occlusion, scale changes, and appearance variations.

def track(self, frame): 

    """ 

    Update the tracked object's bounding box using the current frame. 

    """ 

    # Get updated bounding box from the underlying tracker 

    tracked_bbox = self._tracker.track(frame) 
 
    # Convert [x, y, w, h] to [x_min, y_min, x_max, y_max] 

    x, y, w, h = tracked_bbox 

    # Convert center/width/height representation to rectangle coordinates 

    location = cxy_wh_2_rect(...) 


    # int(x_min), int(y_min), int(x_max), int(y_max) 

    x1, y1, x2, y2 = ... 

 

    # Prepare ArcGIS-formatted output 

    all_bboxes = [[float(self.obj_id), float(x1), float(y1), float(x2), float(y2)]] 

    return all_bboxes

Use an Esri .emd file for object tracking

After creating a custom Python object tracking function, you must reference it in the .emd (Esri Model Definition) file by specifying it under the InferenceFunction parameter. This links the .emd file to your Python tracking function so ArcGIS Pro can load and run it during the object tracking workflow.

A typical .emd file looks similar to the following:

 { 

    "InferenceFunction": "MyObjectTracker.py", 

    "ModelType": "ObjectTracker", 

    "ModelFile": "model_weights.pth", 

    // additional keys here 

    ... 

}

Use a custom .dlpk file

To complete a custom object tracking setup, you must package your tracking function and model assets into a .dlpk file. The .dlpk file allows ArcGIS Pro to load your custom tracker through the object tracking capability in Motion Imagery.

Organize the files as follows:

Create a folder that contains the custom object tracking Python file (for example, MyObjectTracker.py) and the Esri .emd model definition file (for example, ObjectTracker.emd).

The folder name must match the .emd file name (excluding extension).

For example, the structure may look similar to the following:
```
ObjectTracker/
├── MyObjectTracker.py
└── ObjectTracker.emd
```
Include any additional assets required by your tracker, such as the following:
- Model weight files (.pth, .onnx, and so on)
- Configuration files
- Supporting modules or utility scripts
- Tracker dependencies (if included locally)
Compress the folder into a ZIP archive.
Rename the .zip file so that it matches the .emd file name but uses the .dlpk extension.

See the following example:
```
ObjectTracker.zip  →  ObjectTracker.dlpk
```

Once packaged, the model can be loaded on the Object Tracking tab and used immediately for initializing, tracking, and managing objects across video frames, supporting efficient and scalable tracking workflows.