Feature Layer (new)
The Feature Layer (new) output in ArcGIS Velocity allows you to write output features to a new ArcGIS feature layer. When you write to this output type, ArcGIS Velocity creates either a spatiotemporal feature layer and map image layer or an ArcGIS Online hosted feature layer. After creation, you can use the layer or layers in Velocity and add them to maps, apps, and more in your ArcGIS Online organization. The output can accept multiple inbound pipelines. When more than one pipeline is connected, at least one pipeline must include every field in the feature layer schema.
Feature layers created by Velocity have key differences from other ArcGIS Online hosted feature layers, including the following:
The associated data is stored in a spatiotemporal feature layer. As a result, the corresponding map image layer provides dynamic rendering and on-the-fly aggregation.
You must define a data retention policy to delete older data from the feature layer. You can set the data retention options to 1 hour, 1 day, 1 week, 1 month, 6 months, or 1 year. You must specify a value in the Data retention (time period for retaining data) parameter for any feature layer whose data grows over time. For example, if the Data storage method parameter is set to Add all new features and you enable the Keep existing features and schema option each time the analytic runs, you must set the data retention option.
You can export older data to a feature layer archive before it is deleted using the Data retention (time period for retaining data) parameter. Data exported to the feature layer archive is maintained for one year from the date of export. To ensure data is exported to the feature layer archive, choose the Keep existing features and schema option every time the analytic runs. Specify the data retention options, and choose the Export data option under Data export (feature layer archive).
The following are restrictions on the number of records that can be written to ArcGIS Online hosted feature layer outputs:
Real-time analytic output—When writing to an ArcGIS Online hosted feature layer from Velocity real-time analytics, up to 120 records can be written every 15 seconds, for a total of 480 record adds or updates per minute.
Runs once big data analytic output—Up to 50,000 records can be written to an ArcGIS Online hosted feature layer output.
Scheduled and recurring big data analytic output—The number of records that can be written to an ArcGIS Online hosted feature layer output depends on the run frequency of the analytic. For each minute between each run, up to 480 records can be written per minute. For example, an analytic that runs every minute can write 480 output features per run. An analytic that runs every 5 minutes can write 2,400 output features per run. An analytic that runs every 30 minutes can write 14,400 output features per run.
Examples
The following are example use cases for the output:
After ingesting event data from IoT sensors using a feed and processing it in a real-time analytic, you want to store the features in a new feature layer.
After performing analysis in a big data analytic, you want to write the resulting analysis features to a new feature layer.
You have configured a feed in a real-time analytic to process events as they occur. You only want to store the latest state of your features as identified by the track ID in a new feature layer.
You monitor environmental stations that each have an air quality sensor and a weather sensor reporting at different times, and you want a new feature layer that stores the latest state of each station. One pipeline carries the full station schema, and separate pipelines carry only the air quality or weather readings. All pipelines connect to the same output, so an update from one sensor changes only that sensor’s fields and keeps the latest values from the other sensor.
Usage notes
Note the following when working with the output:
If you enable the Keep existing features and schema option each time the analytic is run, the features written to the new output feature layer must maintain a consistent schema.
The new feature layer and map image layer created by the output do not appear until the analytic is started and has successfully generated the output.
You must specify a data retention strategy in the Data retention (time period for retaining data) parameter. The Data retention (time period for retaining data) and Data export (feature layer archive) options do not apply if you enable the Replace existing features and schema option each time the analytic runs.
If the Data storage method parameter is set to Keep only latest feature for each Track ID value, Velocity updates only the editor user and date for a track observation after the first observation.
When the Data storage method parameter is set to Keep only latest feature for each Track ID value, choose the Keep the newer feature for a Track ID based on Start Time option in the Advanced options section. Velocity then compares the time specified in the Start Time option of each incoming feature to the stored feature for that track ID and skips the write if the incoming feature is older. This preserves the latest observation when events arrive out of order (for example, from multiple sources or Kafka).
This parameter requires a
START_TIMEfield. If none is defined, a validation warning appears and each incoming feature is treated as new. This applies only to spatiotemporal feature layers.
Note:
The spatiotemporal feature layer name must be unique within an organization.
In Velocity, the spatiotemporal feature layer name must be different from the feed and stream layer names. If there is a duplicate name, you cannot create a real-time or big data analytic in Velocity. This only applies to Velocity feature layer outputs; it does not apply to ArcGIS Online hosted feature layers.
- The Simplify feature geometry parameter is only available when the Store data in spatiotemporal feature layer parameter is turned on. If your feature geometries are already simple, you can leave this parameter off to improve write performance.
Multiple inbound pipelines
Multiple pipelines can connect to a single Feature Layer (new) output in real-time and big data analytics. There is no upper limit on the number of inbound pipelines.
Velocity checks all inbound pipelines. At least one pipeline must include every field in the schema. The feature layer is created with that full schema. The schema is not a union of the fields from all pipelines. Every other pipeline must carry the same schema or a subset of it. For example:
| Inbound pipeline | Pipeline schema | Result |
|---|---|---|
| Pipeline 1 | A, B, C, D, E | Valid. Includes every field. The feature layer is created with schema A, B, C, D, E. |
| Pipeline 2 | A, B, C | Valid. A subset of the full schema. |
| Pipeline 3 | A, C, D, E | Valid. A subset of the full schema. |
If only pipelines 2 and 3 are connected (A, B, C and A, C, D, E), neither pipeline includes every field, so validation fails and the feature layer is not created.
Note the following when connecting multiple pipelines to the output:
All inbound pipelines must have the same geometry type (point, line, polygon, or tabular).
Each field in a subset pipeline must have the same name and data type as the matching field in the full schema. A field that is not in the full schema returns a validation error.
When the Data storage method parameter is set to Keep only latest feature for each Track ID value, each inbound pipeline must include the field tagged as Track ID so features from every pipeline can be matched to the correct record.
A feature layer output cannot be added more than once to the same analytic. To write several pipelines to one feature layer, connect each pipeline to the single output.
Field values are written as follows:
Fields that are not in a pipeline’s schema are not written for features from that pipeline. A field that is missing is different from a field that is present with a null value.
When the Data storage method parameter is set to Keep only latest feature for each Track ID value, a feature from a subset pipeline updates only the fields in that pipeline’s schema. Existing values in the other fields for that track ID are kept.
When the Data storage method parameter is set to Add all new features, each feature is added as a new record. Fields that are not in the pipeline’s schema are null for that record.
If a field is in a pipeline’s schema and its value is null, the null value is written and replaces the existing value.
Parameters
The following table describes the parameters for the output:
|
Parameter |
Description |
Data type |
|---|---|---|
|
Feature layer name |
The name of the new feature layer. |
String |
|
Store data in spatiotemporal feature layer |
After configuring source connection parameters, define the schema and the key parameters. If you uncheck this parameter (false), Velocity writes data to a hosted feature layer in your organization, and its storage consumes credits. |
Boolean |
|
Data storage method |
Specifies whether all new features are added to the feature layer or existing features are updated. Options include the following:
Note:If Keep only latest feature for each Track ID value is chosen, this value is |
Boolean |
|
Keep the newer feature for a Track ID based on Start Time |
This parameter is available in the Advanced options section when the Data storage method parameter is set to Keep only latest feature for each Track ID value. When this parameter is set to Yes with the Track ID and Start Time options specified, Velocity keeps the newer feature per track ID and skips writes for older incoming features. When this parameter is set to No (default), incoming features overwrite the stored feature for a track ID as they arrive. This parameter requires a This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
Boolean |
|
Each time the analytic runs |
Specifies how existing data in the feature layer is handled each time the analytic starts. Options include the following:
Note:If Replace existing features and schema is chosen, this value is |
Boolean |
|
Aggregation styles |
Specifies bin styles for dynamic aggregation of the data. The default bin styles are populated automatically, and you can modify them. You can also define bin styles for more spatial references or projections. This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
Geohashes |
|
Simplify feature geometry |
Simplifies feature geometries to ensure that Velocity writes them to the database correctly. If you know the feature geometries are already simple, you can leave this parameter turned off to improve write performance. This parameter is available in the Geometry options section. This parameter is only applicable when the Store data in spatiotemporal feature layer parameter is turned on. |
Boolean |
|
Editor tracking |
Editor tracking sets the username and date when feature records are created or updated in the output feature layer. Turn on the Enable editor tracking option on the output feature layer. If you turn this parameter on, it appends the This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
Boolean |
|
Creator and editor username |
Specifies the creator or editor username for processed data. The username can be derived from a field of the incoming data or set as the username of the analytic owner. The Creator and editor username parameter does not apply if the Editor tracking parameter has been turned off. This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
String |
|
Feature Service Portal Item ID |
The portal item ID of the corresponding feature layer item created in ArcGIS Online. This parameter is not displayed when configuring the output and cannot be populated or modified. It is automatically populated by ArcGIS Velocity. |
String |
|
Map Service Portal Item ID |
The portal item ID of the corresponding map layer item created in ArcGIS Online. This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
String |
|
Data retention (time period for retaining data) |
Specifies the data retention strategy, or time period after which older data is deleted from the feature layer. Options include the following: 1 hour, 1 day, 1 week, 1 month, 6 months, and 1 year. This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
String |
|
Data export (feature layer archive) |
Specifies whether purged data is exported to a feature layer archive before being purged. Options include the following: Do not export data and Export data. Data exported to the feature layer archive is maintained for one year from the date of the export. The Data retention (time period for retaining data) parameter does not apply if the Replace existing features and schema option is chosen. This parameter is only applicable when data is stored in a spatiotemporal feature layer. |
String |
|
Additional logging (optional) |
Specifies whether logging for raw requests issued by Velocity is turned on. The default is False. Note:Only turn on this parameter for troubleshooting purposes and turn it off when troubleshooting is complete. Contact Esri Technical Support for help with troubleshooting. |
Boolean |
Considerations and limitations
Consider the following when using the output:
If you choose the Keep existing features and schema option, each time the analytic is run, the features written to the new output feature layer must maintain a consistent schema.
The creator must own the feature layer and map image layer portal items associated with the Feature Layer (new) output for Velocity to write to the output. To change ownership of an item, reconfigure the analytic to write to an existing feature layer using the Feature Layer (existing) output.
When multiple pipelines are connected and no single pipeline includes every field, validation returns an error and the feature layer is not created. A subset pipeline with a field that is not in the full schema, or with a field whose data type is different, also returns a validation error.
When a feature layer is hosted in a spatiotemporal data store
If you check the Store data in spatiotemporal feature layer parameter, Velocity creates a hosted feature layer in the ArcGIS Data Store spatiotemporal big data store. You cannot share these layers publicly.
When a feature layer is not hosted in a spatiotemporal data store
If you uncheck the Store data in spatiotemporal feature layer parameter, Velocity creates a hosted feature layer in ArcGIS Online. Consider the following when using this output:
If you choose the Replace existing features and schema option, each time the analytic runs, the existing feature layer is replaced. If the feature layer has associated hosted feature layer views, the analytic returns an error message in the logs. All associated views must be deleted and removed from the recycle bin before restarting the analytic.
Hosted feature layer outputs in ArcGIS Online consume credits for storage.
Real-time analytic restrictions for ArcGIS Online hosted feature layer outputs include the following:
Features are written from Velocity real-time analytics to the hosted feature layer once every 15 seconds.
A maximum of 120 events can be written every 15 seconds, allowing up to 480 events per minute. Therefore, the average output data rate over time must not exceed 120 events per 15 seconds, or eight events per second.
If an analytic produces more than 480 feature updates in one minute, excess features are stored in a temporary cache and written to the hosted feature layer during the next edit operation. Temporal order is maintained as features are emptied from the cache. The cache is maintained for up to 4,800 features, after which new records can be dropped while the cache is emptied.
When using the Keep only latest feature for each Track ID value storage option, if there is more than one observation for a given track ID within the 15 seconds between edits being sent, only the latest feature for that track is written to the hosted feature layer. The additional observations for the track do not count toward the 480 edits per minute maximum. In other words, higher-velocity ingestion can be supported with this storage option if there are 480 or fewer unique assets or devices being tracked with updates more frequently than every minute.
Big data analytic restrictions for ArcGIS Online hosted feature layer outputs include the following:
Features are written from Velocity big data analytics to the hosted feature layer at the end of each run.
Non-scheduled big data analytics that run once write up to 50,000 records to an ArcGIS Online hosted feature layer output.
For big data analytics that are scheduled or run periodically, the number of output features that can be written to an ArcGIS Online hosted feature layer output depends on the run frequency. For each minute between each run, up to 480 records can be written per minute. For example, an analytic that runs every minute can write 480 output features per run, an analytic that runs every 5 minutes can write 2,400 output features per run, and an analytic that runs every 30 minutes can write 14,400 output features per run.
Currently, big data analytics that run once can only write output to ArcGIS Online hosted feature layers in the WGS84 (4326) spatial reference. Use the Project tool before this output if necessary.