Skip to main content

Watch a Folder for Local Files

The Watch a Folder for Local Files feed, listed in the Web and Messaging group with other types of feeds, is used to read event data from a local file. For example, a file containing delimited text with the location (latitude and longitude), timestamp, and observed speed for a vehicle—recorded in a text file in the Watch a Folder for Local Files feed—can read the data in this type of data file.

Example

The following is an example use case for the feed:

A utility company uses the Watch a Folder for Local Files feed to monitor a directory where field crews upload daily CSV reports. Each file contains sensor readings and equipment status updates. ArcGIS Velocity reads each file line by line, processes the data in batches, and updates dashboards in near real-time. If a file contains headers or is particularly large, the company configures batch size and interval settings to optimize performance. Files are deleted after processing to keep the folder clean and avoid reprocessing.

Usage notes

Keep the following in mind when working with the feed:

  • It is recommended that you use absolute paths for the File Directory parameter—for example, C:\Velocity\input.

  • The Maximum number of lines per batch parameter specifies how many lines to include in each batch of data when reading the file. The Interval for reading lines from the file parameter specifies an amount of time to wait before reading another batch of lines from the file. These parameters must both be set to a value greater than or equal to one to start the feed.

  • The feed watches a specified directory for files matching a name or pattern. When a file matching the specified name or pattern is added to the directory, the feed reads the file.

  • Files in a watched folder are reread, from the beginning of the file, in the following situations:

    • The feed parameters are updated and saved.

    • The feed is stopped and restarted.

  • Data formatted as delimited text might have one or more header rows used to specify attribute field names or attribute data types. Set the Read file as individual lines of text parameter value if you want to specify a number of header rows to skip when reading data starting at the top of the file. Data formats such as JSON do not incorporate header rows.

  • Particularly large files can be read in batches. Set the Max Number of Lines per Batch parameter value to help manage data retrieval by limiting the number of lines retrieved as the file content is ingested. You can set the Interval for Reading Lines from the File (milliseconds) parameter value to specify how many milliseconds to wait before the next batch of lines are retrieved from the file. This helps regulate the pace of data ingestion.

  • If the Delete file after processing parameter is turned on, but the user account does not have permission to delete the content from the directory, Velocity logs an error message.

  • Permission to access, read, and/or delete a specified file is controlled by the operating system. If the operating system fails to indicate required permissions, a feed is timed out after 30 seconds. An error message is displayed to indicate that Velocity could not complete the operation in this case.

  • If the file contains a control character outside the range of alphanumeric printable characters or if there is a schema mismatch while the feed is running, Velocity logs an error.

Parameters

The following are the parameters for the feed:

Parameter

Description

Data type

File Directory

(Required)

The path of the local file to be read—for example, C:\Velocity\feed.

File path

File Filter

Filter based on file name or type. The default is asterisk (*), which means all files in the directory are read.

String

Read file as individual lines of text

Specifies how the content of the file should be read and parsed. By default, this parameter is turned off.

If this parameter is turned on, the contents of the file are read and parsed as individual lines of text.

If this parameter is turned off, the entire file is read and parsed as a complete document.

Boolean

Maximum number of lines per batch

(Conditional)

The maximum number of lines to read from the file in each batch or interval. The default is 1.

The parameter is shown when the Read file as individual lines of text parameter is turned on and is hidden when turned off.

Integer

Interval for reading lines from the file (milliseconds)

(Conditional)

The interval set for reading the number of lines from the file in milliseconds. The default is 1000 milliseconds (one second). The parameter is shown when the Read file as individual lines of text parameter is turned on and is hidden when turned off.

Long

Delete file after processing

Specifies whether the files in the directory are deleted after their content has been processed. By default, this parameter is turned off.

If this parameter is turned on, files are deleted from the directory after being processed.

If this parameter is turned off, files are not deleted from directory after being processed.

Files that are not deleted are reread, from the beginning of the file, if the feed’s parameters are changed and saved or if the feed is stopped and restarted.

Boolean