Manage ArcGIS Data Pipelines Server using the Data Pipelines app
Administrators can use the ArcGIS Data Pipelines app to view and manage ArcGIS Data Pipelines Server logs, compute resources, and system properties.
You must be signed in as an ArcGIS Enterprise administrator to access the Data Pipelines Server page.
Open the Data Pipelines Server page
To open the Data Pipelines Server page, complete the following steps:
Sign in to ArcGIS Enterprise as an administrator and access the Data Pipelines app using the app launcher.
Expand the Manage dropdown then click Data Pipelines Server.
The Data Pipelines Server page appears and displays the Logs view.
View and manage logs
The Logs view displays log messages for the ArcGIS Data Pipelines Server site. By default, it displays messages at the warning level and above for all available time periods.
To find and manage log messages, the following options are available:
Filter messages by Log level, Log time, Source, Machine name, Log codes, or Users. On smaller screens, click Show more filters to access all filters.
For Log codes, enter a single integer, an ascending range of integers, or multiple values separated by commas. For example, enter
2050,1000-1002, or1000-1002, 2050, 3000.Click Manage columns to choose which message properties appear in the table.
Hover over a message and click See more to view the full text in the Log message pane. Click Copy to copy the message text to the clipboard.
Click Load more logs when more messages are available.
Click the refresh button to retrieve the latest messages using the current filters.
Click Delete all logs, and click Delete to confirm.
Caution:
Deleting all logs permanently removes every log message from the server site, not only the messages that match the current filters.
Manage compute resources
The Compute resources view displays the compute resources for the server site. The table includes the process ID, machine name, status, creation time, owner, mode, and peak memory usage for each resource.
To review or terminate compute resources, complete the following steps:
Click Compute resources.
To terminate a resource, open its actions menu and click Terminate.
Terminating an interactive resource disconnects the editors that use it. Terminating a job resource cancels the associated run.
See compute resources for more information.
Configure server settings
The Server settings view contains log properties and system properties for the server site.
Configure log properties
To configure log properties, complete the following steps:
Click Server settings.
In the Log properties section, configure the following settings:
Log location—The directory where the server writes log files.
Log level—The minimum level of messages that the server records.
Max log file age in days—The number of days the server retains log files.
Click Save.
Configure system properties
To configure system properties, complete the following steps:
Click Server settings.
In the System properties section, configure the available settings:
Max workers per machine—The maximum number of compute resources that can be active at the same time on each machine. Compute resources support interactive authoring, scheduled task runs, and stand-alone data pipeline runs.
Reserved interactive workers per machine—The number of compute resources on each machine that are reserved for interactive authoring in the Data Pipelines editor. The remaining resources are available for scheduled task runs and stand-alone data pipeline runs.
Max memory per worker (MiB)—The maximum amount of memory that each compute resource can use. Increase this value when runs that process large amounts of data exceed the available memory.
Max cores per worker—The maximum number of processor cores that each compute resource can use. Choose to use all available cores or specify a maximum number of cores. When you specify a maximum, Max memory per worker (MiB) must provide at least 512 MiB for each core. For example, a maximum of four cores requires at least 2,048 MiB of memory per worker.
Allowed hosts—The hosts that the URL input can access. The following options are available:
Allow all hosts—Uses a
(.*)wildcard to allow access to all hosts.Block all hosts—Uses an empty list to disallow access to all hosts. This is the default.
Allow specific hosts—Uses an explicit list of allowed hosts.
To allow all subdomains of a host, add
(.*).example.comthen addexample.comto allow the root host.For URLs that redirect to another host, add both hosts to the allowed list.
Caution:
Allowing all hosts permits users to read data from any host. Allow only the hosts that users need to access.
Allowed file share paths—The file share paths that the File share input can access. The following options are available:
Allow all paths—Allow access to all file share paths that are accessible to the ArcGIS Data Pipelines Server account.
Block all paths—Disallow access to all file share paths.
Allow specific paths—Allow access to specific file share paths.
Install directories and system-managed or operating system-protected directories are permanently restricted and cannot be configured for access using this property.
Note:
Local paths are supported for single-machine sites. For multiple-machine sites, specify network paths that every machine in the site can access.
Click Save in the System properties section.
See Update system properties for more information.