Skip to main content

ArcGIS Data Pipelines Server administrator operations

The following sections describe tools and properties available to ArcGIS Data Pipelines Server administrators. Each tool or property is described along with its path. Tools available in the install directory can be found at <ArcGIS Data Pipelines Server install directory>\tools, and operations located in the ArcGIS Data Pipelines Server administration site can be found by browsing to the URL https://machine.domain.com:14443/arcgis/admin.

Update system properties

Administrators can edit ArcGIS Data Pipelines Server properties to fit their organization. Custom property values can be set at https://machine.domain.com:14443/arcgis/admin/system/properties/update. The following properties are available for configuration:

  • maxWorkersPerMachine—Use this property to increase or decrease the number of workers that can be active simultaneously. Workers are the compute resources that are allocated to power the Data Pipelines editor, scheduled task runs, and stand-alone data pipeline runs.

  • maxMemoryPerWorkerInMiB—Use this property to increase or decrease the amount of memory each worker can use. For example, if you process large amounts of data, you may need to increase this value in order for the runs to complete successfully.

  • reservedInteractiveWorkersPerMachine—Use this property to increase or decrease the number of workers that are reserved for interactive authoring in the Data Pipelines editor. For example, if you have a single-machine ArcGIS Data Pipelines Server site, and maxWorkersPerMachine is set to 6, then by default two of those workers will be reserved specifically for working in the Data Pipelines editor, and four will be available for scheduled task runs and stand-alone data pipeline runs.

  • allowedHosts—Use this property to specify which hosts can be used to read in data when using the URL input. The default is none (""). It is recommended you specify a list of hosts to allow read access to. For example, specify the hosts in a list as follows:

    "allowedHosts": "exampleHost.com, mydata.org, www.test.ca"

    To allow users to read data from all hosts, specify a wildcard value of (.*). Use wildcards (*) with caution as it grants users the ability to read data from any host. For example, to allow reading from all hosts specify the following:

    "allowedHosts": "(.*)"

    If a URL redirects to another host, both hosts must be specified in the allowed hosts list.

  • allowedFileSharePaths—Use this property to specify which file share paths are accessible by the File share input. By default, all file shares that are accessible by the ArcGIS Data Pipelines Server account will be available for use in the File share input. Some directories are always restricted, including install directories and system managed or OS protected directories.

    For example, to allow access to two different network shares, specify the following property:

    "allowedFileSharePaths": [ "\\\\network_path\\my_data", "\\\\other_network_path\\my_other_data"]

    "allowedFileSharePaths": [ "/net/my_network/my_data", "/other_network_path/my_other_data"]

    To block access to all file shares, specify an empty list ([]) as follows:

    "allowedFileSharePaths": []

    Note:

    For single machine ArcGIS Data Pipelines Server sites, local drives such as c:\ are supported. For sites with multiple machines, local drives are not supported. You must use shared network paths that are accessible to all machines in the site.

Import an existing server certificate

To import an existing server certificate, click Home > Machines > MachineName > sslCertificates > importExistingServerCertificate.

This operation imports an existing server certificate into the keystore. If the certificate is a Certificate Authority (CA) signed certificate, you must first import the CA root or intermediate certificate using the importRootOrIntermediate operation.

Import a root certificate

To import a root certificate, click Home > Machines > MachineName > sslCertificates > importRootOrIntermediate.

This operation imports a CA's root and intermediate certificates into the keystore. To create a production-quality CA-signed certificate, add the CA's certificates to the keystore that enables the SSL mechanism to trust the CA (and the certificates it has signed). While most of the popular CA's certificates are already available in the keystore, you can use this operation if you have a custom CA or specific intermediate certificates.

Update the security configuration

To update the security configuration, click Home > Security > SecurityConfig > UpdateSecurityConfig.

This operation updates the security configuration, including TLS protocols and cipher suites, for your ArcGIS Data Pipelines Server site. This operation causes the REST service endpoints to be redeployed on every server machine in the site. If you updated the communication protocol as part of this operation, it takes ArcGIS Web Adaptor one minute to recognize changes to the communication protocol of your site.

Unregister a machine from a site

Before you remove an ArcGIS Data Pipelines Server machine from a site, consider the following:

  • If there are active compute resources running on the machine you remove, they will be terminated and the progress for any running data pipelines will be lost. It is recommended that you wait for all processes to complete or actively terminate them before removing a machine from the site. To actively terminate compute resources in the Data Pipelines app, all users must disconnect from the editor and cancel any active task runs.

    To actively terminate compute resources using the REST API, navigate to https://machine.domain.com:14443/arcgis/rest/services/datapipelines/computeresources/<compute resource id> and click terminate. For compute resources with a MODE value of INTERACTIVE, terminate will disconnect any connected editors associated with the compute resource. For compute resources with a MODE value of JOB, terminate will cancel the associated run.

  • After removing a machine from the site, it is recommended to stop the ArcGIS Data Pipelines Server service, and power down or restart the machine. These steps ensure there are no remaining ArcGIS Data Pipelines Server processes running.

  • If a Web Adaptor is registered with a site, it may take some time for the Web Adaptor to recognize that a machine has been deleted. By default, the Web Adaptor checks every minute to see if machines have been added or deleted from the site. For example, depending on when you delete a machine, the Web Adaptor will not recognize the deleted machine until the next time it checks the site. During this interval, requests will still be forwarded to the deleted machine. This may cause requests to fail and events to be written to the server logs.

    To alleviate this potential issue, you can force the Web Adaptor to recognize the deleted machine by reconfiguring the Web Adaptor. See Install and configure ArcGIS Web Adaptor for ArcGIS Data Pipelines Server for instructions.

  • If you only have one ArcGIS Data Pipelines Server machine in the site and you delete it, the site is also deleted and you are returned to a page where you'll be asked whether you want to create a site or join an existing site. Keep in mind that your existing services and settings are preserved in the configuration store. If you want to re-create the site on another machine, reference the existing configuration store when creating the site.

To unregister a machine from a site, click Home > Machines > Machine name > unregister.

Delete a site

To delete a site, click Home > Delete Site.

This operation deletes the site configuration and releases all server resources. It is suited for development or test servers that need to be cleaned up regularly and can also be performed before uninstallation. Use caution with this option because it deletes all settings and other configurations and is an unrecoverable operation.

This operation performs the following tasks:

  • All server machines participating in the site are stopped.

  • All server machines are unregistered from the site.

  • The configuration store is deleted.

Configure ArcGIS Web Adaptor

To configure ArcGIS Web Adaptor, click Home > System > Web Adaptors > Configuration.

The ArcGIS Web Adaptor configuration is a resource for all the configuration parameters shared across the web adaptors in the site. This resource identifies the shared key used by all the web adaptors to encrypt key data bits in the incoming requests to the server.

Back up the site

It is recommended that you frequently export your site settings through the admin API. You can do this by browsing to Home > Export Site. It is recommended when defining a destination property for the exported site that the location be a network drive, or a location other than the ArcGIS Data Pipelines Server machine itself. The destination must be a location that is accessible to the ArcGIS Data Pipelines Server machine. In the event of a site failure, the site can be recovered by creating a new server site and browsing to Home > Import Site.

Note:

If you have a multimachine ArcGIS Data Pipelines Server site and the site fails, it is only necessary to import the site once as described above and then join other ArcGIS Data Pipelines Server machines to the site normally.

Edit the log settings

To edit the log settings, click Home > Logs > Settings > Edit.

This procedure updates the log settings for the entire server site, such as log output location, level, and format, as well as log file age.