Skip to main content

FAQ

Answers to frequently asked questions about ArcGIS Data Pipelines in ArcGIS Enterprise are provided.

What is ArcGIS Data Pipelines?

Data Pipelines is an ArcGIS app that allows you to connect to, prepare, and integrate data from various sources. You can perform data preparation and save the results to your Web GIS to complete your organization's workflows. All of this is completed using an intuitive, drag-and-drop interface where you can construct, run, save, share, and reproduce your data preparation workflows.

Does Data Pipelines charge credits?

No. Data Pipelines in ArcGIS Enterprise does not consume credits, and does not support any additional services that consume credits.

How do I access Data Pipelines?

You can access Data Pipelines by using the app launcher and choosing Data Pipelines.

To access Data Pipelines, your organization must have ArcGIS Data Pipelines server federated and set to the Data Pipelines Server role. Additionally, your user account must have the required privileges.

See Requirements to learn more about the privileges and other requirements to access Data Pipelines.

If you are unsure whether you or your organization meet the requirements above, contact your organization administrator.

How can I get started with Data Pipelines?

To get started with installing and deploying ArcGIS Data Pipelines, see the install guide.

To get started with Data Pipelines, see the following tutorials to learn how to connect to and process data, run a data pipeline, and more:

What data can I use in Data Pipelines?

See the Dataset configuration topic for a list of all supported input data sources. Each input has a documentation topic with more information about how to connect to the source, supported file formats, and more.

Can I use ArcGIS Living Atlas layers as input to my data pipeline?

Yes. You can use ArcGIS Living Atlas feature layers as input. To add a layer to a data pipeline, see Feature layer. By default, the feature layer browse dialog box opens to My content. To search for an ArcGIS Living Atlas layer, switch to Living Atlas on the dialog box.

My data was updated in its source location. How do I sync my dataset in my data pipeline?

If the data is regularly updating in the source location and you want to use it in a data pipeline, it is recommended that you do not use the Use caching parameter for inputs. If you do not use caching, Data Pipelines reads the latest data every time you request a preview or run in the editor. If you use caching, only the data available at the time you cached is used.

If you created an output item and need to update it with the latest data, update the output method to update or overwrite the existing data, and run the data pipeline again. You can automate rerunning a data pipeline by scheduling a task for the data pipeline item.

To learn more about automating data pipelines in the Data Pipelines app, see the Schedule a data pipeline task topic.

Where can I store my Data Pipelines results? Can I store them in Amazon S3?

No. The only output formats currently supported by Data Pipelines are feature layers and files written to your content. You cannot write results to other formats or storage containers, including Amazon S3. Data Pipelines can only read from your S3 bucket.

Learn more about output feature layers in Data Pipelines

The editor fails to connect after ArcGIS Data Pipelines Server is configured with ArcGIS Enterprise. How do I resolve this?

Refresh the browser page and try again. If the issue persists, contact Esri Technical Support.

Can I share a data pipeline?

Yes. You can share data pipeline items with groups in your organization or with the public. Only the owner of the item can edit data pipeline items. Use shared update groups so everyone in the group can edit and save the data pipeline. If a data pipeline is shared with a group that does not have shared update capabilities, you can save the data pipeline as an editable copy in your content using the Save As option on the editor toolbar.

Learn more about shared update groups.

Can I undo or redo an action in the Data Pipelines editor?

No, not yet. Undo and redo are not currently supported actions for the editor. These actions may be available in a future release.

Can I copy and paste elements in a diagram?

Yes. You can use command keys to cut (Ctrl+X), copy (Ctrl+C), paste (Ctrl+V), and delete (Delete) elements. Select the elements, and use the command keys to complete the actions.

Can I schedule a data pipeline run?

Yes. You can schedule data pipelines to run automatically using the following options:

How is Data Pipelines different from ArcGIS Velocity?

There are certain similarities between Data Pipelines and Velocity. Both apps allow you to connect to external data sources and import the data into ArcGIS for use across the ArcGIS system. However, they serve distinct purposes. Velocity is specifically designed for real-time processing of high-speed data streams from sensors and similar sources. It also is focused on enabling analytics such as device tracking, incident detection, and pattern analysis. Data Pipelines is primarily a data integration app that focuses on data engineering tasks, particularly for non-sensor-based data streams. While Velocity is used for handling real-time data, Data Pipelines is used for managing and optimizing data that requires updates on a less frequent basis.

How is Data Pipelines different from ArcGIS Data Interoperability?

Both are no-code ETL tools for ArcGIS, supporting data integration, transformation, and cleaning. However, they are different in that Data Pipelines is a web-based app available in ArcGIS Enterprise and ArcGIS Online, while Data Interoperability is an extension for ArcGIS Pro and ArcGIS Enterprise. Data Pipelines is focused on data integration for ArcGIS, with results being written out to ArcGIS content, while Data Interoperability supports a larger set of inputs and file types, and it can write results out to many supported formats.

Can I upgrade ArcGIS Data Pipelines (Beta) from 12.0?

No, upgrading ArcGIS Data Pipelines (Beta) is not supported. Upgrades are supported for ArcGIS Data Pipelines Server version 12.1 and later. For more information, see Upgrade ArcGIS Data Pipelines Server.

What does it mean when a feature is in beta?

When features are in beta, they are subject to change without notice and they are not officially supported. To get support for a feature that is currently in beta, share your experience and seek support through the Beta Features Feedback forum in the Data Pipelines Community.