FAQ
Answers to frequently asked questions about ArcGIS Data Pipelines are provided.
Can I use ArcGIS Living Atlas layers as input to my data pipeline?
My data was updated in its source location. How do I sync my dataset in my data pipeline?
Where can I store my Data Pipelines results? Can I store them in Amazon S3?
Is there a way to undo or redo an action in the Data Pipelines editor?
How is Data Pipelines different from ArcGIS GeoEvent Server?
How is Data Pipelines different from ArcGIS Data Interoperability?
What is ArcGIS Data Pipelines?
Data Pipelines is an ArcGIS app that allows you to connect to, prepare, and integrate data from various sources. You can perform data preparation and save the results to your Web GIS to complete your organization's workflows. All of this is completed using an intuitive, drag-and-drop interface where you can construct, run, save, share, and reproduce your data preparation workflows.
Does Data Pipelines charge credits?
No. Data Pipelines in ArcGIS Enterprise does not consume credits, and does not support any additional services that consume credits. Data Pipelines only consumes credits in ArcGIS Online. To learn more about how Data Pipelines consumes credits in ArcGIS Online, see the ArcGIS Online Compute resources documentation.
Is Data Pipelines available in ArcGIS Online?
Yes. Data Pipelines is available in ArcGIS Online and ArcGIS Enterprise. See the ArcGIS Online topic Introduction to ArcGIS Data Pipelines for more information.
How do I access Data Pipelines?
You can access Data Pipelines by using the app launcher and choosing Data Pipelines (beta).
To access Data Pipelines, your organization must have a Data Pipelines server federated and set to the Data Pipelines role. Additionally, the user account must have the required privileges. See Requirements to learn more about the privileges and requirements to access Data Pipelines.
If you are unsure whether you or your organization meets the requirements above, contact your organization administrator.
How can I get started with Data Pipelines?
If you already have ArcGIS Data Pipelines installed and configured with ArcGIS Enterprise, follow the tutorial to create your first data pipeline.
To get started with installing ArcGIS Data Pipelines, see the install guide.
What data can I use in Data Pipelines?
See the Dataset configuration topic for a list of all supported input data sources. Each input has a documentation topic with more information about how to connect to the source, supported file formats, and more.
Can I use ArcGIS Living Atlas layers as input to my data pipeline?
Yes. You can use ArcGIS Living Atlas feature layers as input. To add a layer to a data pipeline, see Feature layer. By default, the feature layer browse dialog box opens to My content. To search for an ArcGIS Living Atlas layer, switch to Living Atlas on the dialog box.
Can I connect to my datasets from Enterprise Geodatabases?
No, not yet. In future releases, the following types of Enterprise Geodatabases will be supported:
Microsoft SQL Server
PostgreSQL
Oracle
The data sources in this list are not guaranteed for a specific release, and data sources that are not listed here may be added. If you have suggestions for data sources that will improve your workflows, leave a comment in the Data Pipelines Community forums.
My data was updated in its source location. How do I sync my dataset in my data pipeline?
If the data is regularly updating in the source location and you want to use it in a data pipeline, it is recommended that you do not use the Use caching parameter for inputs. If you do not use caching, Data Pipelines reads the latest data every time you request a preview or run in the editor. If you use caching, only the data available at the time you cached is used.
If you created an output feature layer and need to update it with the latest data, use the Replace or Add and update options in the Feature layer tool, and run the data pipeline again. You can automate rerunning a data pipeline by scheduling a task for the data pipeline item.
To learn more about automating data pipelines in the Data Pipelines app, see the Schedule a data pipeline task topic.
Where can I store my Data Pipelines results? Can I store them in Amazon S3?
No. The only output format currently supported by Data Pipelines is a feature layer. You cannot write results to other formats or storage containers, including Amazon S3. Data Pipelines can only read from your S3 bucket.
Learn more about output feature layers in Data Pipelines
Can I geocode addresses using Data Pipelines?
No, not yet. This capability may come in a future release.
What tools are coming in future releases?
The following tools may be included in future releases:
Find and replace—Search fields for specific values and replace them with a new value.
Geocode addresses—Use string addresses from a table or file to return the geocoded results.
The tools in this list are not guaranteed for any release, and tools that are not listed here may be added. If you have suggestions for tools that will improve your workflows, leave a comment in the Data Pipelines Community forums.
Can I share a data pipeline?
Yes. You can share data pipeline items with groups in your organization or with the public. Only the owner of the item can edit data pipeline items. Use shared update groups so everyone in the group can edit and save the data pipeline. If a data pipeline is shared with a group that does not have shared update capabilities, you can save the data pipeline as an editable copy in your content using the Save As option on the editor toolbar.
Is there a way to undo or redo an action in the Data Pipelines editor?
No, not yet. Undo and redo are not currently supported actions for the editor. These actions may be available in a future release.
Is there a way to copy and paste elements in a diagram?
Yes. You can use command keys to cut (Ctrl+X), copy (Ctrl+C), paste (Ctrl+V), and delete (Delete) elements. Select the elements, and use the command keys to complete the actions.
Can I schedule a data pipeline run?
Yes. You can schedule data pipelines to run your data integration workflows on a defined schedule. To learn more about creating data pipeline tasks, see Schedule a data pipeline.
How is Data Pipelines different from ArcGIS GeoEvent Server?
There are certain similarities between Data Pipelines Server and GeoEvent Server. Both server roles allow you to connect to external data sources and import the data into ArcGIS for use across the ArcGIS system. However, they serve distinct purposes. GeoEvent Server is specifically designed for real-time and big data processing, efficiently handling high-speed data streams from sensors and similar sources. It also is focused on enabling analytics such as device tracking, incident detection, and pattern analysis. Data Pipelines Server is primarily a data integration app that focuses on data engineering tasks, particularly for non-sensor-based data streams. While GeoEvent Server is used for handling real-time data, Data Pipelines Server is used for managing and optimizing data that requires updates on a less frequent basis.
How is Data Pipelines different from ArcGIS Data Interoperability?
Both are no-code ETL tools for ArcGIS, supporting data integration, transformation, and cleaning. However, they are different in that Data Pipelines is a web-based app available in ArcGIS Enterprise and ArcGIS Online, while Data Interoperability is an extension for ArcGIS Pro and ArcGIS Enterprise. Data Pipelines is focused on data integration for ArcGIS, with results being written out to a hosted feature layer, while Data Interoperability supports a larger set of supported inputs and file types, and it can write results back to the source.
What does it mean when a feature is in beta?
Features in beta are subject to change without notice and they are not officially supported. To get support for a feature that is currently in beta, share your experience and seek support through the ArcGIS Enterprise 12.0 Beta Features Early Adopter Community.