Skip to main content

Create Cloud Storage Connection File (Data Management Tools)

Summary

Creates a connection file for ArcGIS-supported cloud storage. This tool allows existing raster geoprocessing tools to write cloud raster format (CRF) datasets into the cloud storage bucket or read raster datasets (not limited to CRF) stored in the cloud storage as input. The tool also creates a cloud storage connection file that you can use to access Apache Parquet files for mapping.

Usage

  • You must provide the necessary information to make a cloud storage connection—such as access key ID, secret access key string, and bucket name—to run this tool. You can also use alternate authentication methods using the Provider options parameter.

    Learn more about connecting to cloud stores

  • The tool outputs a binary cloud storage connection file (.acs) in ArcGIS Cloud Storage format.

  • When accessing a raster dataset or Parquet file stored in a cloud storage location, you can reference it through a file path such as c:/temp/amazons3.acs/someraster. If the dataset or file is stored in a folder in the bucket, the folder name must be included in the path, for example, c:/temp/amazons3.acs/foldername/someraster.

  • This tool supports connections to Amazon Simple Storage Service (S3) buckets, Microsoft Azure Blob Storage containers, Microsoft Azure Data Lake Storage Gen2, Alibaba Cloud Object Storage Service (OSS) buckets, Google Cloud Storage Service (GCS) buckets, WebHDFS, MinIO, and Ozone Object Storage Service buckets. Amazon S3, Azure Blob Storage containers, Microsoft Azure Data Lake Storage Gen2, and GCS buckets are supported when accessing a Parquet file.

  • The tool validates the credentials provided at run time. If the connection cannot be made, the tool returns a warning.

  • OAuth 2.0 authentication with Azure Active Directory is supported with user-interactive login for Azure and Azure Data Lake Storage Gen2. OAuth 2.0 authentication for Google Cloud Storage Service is also supported. See Connect to authentication providers from ArcGIS Pro for configuration. Once configured and the user is signed in, the authentication parameter becomes active for Google, Azure, and Azure Data Lake Storage Gen2. Region and Endpoint will be retrieved from the configuration.

  • Support for user-assigned managed identities using Entra ID on an Azure virtual machine is available for Azure and Azure Data Lake Storage Gen2.

  • Role-based access control (RBAC) is available for Amazon, Azure, and Google cloud providers. Keeping all authentication parameters empty while using an Amazon Elastic Compute Cloud (EC2), Azure virtual machine, or Google Compute Engine will enable ArcGIS Pro to access Blob storage using IAM roles or Azure RBAC. For Amazon, IMDSv1 and IMDSv2 are supported.

  • Microsoft Azure Data Lake Storage Gen2 follows the same options as Azure but provides true directory support and atomic operations using a DFS endpoint. Some network errors during cloud operations are retried following exponential backoff.

  • Support for AWS IAM Identity Center (AWS SSO) is available for Amazon. To use AWS SSO, install AWS CLI, and configure and sign in to AWS SSO. Get the SSO start URL and SSO region for AWS SSO configuration from your administrator. The session token management will be handled by ACS automatically until the AWS SSO login expires. If the login expires, sign in again to keep the ACS valid.

    To work with AWS SSO, the Region (Environment) and Service End Point parameter values and the Provider Options parameter's AWS_PROFILE option are required and must be accurate.

  • Support for an Amazon S3 Express One Zone directory bucket is available for Amazon. Your administrator needs to provide S3 access control list (ACL) permission to access the directory bucket for your account. All the authentication methods available for regular buckets are supported for directory buckets as well. If you have a temporary session token to access directory buckets, use the Access Key ID and Secret Access Key parameters and the Provider Options parameter's AWS_S3SESSION_TOKEN option.

  • For performance considerations and additional information, see the GDAL virtual file systems documentation.

    Caution:

    Esri does not test, certify, or guarantee that S3-compatible storage providers will be compatible with ArcGIS, and Esri Technical Support is not available to troubleshoot compatibility issues.

Parameters

Label Explanation Data type

Connection File Location

The folder path where the connection file (.acs) will be created.

Folder

Connection File Name

The name of the cloud storage connection file.

String

Service Provider

Specifies the cloud storage service provider that will be used.

  • Azure—The service provider will be Microsoft Azure, and accessing a Parquet file is supported.

  • Amazon—The service provider will be Amazon S3, and accessing a Parquet file is supported.

  • Google—The service provider will be Google Cloud Storage, and accessing a Parquet file is supported.

  • Alibaba—The service provider will be Alibaba Cloud Storage.

  • WebHDFS—The service provider will be WebHDFS.

  • MinIO—The service provider will be MinIO.

  • Azure Data Lake—The service provider will be Microsoft Azure Data Lake Storage, and accessing a Parquet file is supported.

  • Ozone—The service provider will be Ozone.

  • Web—The service provider that will be used to access generic HTTP URLs.

String

Bucket (Container) Name

The name of the cloud storage container where the raster dataset or Parquet file is stored. If you're using this location for raster geoprocessing tool output, this is the container where the output raster dataset will be stored. Many cloud providers also call this container a bucket.

String

Access Key ID (Account Name)

(Optional)

The access key ID string for the specific cloud storage type. It can also be the account name, as is the case with Azure.

String

Secret Access Key (Account Key)

(Optional)

The secret access key string that will be used to authenticate the connection to cloud storage.

Encrypted String

Region (Environment)

(Optional)

The region string for the cloud storage. If provided, the value must use the format defined by the cloud storage choice. The default is the selected cloud provider's default account.

String

Service End Point

(Optional)

The service endpoint (URI) of the cloud storage, such as oss-us-west-1.aliyuncs.com. If no value is provided, the default endpoint for the selected cloud storage type will be used. The CNAME redirected endpoint can also be used if needed.

String

Provider Options

(Optional)

The configuration options pertaining to the specific type of cloud service. Some services offer options, some do not. You only need to set this parameter if you want to turn on the options. Only a subset of options are supported when creating a cloud storage connection file to access a Parquet file, as noted in the following descriptions.

  • Azure and Microsoft Azure Data Lake Storage

    • AZURE_STORAGE_SAS_TOKEN—Specify a shared access signature. Ensure that its value is URL encoded and does not contain leading '?' or '&' characters. When using this option, the Secret Access Key (Account Key) parameter must be empty. This option is supported for accessing a Parquet file.

    • AZURE_NO_SIGN_REQUEST—Anonymously connect to buckets (containers) that don't require authenticated access. When using this option, the Secret Access Key (Account Key) parameter must be empty. The default value is False. This option is supported for accessing a Parquet file.

    • AZURE_STORAGE_CONNECTION_STRING—Specify an Azure Storage connection string. This string embeds the account name, key, and endpoint. When using this option, the Access Key ID (Account Name) and Secret Access Key (Account Key) parameters must be empty. This option is supported for accessing a Parquet file.

    • CPL_AZURE_USE_HTTPS—Set to False to use HTTP requests. Some servers may be configured to only support HTTPS requests. The default value is True.

    • AZURE_IMDS_OBJECT_ID—Specify the Object ID of the managed identity authenticated using Azure Instance Metadata Service (IMDS) if your Azure VM has multiple user-assigned managed identities set.

    • AZURE_IMDS_CLIENT_ID—Specify the Client ID of the managed identity authenticated using Azure IMDS if your Azure VM has multiple user-assigned managed identities set.

    • AZURE_IMDS_MSI_RES_ID—Specify the Resource ID of the managed identity authenticated using Azure IMDS if your Azure VM has multiple user-assigned managed identities set.

  • Amazon and MinIO

    • AWS_NO_SIGN_REQUEST—Anonymously connect to buckets (containers) that don't require authenticated access. The default value is False. This option is supported for accessing a Parquet file.

    • AWS_SESSION_TOKEN—Specify temporary credentials. This option is supported for accessing a Parquet file.

    • AWS_S3SESSION_TOKEN—Specify temporary credentials to access a directory bucket.

    • AWS_PROFILE—AWS credential profiles are automatically used when the access key or ID is missing. This option can be used to specify the profile to use. This option is supported for accessing a Parquet file.

    • AWS_REQUEST_PAYER—Requester Pays buckets can be accessed by setting this option to requester.

    • AWS_Virtual_Hosting—If you use Amazon S3 or S3-compatible cloud providers that support only path-style requests, set this option to True. It is recommended that you use virtual hosting if it's supported. The default value is True.

    • CPL_VSIS3_USE_BASE_RMDIR_RECURSIVE—Some older S3-compatible implementations do not support the Bulk Delete operation. Set this option to False for these providers. The default value is True.

    • AWS_HTTPS—Set to False to use HTTP requests. Some servers may be configured to only support HTTPS requests. The default value is True.

  • Google

    • GS_NO_SIGN_REQUEST—Anonymously connect to buckets (containers) that do not require authenticated access. The default value is True. This option is supported for accessing a Parquet file.

    • GS_USER_PROJECT—Requester Pays buckets can be accessed by setting OAuth2 keys and a project for billing. Set the project using this option and set OAuth2 keys using other options and not HMAC keys as a secret access key or ID.

    • GS_OAUTH2_REFRESH_TOKEN—Specify OAuth2 Refresh Access Token. Set OAuth2 client credentials using GS_OAUTH2_CLIENT_ID and GS_OAUTH2_CLIENT_SECRET.

    • GOOGLE_APPLICATION_CREDENTIALS—Specify Service Account OAuth2 credentials using a .json file containing a private key and client email address. This option is supported for accessing a Parquet file.

    • GS_OAUTH2_PRIVATE_KEY—Specify Service Account OAuth2 credentials using a private key string. GS_AUTH2_CLIENT_EMAIL must be set.

    • GS_OAUTH2_PRIVATE_KEY_FILE—Specify Service Account OAuth2 credentials using a private key from a file. GS_AUTH2_CLIENT_EMAIL must be set.

    • GS_AUTH2_CLIENT_EMAIL—Specify Service Account OAuth2 credentials using a client email address.

    • GS_AUTH2_SCOPE—Specify Service Account OAuth2 scope. Valid values are https://www.googleapis.com/auth/devstorage.read_write (the default) and https://www.googleapis.com/auth/devstorage.read_only.

    • GDAL_HTTP_HEADER_FILE—Specify bearer authentication credentials stored in an external file.

  • Alibaba

    • OSS_Virtual_Hosting—If you use Alibaba or S3-compatible cloud providers that support only path-style requests, set this option to True. It is recommended that you use virtual hosting if it's supported. The default value is True.

    • OSS_HTTPS—Set to False to use HTTP requests. Some servers may be configured to only support HTTPS requests. The default value is True.

  • WebHDFS

    • WEBHDFS_REPLICATION (integer)—The replication value is used when creating a file.

    • WEBHDFS_PERMISSION (decimal)—A permission mask is used when creating a file.

If multiple authentication parameters are provided, precedence is as follows:

  • Azure—AZURE_STORAGE_CONNECTION_STRING, account name or key, AZURE_STORAGE_SAS_TOKEN, AZURE_NO_SIGN_REQUEST, or RBAC.

  • Amazon—AWS_NO_SIGN_REQUEST, access ID or key or AWS_SESSION_TOKEN, AWS Credential Profile, or IAM Role.

  • Google—GS_NO_SIGN_REQUEST, access ID or key, GDAL_HTTP_HEADER_FILE, (GS_OAUTH2_REFRESH_TOKEN or GS_OAUTH2_CLIENT_ID and GS_OAUTH2_CLIENT_SECRET), GOOGLE_APPLICATION_CREDENTIALS, (GS_OAUTH2_PRIVATE_KEY or GS_OAUTH2_CLIENT_EMAIL), (GS_OAUTH2_PRIVATE_KEY_FILE or GS_OAUTH2_CLIENT_EMAIL), or IAM Role.

  • Ozone

    • AWS_PROFILE—AWS credential profiles are automatically used when the access key or ID is missing. This option can be used to specify the profile to use.

    • AWS_Virtual_Hosting—If you use Amazon S3 or S3-compatible cloud providers that support only path-style requests, set this option to True. It is recommended that you use virtual hosting if it's supported. The default value is True.

    • AWS_HTTPS—Set to False to use HTTP requests. Some servers may be configured to only support HTTPS requests. The default value is True.

    • CPL_VSIS3_USE_BASE_RMDIR_RECURSIVE—Some older S3-compatible implementations do not support the Bulk Delete operation. Set this option to False for these providers. The default value is True.

    • x-amz-storage-class—Specify REDUCED_REDUNDANCY for writing to a single container ozone, as it has a single data node.

  • Web

    When using the Web option, an authenticated HTTP/HTTPS endpoint can be accessed using an ACS. Authentication parameters are passed as provider options and will be used to access the endpoint.

    • GDAL_HTTP_USERPWD—The HTTP user and password to use for the connection. It must be in the form of [user name]:[password]. Use GDAL_HTTP_AUTH to determine the authentication method.

    • GDAL_HTTP_NETRC_FILE—Set the location of a .netrc file.

    • GDAL_HTTP_HEADERS—Specify headers as a comma-separated list of key:value pairs.

    • GDAL_HTTP_BEARER—Set HTTP OAuth 2.0 Bearer Access Token to use for the connection. This must be used with GDAL_HTTP_AUTH=BEARER.

    • GDAL_HTTP_AUTH—Select the type of authentication method to use: [BASIC/NTLM/NEGOTIATE/ANY/ANYSAFE/BEARER].

    • GDAL_HTTP_SSLCERT—The file name of the SSL client certificate for PKI authentication.

    • GDAL_HTTP_SSLKEY—The private key file path for the TLS and SSL client certificate for PKI authentication.

    • GDAL_HTTP_KEYPASSWD—The pass phrase to the private key of GDAL_HTTP_SSLKEY.

In addition to the provider options listed above, the ARC_DEEP_CRAWL option can be used with all of the service providers. If set to True, it is used to identify CRFs with no extension and cloud-enabled raster products in the cloud. This is operation intensive and it is recommended that you set this option to False for faster catalog browsing and crawling. The default value is False.

Custom token vending services—such as Planetary Computer's data collection, for example—can be authenticated using the ARC_TOKEN_SERVICE_API (the URL of the token vendor) and ARC_TOKEN_OPTION_NAME (the type of token from the service provider) provider options.

Note:

The GDAL_DISABLE_READDIR_ON_OPEN option is available with all the service providers. To improve the performance of loading cloud-based rasters, this option is set to NO by default. If the raster resides in a folder that contains more than 30,000 items, set this option to YES.

Value table columns:

  • Name—The name of the provider configuration.

  • Value—The value of the provider configuration.

Value Table

Folder

(Optional)

The folder in the Bucket (Container) Name parameter value where the raster dataset or Parquet file is stored. If using this location for output for raster geoprocessing tools, this is the folder where the output raster dataset will be stored.

String

Authentication

(Optional)

The connection name of OAuth 2.0 authentication.

A valid connection must be configured on the Options dialog box on the Authentication tab.

String

Derived output

Label Explanation Data type

Output Connection File

The output cloud storage connection file path.

File

Environments

This tool does not use any geoprocessing environments.

Licensing information

  • Basic: Yes
  • Standard: Yes
  • Advanced: Yes