Create a data store

You can use the Data Store configuration app to create most ArcGIS Data Store types and join them to your hosting server.

Tip:

If any of the following are true, you must run the configuredatastore utility on the machine where you installed ArcGIS Data Store rather than using the Data Store configuration app:

• You don't have access to a web browser.

• You want to deploy each node in a spatiotemporal big data store with a specific role.

Configuration is done per machine where ArcGIS Data Store is installed. Because the Data Store configuration app is a web app, you can run it from another machine that has access to the machine where you installed ArcGIS Data Store, whereas you must run the configuredatastore utility on the machine where you installed ArcGIS Data Store.

Considerations before creating a data store

Be aware of the following before you create a data store.

Disk space and memory

Consider the amount of disk space and memory needed on the machine where you will create a data store.

The following are examples of how a data store can use large amounts of disk space:

• The relational store can store data for thousands of hosted feature layers, which can take up several gigabytes of disk space.

• If you archive data that is continually coming in from a stream layer or you track locations for numerous mobile workers, the spatiotemporal big data store can fill up quickly.

• Caches for 3D data can be multiple gigabytes or even terabytes, depending on the area included in the 3D layer and the complexity of the data. Size the object store machine (or machines) accordingly.

• To take advantage of cached query responses to improve drawing performance for feature layers, the caches must persist for as long as possible in the object store. To facilitate storage of query response caches for longer periods of time, use a single machine for the object store, and ensure that the single machine contains large amounts of free disk space. How much disk space you need depends on how many layers are enabled for caching, how many unique queries are made on those layers, and how long the queries persist. When used for query response caches, using a cluster of object stores has not been shown to provide as much improvement in drawing performance as using a single machine with large amounts of free disk space.

In all cases, it is important to monitor the data store machines and add disk space as necessary.

You must also ensure that the machines have plenty of memory available. For example, the spatiotemporal big data store consumes approximately 50 percent of available memory on the machine when initially configured. The relational store also preallocates memory to ensure there is enough to process data. High-volume traffic on hosted layers can also consume a considerable amount of memory. Traversing and analyzing the relationships in knowledge graphs is a memory-intensive process.

For these reasons, it is recommended that you configure each type of ArcGIS Data Store on machines separate from each other and separate from ArcGIS Server sites. If you configure more than one type of data store on the same machine, use robust machines with plenty of memory and disk space, but be aware that resource contention could still render one or more of the data stores inoperable.

In all cases, monitor the data store logs for messages about disk space and memory. If a machine runs out of either resource, the data store will stop functioning.

See ArcGIS Data Store system requirements for information about the minimum disk space and memory needed to create each type of data store, and add to that the estimated size of the data or caches you intend to store in each and the processes that will be run on each machine.

Identify the IP address and host name for machines with multiple NIC cards or DNS entries

If you install ArcGIS Data Store on a machine that contains more than one network interface controller (NIC) card or multiple DNS entries (also called records), you must identify which IP address and host name the data store will use for network communication. Before you create a data store on the machine, determine which IP address can be used by other machines on your network to communicate with the ArcGIS Data Store machine, open the ArcGIS Data Store hostidentifier.properties file, and add the IP address to the file.

1. Determine which IP address to use. Open a command prompt on a different machine on the same network and ping the ArcGIS Data Store machine.

2. Sign in to the machine where ArcGIS Data Store is installed and open hostidentifier.properties in a text editor.

   Use the same login account used when ArcGIS Data Store was installed.

   The file is installed in framework\etc of the ArcGIS Data Store installation directory. By default, that location is /home/ags/arcgis/datastore/framework/etc.

   Use the ArcGIS Data Store login account.

   The file is installed in framework\etc of the ArcGIS Data Store installation directory. By default, that location is C:\Program Files\ArcGIS\DataStore\framework\etc.

3. Remove the comment mark (#) from the hostidentifier property, and set the property to the IP address of the NIC card or DNS entry you want the data store to use for network communication.

   In this example, the IP address used to communicate with the machine is 12.34.56.789.

   hostidentifier=12.34.56.789

4. Save and close the hostidentifier.properties file.

5. Restart the ArcGIS Data Store service on that machine.

If necessary, follow the steps to configure other ArcGIS Enterprise software components on machines with multiple NIC cards or DNS entries:

• ArcGIS Server

• Portal for ArcGIS

The hosting server

An ArcGIS Enterprise deployment includes one GIS Server site that acts as its hosting server. ArcGIS Data Store types are configured with the hosting server.

Create a data store using the Data Store configuration app

If you run the ArcGIS Data Store installation wizard, the Data Store configuration app automatically opens in your default web browser. Alternatively, you can open the configuration app from the computer's shortcut menu or type the URL in a browser. The Data Store configuration app URL is in the format https://datastoremachine.example.com:2443/arcgis/datastore.

1. When the app opens, click Start.

2. On the Hosting server details tab, provide one of the following in the Hosting server field to specify the ArcGIS GIS Server site that will be used as the hosting server:

   • The fully qualified domain name of one of the machines in the hosting server site, in the format gisserver.example.com

     If you have a multiple-machine site, you can use the fully qualified domain name of any of the machines in the GIS Server site.

   • The URL of the hosting server site in the format https://gisserver.example.com:6443

     Use the format noted above; do not use the URL that includes the web adaptor or load balancer name.

   • The IP address of one of the machines in the hosting server site

     If you have a multiple-machine site, you can use the IP address of any of the machines in the GIS Server site.

3. Provide the username and password of the ArcGIS Server primary site administrator for that site and click Next.

4. Provide a location for the ArcGIS Data Store content directory.

   This directory contains the data store files. When you create the relational store, the content directory also contains the default backup directory for the relational store.

   Note:

   • The data store content directory must be on the same machine where you install ArcGIS Data Store.

   • Do not use a file share for this directory.

   • You cannot use a UNC path when specifying the data store content directory.

   • The directory path cannot contain a space when you specify an ArcGIS Data Store directory for a spatiotemporal big data store.

5. Click Next.

6. Choose the type of data store to create on this machine.

   • Relational—Configures the relational store.

   • Spatiotemporal—Configures the spatiotemporal big data store. The first three machines are assigned the coordinator and data roles. Starting with the fourth machine, all machines added to the cluster are deployed in the data role only.

   • Graph—Configures the graph store. Choose the mode to use: Single-machine or Cluster.

     When deployed in cluster mode, the graph store is not viable until it contains three machines.

   • Object—Configures the ArcGIS Data Store object store. Choose the mode to use: Single-machine or Cluster.

     When deployed in cluster mode, the object store is not viable until it contains at least three machines.

   If a data store type option is not enabled, it could be that type is not installed on the machine or that data store type has already been configured on this machine.

7. Click Next.

8. Review the information on the Configuration summary tab. If it is correct, click Finish. If changes are needed, click Back and make corrections.

   When the setup completes successfully, you have a data store machine.

   If there were errors with the setup, click Next on the Configuration summary window to reopen the Data Store configuration app so you can provide different information. See Common deploy questions for causes and resolutions for common setup errors.

If you are creating a highly available data store, configure ArcGIS Data Store on additional machines.

After the data store type is created, configure a backup directory in a location that is separate from the ArcGIS Data Store installation. See Manage data store backups for more information.

Create a data store using the configuredatastore utility

You can use the configuredatastore utility to create all data store types implemented through ArcGIS Data Store. When you configure a data store, it is also registered with the hosting server, which you specify when you run the utility.

The configuredatastore utility is installed in <ArcGIS Data Store installation directory>/tools.

The configuredatastore utility is installed in <ArcGIS Data Store installation directory>\arcgis\datastore\tools.

Note:

If you script the creation of multiple machines in a spatiotemporal big data store, graph store, or object store, the first machine must be manually configured with the GIS Server site before you can script the creation of additional machines in the cluster. Include wait times in your script to be sure the additional machines are not added at the same time.

1. Open a command shell on the machine where ArcGIS Data Store is installed.

2. On the machine where ArcGIS Data Store is installed, open a command prompt using the Run As Administrator option.

3. Run the configuredatastore.sh utility and specify the fully qualified name of one of the machines in the GIS Server site with which you want to register the data store, the username and password of an ArcGIS Server administrator, the path to use for the ArcGIS Data Store data directory, and the type of data store to create.

   If you have a multiple-machine GIS Server site, you can use the fully qualified domain name of any of the machines in the site.

   If you specify a data store type that is not installed on the machine where you run the configuredatastore utility, you'll receive an error message.

   In the following examples, the name of a machine in the GIS Server site is myserver.example.com, the administrator username and password are siteadmin and T1n@sp, respectively, and the data directory is /home/dsuser/data.

   In this example, a relational store is created:

   ./configuredatastore.sh myserver.example.com siteadmin T1n@sp /home/dsuser/data --stores relational

   In the following example, a spatiotemporal big data store is created. This machine will act as a coordinator.

   ./configuredatastore.sh myserver.example.com siteadmin T1n@sp /temporal/data --stores spatiotemporal --roles coord

   The --mode operation is also used to specify the deployment mode of the object store. For object stores, the default deployment is a single-machine deployment. If you intend to use multiple machines in the object store, specify --mode cluster when creating the object store.

   In this example, a cluster of three machines are deployed as the object store. Run the same utility from each of the three ArcGIS Data Store object store machines.

   ./configuredatastore.sh myserver.example.com siteadmin T1n@sp /home/dsuser/ftcacheds
   --stores object --mode cluster
   

   The --mode operation also specifies the deployment mode of the graph store. By default, the graph store deploys in single instance mode. To create a highly available graph store, specify --mode cluster when you create the graph store.

   A graph store cluster must contain three machines. In this example, the configuredatastore utility is run from each of the three ArcGIS Data Store graph store machines.

   ./configuredatastore.sh myserver.example.com siteadmin T1n@sp /home/dsuser/ftcacheds
   --stores graph --mode cluster
   

   Note:

   The directory path cannot contain a space when you specify an ArcGIS Data Store directory for the spatiotemporal big data store or object store.

   To create more than one type of data store on the same machine using the same data directory, specify the values separated by a comma.

4. Run the configuredatastore.bat utility and specify the fully qualified domain name of a machine in the GIS Server site with which you want to register the data store, the username and password of an ArcGIS Server administrator, the path to use for the ArcGIS Data Store data directory, and the type of data store to create.

   In this example, the name of a machine in the GIS Server site is myserver.example.com, the administrator username and password are siteadmin and T1n@sp, respectively, the data directory is C:\data\, and a relational store is created:

   configuredatastore myserver.example.com siteadmin T1n@sp c:\data\ --stores relational

   In the following example, a spatiotemporal big data store is created. This machine will act as a coordinator.

   configuredatastore myserver.example.com siteadmin T1n@sp c:\temporal\data --stores spatiotemporal --roles coord

   The --mode operation is also used to specify the deployment mode of the object store. For object stores, the default deployment is a single-machine deployment. If you intend to use multiple machines in the object store, specify --mode cluster when creating the object store.

   In this example, a cluster of three machines are deployed as the object store. Run the same utility from each of the three ArcGIS Data Store object store machines.

   configuredatastore myserver.example.com siteadmin T1n@sp c:\ftcacheds
   --stores object --mode cluster
   

   Note:

   You cannot change an object store's mode after you create it.

   The --mode operation also specifies the deployment mode of the graph store. By default, the graph store deploys in single instance mode. To create a highly available graph store, specify --mode cluster when you create the graph store.

   A graph store cluster must contain three machines. In this example, the configuredatastore utility is run from each of the three ArcGIS Data Store graph store machines.

   configuredatastore myserver.example.com siteadmin T1n@sp /home/dsuser/ftcacheds
   --stores graph --mode cluster
   

   Note:

   The directory path cannot contain a space when you specify an ArcGIS Data Store directory for the spatiotemporal big data store or object store.

   To create more than one type of data store on the same machine using the same data directory, specify the values separated by a comma.

Next steps

• If you are using ArcGIS Data Store in a production system, configure each data store to create backups in a shared location.

  ArcGIS Data Store automatically generates backup files for relational stores. The default location for the relational store backup files is on the same machine as ArcGIS Data Store. Manually move these files to a different machine to secure them in the event of a data store failure, or set up a shared network location, and configure ArcGIS Data Store to write the backup files to that location.

  There is no default backup location for the other data store types implemented through ArcGIS Data Store. Register a backup location on a shared network directory or in a cloud storage location before backups can be created for these data stores.

• If you have not done so already, configure the hosting server.

  Each type of ArcGIS Data Store contains data that populates different types of hosted layers in ArcGIS Enterprise. That means the GIS Server site with which you register ArcGIS Data Store must be set as your hosting server. If you have not already federated your server site and specified it as the hosting server, do so now.

• If you require any of the data stores to be highly available or scalable, configure additional machines.

  Different data store types support different configuration modes that offer either high availability or scalability. See Add machines to a data store for instructions.

• If you require the data store to use an SSL certificate from a certifying authority or a domain certificate, replace the self-signed certificate on each data store machine.

• If your organization requires the administrator to set the passwords for databases your organization uses, change the relational store passwords.

  Relational store passwords are randomly generated when you install ArcGIS Data Store. If your site requires you to set your own passwords, change the passwords using the changepassword utility.

• If you require more than one type of data store, create additional data store types.