Skip to main content

Tune services to meet user needs

The default service property values may not be appropriate for your users' needs. That is especially true if there are a large number of users or they are making many requests to your ArcGIS Server site. This topic provides an overview of concepts, properties, and techniques that you can use to best configure your services.

Understanding service instances

When a request is made to a service in your ArcGIS Server site, such as to pan a map, navigate to an address, or display an image with a rendering rule, it is handled by an instance of the published service running on a server machine. Service instances are powered by Esri proprietary server processes called ArcSOC processes. Each ArcSOC process takes a certain amount of your machine's memory to run.

If you have many services on your ArcGIS Server site, and each uses one or more service instances that are always running, your available computer memory could eventually reach its limit. There is also an energy cost for your organization to run service instances, and if you deploy ArcGIS Server on cloud infrastructure, there is a direct monetary cost to each service instance you are running.

Accordingly, it's important for ArcGIS Server administrators to monitor the number of instances their site is running, and to limit running instances when performance is inhibited by memory usage.

Users expect quick results when they interact with your services (including products built atop services, such as web maps and apps). Adequate ArcSOC processes are required to handle the traffic your services receive. However, provisioning more server resources than a service needs wastes computer memory, energy, and money. A good goal for administrators is to pare down the number of running service instances to as many as are needed without affecting performance.

Shared and dedicated service instances

ArcGIS Server allows you to use either shared instances or dedicated instances for each compatible map or image service published to an ArcGIS Server site from ArcGIS Pro. Using shared instances conserves memory usage by pooling several active server processes for use by multiple services. Dedicated instances, on the other hand, make a service always available to handle requests using one or more server processes and are ideal to use for services that receive constant or particularly compute-intensive requests.

For new deployments of ArcGIS Server, shared instances are the default. Administrators can both choose a default instance type (whether compatible map services should initially use shared or dedicated instances) and change the instance type for an individual service at any time.

Tip:

To determine which application a service has been published from, see the Service Runtime and Instance Type properties for each service in the ArcGIS Server Manager app.

The following restrictions limit what services can use the shared instance pool:

  • Only map and image services can be configured to use the shared instance pool. Other service types, such as geoprocessing services, are not supported.

  • Only the following capabilities can be enabled: mapping, imaging, feature access, WFS, WMS, and KML. Turn off all other capabilities before converting a dedicated service to shared.

A dedicated instance map or image service runs using a designated pool of ArcSOC processes. The processes in that pool won’t be used for any other service. A shared instance map or image service runs using a pool of ArcSOC processes that are also used for any other shared instance service. By default, each shared instance ArcSOC caches the 50 most recently used services so they are ready to handle requests. If a shared instance ArcSOC already has the maximum number of services cached when a user makes a request on a new service, the ArcSOC will unload the least recently used service and load the new service. If users regularly make requests on more than 50 shared instance services, you can increase the default number of cached services per shared instance.

When to use each instance type

Shared instances are generally recommended over dedicated instances. Shared instances provide greater overall efficiency because they have the same median performance and throughput, but require significantly fewer system resources.

There are two situations where dedicated instances are recommended.

  • When business reasons require a small set of services to have elevated performance and scalability relative to other services.

  • When functionality is not supported on shared instances such as SOEs that are not thread-safe or utility networks.

Designating services as dedicated won’t automatically improve performance. To improve performance using dedicated instances, you must also reduce the number of instances available for shared instances and carefully set the number of instances for your dedicated service. Designating too many services as dedicated will likely hurt performance. It is recommended to have as few dedicated map or image services per site as possible.

Map and image services are CPU-intensive. Typically, the throughput of an ArcGIS Server focusing on map and image services is limited by the number of CPU cores. If your server machine has eight cores and you all your services are shared instances your maximum throughput is likely to reach its peak when the machine is working on eight simultaneous requests. ArcGIS Server will handle more, but those additional requests will share the CPUs. All services that use shared instances are treated with the same priority and can use the full CPU power of your server.

Dedicated instances are appropriate for a service that requires consistent performance, even if it causes other services to perform poorly. For example, if you have eight cores, you might set aside half of them just for this one service by setting the minimum and maximum number of dedicated instances to four. To ensure that at least four cores remain available for this dedicated service at all times, you might reduce the number of shared instances to four so that they never use more than the other half of the eight cores.

ArcGIS Server allows over-allocation of instances. This means that even if a machine only has eight cores, you can assign more instances than you have cores. While over-allocation is possible, it makes performance hard to predict and hard to control. Using 10% - 20% over-allocation might improve efficiency, but a large factor for over-allocation is likely to hurt your performance. Instead of over-allocating dedicated instances, you will likely see improved performance by converting some of the dedicated services to shared instances.

Minimum and maximum service instances

When a service is using dedicated instances, you can adjust the minimum and maximum number of instances allowed per machine. These parameters can help your site's services accommodate swings in traffic volume.

The minimum number of instances property represents the number of dedicated instances that are already created and available for use for a service on each ArcGIS Server machine. For example, if you set this parameter to three instances, there will always be at least three instances running on ArcSOC processes at any given time, even when the service is not receiving any requests. If you doubt that many users will concurrently use a service, consider lowering its minimum number of instances.

The maximum number of instances property represents the greatest number of instances of that service running on any given ArcGIS Server machine. As an administrator, determine how many instances of a service configuration satisfies the expected user demand at an acceptable level of performance.

The number of instances you need in a service configuration is best determined by monitoring your server over time. If your client wait times are long or requests are timing out, you may need to adjust the number of available instances or how your application uses those instances.

Consider the length of time users spend using your services. Some requests to the server are more work intensive than others. A large number of light requests for services may not be as difficult for the server to handle compared to a smaller number of work-intensive requests. Each service has a maximum wait time property and a maximum usage time property. If users' requests for services are repeatedly timing out, consider increasing the maximum wait time or the number of available instances of the service.

Once you determine the number of instances that support your clients, divide it by the number of ArcGIS Server machines in your deployment and set the maximum number of instances for the service configuration to the resulting number. For example, if you need a maximum of ten instances of a service able to simultaneously handle its requests, and you have two available ArcGIS Server machines, set the maximum number of instances to five.

Each instance consumes memory, even when the service is not being used. Setting the minimum number of instances per machine below the maximum number of instances per machine allows you to free memory when it's not being used. This feature usually comes at a small impact on performance. When new instances need to be started, requests may experience delays. To prevent this delay, you can set the minimum number of instances per machine to be equal to the maximum number of instances per machine.

Use the logs and server statistics to determine whether excessive requests are causing time-outs and if services are being used beyond their maximum usage time. Use Server Manager to adjust the number of available service instances and the maximum wait and usage times for a service.

Service instance pooling

All services published with ArcGIS Server are pooled. This means that instances of the service can be shared between multiple application sessions.

An application that uses an instance of a pooled service only uses it for the amount of time it takes to complete one request (for example, to draw a map or geocode an address). After the request is completed, the application releases its reference to the service and returns it directly to the available pool of instances.

Service instance recycling

Service recycling allows services that have become unusable to be destroyed and replaced with fresh services; recycling also reclaims resources taken up by stale services.

Services are typically shared between multiple applications and users of those applications. Through reuse, a number of things can happen to a service to make it unavailable for use by applications. For example, an application may incorrectly modify a service's state, or an application may incorrectly hold a reference to a service, making it unavailable to other applications or sessions. In some cases, services may become corrupted and unusable. Recycling allows you to keep the pool of services fresh and cycle out stale or unusable services.

During recycling, the server destroys, then re-creates, each instance in the service configuration. Recycling occurs as a background process on the server. Although you will not see anything on your screen notifying you that recycling is occurring, you can see the events associated with recycling in the log files.

Recycling destroys and re-creates all running instances of a service, whether or not those instances are above the minimum specified. To periodically return the number of running instances to the minimum specified, the service must be stopped and restarted. A good way to automate this process is to create a Python, shell, or Windows batch script that executes a custom ArcGIS Server administrative API command line executable file. This custom executable file would take the server name, service name, service type, and whether the service should be started or stopped as command line arguments.

The time between recycling events is the recycling interval. The default recycling interval is 24 hours, which you can change on the Service Editor dialog box. You can also choose the time that the configuration is initially recycled. From that time forward, recycling will occur each time the recycling interval is reached.

Services are recycled one instance at a time to ensure that instances remain available and to spread out the performance hits caused by creating a new instance of each service. Recycling occurs in random order; however, instances of services in use by clients are not recycled until released. In this way, recycling occurs without interrupting the user of a service.

If there are not enough instances available during recycling, a request is queued until an instance becomes available. If the service's maximum wait time is reached during this period, the logs record the same message that they normally would.

Checking for invalid data connections

When a service instance sits idle, it can be difficult for a server administrator to determine if connections to the source data are being successfully maintained. ArcGIS Server has built-in mechanisms to check for invalid connections to enterprise geodatabases. These checks prevent your service from appearing unresponsive after a connection to the database is dropped or interrupted.

Note:

Support for the invalid data connection check does not include file geodatabases.

You can enable the data connection validity checks by opening the Processes tab of the Service Editor dialog box in ArcGIS Server Manager and checking the Periodically check and repair data connections for idle instances check box. You also need to specify an interval in minutes at which service connections will be automatically validated (and repaired if needed). The default of 30 minutes is usually appropriate.

Enabling these checks may also help you if firewalls are closing ports to enterprise geodatabases after your services sit idle for a certain period of time. In this situation, your choice of time interval may be influenced by firewall timeout settings.

Timeouts

Understanding the various available service timeout values can help you keep your services working and available. These values are available on the Pooling tab of the Service Editor dialog box.

Once a client gets a reference to a service, it uses the service for a certain period of time before releasing it. The amount of time between when a client gets a reference to a service and when it releases it is the usage time. To ensure that clients don't hold references to services for too long (that is, they don't correctly release services), each service has a maximum time a client can use a service. If a client holds on to a service for longer than the maximum usage time, the service is automatically released and the client loses its reference to the service.

Dive in:

When you create a new service, the default value for the maximum usage time is 600 seconds (10 minutes). However, in the pregenerated PublishingTools service that comes with each ArcGIS Server site, the maximum usage time has been set at 3600 seconds (60 minutes). This is to accommodate publishing jobs that have lots of data copied to the server.

Maximum usage time also protects services from being used to do larger volumes of work than the administrator intended. For example, a service that is used by an application to perform geodatabase checkouts might have a maximum usage time of 10 minutes. In contrast, a service with one layer that is only used to draw maps in an application might have a maximum usage time of 1 minute.

When the maximum number of instances of a service is in use, a client requesting a service is queued until another client releases one of the services. The amount of time it takes between a client requesting a service and getting a service is the wait time. Each service has a maximum time a client will wait to get a service. If a client's wait time exceeds the maximum wait time for a service, the request times out.

A third timeout dictates the maximum time an idle instance can be kept running. When services go out of use, they are kept running on the server until another client needs the instance. A running instance that is not in use still consumes some memory on the server. You can minimize your number of running services and therefore conserve memory by shortening this idle timeout, the default of which is 1,800 seconds (30 minutes). The disadvantage of a short idle timeout is that when all running services time out, subsequent clients need to wait for new instances to be created.

When service instances are created in the GIS server, either as a result of the server starting or in response to a request for a server by a client, the time it takes to initialize the service instance is referred to as its creation time. The GIS server maintains a startup timeout that dictates the amount of time that can elapse on a startup attempt before the GIS server assumes its startup is hanging and cancels the creation of the service instance. The default is 300 seconds (5 minutes).

The GIS server maintains statistics both in memory and in its logs about wait time, usage time, and other events that occur within the server. The server administrator can use these statistics to determine if, for example, the wait time for a service is high, which may indicate a need to increase the maximum number of instances for that service.

There may be additional timeouts encountered in your architecture that will cause discrepancies between the service timeout values you specify and the actual timeouts experienced by clients. For example, the web server hosting ArcGIS Web Adaptor or a network load balancer may impose timeouts that affect your services.

Note:

If your site is under very heavy load, expect there to be discrepancies between the timeout values you specify and the timeouts encountered by clients.

Limiting what users can do with a service

To make it easy to control how your web services are used, each type of service has a set of allowed operations. Each operation consists of a set of methods that can be enabled or disabled as a group. Clients of the web service can only call the methods of the operations that have been allowed.

Suppose you wanted to allow consumers of a mapping web service to draw the map but not to query the data sources of the map's layers. You would then need to disable the Data operation and ensure that the Map operation is allowed.

Feature services are of special interest in this discussion because they are used for performing web-based editing of GIS data. Feature services have an additional set of operations that can be used to restrict editing functionality. You can enable or disable these on the Feature Access tab of the Service Editor dialog box in ArcGIS Server Manager. You can also prevent users from editing features they did not originally create by enforcing ownership-based access control.

See Types of services to learn about the allowed operations on various service types.

Scenarios for service tuning

The following scenarios provide some real-world examples of how an administrator might tune services to meet users' needs.

Scenario: slow service response time

You've been contacted by a user in your organization who is experiencing unacceptable display times for a particular map service. After conducting tests with the map service, you discover that a particular layer in the map service is slow to draw. To investigate further, you troubleshoot map service performance with server logs and isolate information pertaining to this map service.

Potential cause #1

Upon review of Server Manager logs, you discover excessive draw times for a layer (or layers) in the service.

Common solutions for #1

Use the following best practices to optimize the map for performance:

  • Use scale-dependent rendering.

  • Remove unused layers and data frames.

  • Use validation for definition queries.

  • Simplify layer symbology

  • Consider using cached maps when possible (if the data changes infrequently, for example).

After reviewing the service, implementing tips for optimization, and republishing the service, you and your colleague see a significant improvement in the responsiveness of the map service.

Potential cause #2

Server Manager logs indicate that lagging network access to a layer within the service could be degrading service performance.

Common solutions for #2

Use the following best practices for data access and management to minimize network latency and optimize service performance:

After reviewing the service, implementing tips for data access and management, and republishing the service, you and your colleague see a significant improvement in the responsiveness of the map service.

Scenario: ensure adequate machine resources

You've created a highly sought after web app and would like to expose it to a wider audience on an announced date, which is coming up later in the week. Because you're anticipating a high volume of requests for the services in this app, you want to ensure you have adequate machine resources to support its use.

To allocate ample server machine resources to support high utilization of this web app, you'll review ArcGIS Server statistics to identify infrequently used services and adjust service properties accordingly to accommodate consumers of this app. In turn, you'll adjust service properties accordingly for the services to be consumed in the web app.

Potential solution

Manage and fine-tune service properties to allocate resources for your site. For example, consider the length of time users spend using services. Are they being used beyond their maximum usage time? Are your end users encountering time-outs due to excessive requests on a service?

Use the following recommendations as a guide to adjust service properties to anticipate and accommodate end users:

  • Identify the most frequently used services and increase minimum instances for each. Doing so will decrease wait time for your end users.

  • Migrate services using dedicated instances that can use the shared instance pool instead.

  • Increase minimum and maximum instances, wait times, idle time, and usage time where appropriate to help mitigate delays for your end users.

  • Decrease instance, wait time, and idle time where appropriate to potentially free up system resources for services that need it most.